Web Automation: Combining Web Scraping with Form Submission

Web automation can significantly streamline repetitive tasks, particularly when dealing with data extraction and form submissions. This article explores an impressive automation project that combines web scraping with automated form submission using the Robot Framework and Selenium.

Project Overview

The automation project demonstrated here accomplishes two main tasks:

1. Web scraping player salary data from a sports website (Hoops Hive)
2. Automatically submitting this data to a practice form

Specifically, the project extracts the names and salaries of the first five players from the Hoops Hive website, saves this information to a CSV file, and then submits each player’s details to a form with predefined data.

Implementation Components

Required Libraries

• Selenium – For web automation and browser control
• Operating System libraries – For CSV file creation and management
• String – For cleaning salary data (removing commas and dollar signs)
• Collections – For managing lists of scraped data

Key Variables and Selectors

The project defines several important variables:

• Browser type (Chrome)
• Website URLs (both the scraping source and form submission target)
• CSV file path
• XPath selectors for targeting specific HTML elements on the Hoops Hive website
• Predefined date of birth list for form submission

Main Functions

The automation process is broken down into several key functions:

1. Data Extraction

• Extracting player names – Retrieves the first five player names from the table, skipping the header row
• Extracting salaries – Pulls salary data and cleans it by removing dollar signs and commas
• Saving to CSV – Creates a properly formatted CSV file with column headers and player data

2. Form Submission

• Opening the form page and waiting for input fields to load
• Reading player data from the CSV file
• Assigning predefined dates of birth to each player
• Filling and submitting the form for each player
• Automatically closing the browser after completion

Execution Process

When the script runs, it performs the following sequence:

1. Opens Chrome browser and navigates to the Hoops Hive website
2. Extracts the names and salaries of the first five players
3. Cleans the data (removing unnecessary characters)
4. Saves the data to a CSV file
5. Closes the browser
6. Opens a new browser session for form submission
7. Reads the player data from the CSV file
8. Submits the form for each player with their name, salary, and a predefined date of birth
9. Closes the browser after all submissions are complete

Benefits of This Approach

This end-to-end automation flow demonstrates how to:

• Extract data from dynamic web pages
• Process and clean raw data
• Store information in a structured format
• Use the extracted data to automate form submissions

Such automation is particularly valuable for testing or data pipeline tasks that would otherwise require manual intervention, saving significant time and reducing the possibility of human error.

Conclusion

The combination of web scraping and form submission automation showcases how relatively simple scripts can create powerful automation workflows. By using tools like the Robot Framework and Selenium, developers and testers can create efficient solutions for repetitive web-based tasks.