Overcoming E-Drocks Data Scraping Challenges with Python
A developer recently shared their experience of tackling a challenging web scraping assignment with Python, detailing the obstacles encountered and solutions discovered during a tight three-day deadline.
The journey began with the first hurdle: understanding what web scraping actually entailed. While familiar with automation and development concepts, the specialized nature of web scraping required additional research on libraries, modules, and drivers before making progress.
On day zero, the developer managed to create a basic version that successfully loaded the E-Drocks login page. Moving to day one, challenges arose when attempting to interact with email and password fields. The website required real-time email verification and OTP authentication, which prompted a shift to using temporary email services rather than personal credentials.
A significant technical challenge emerged when the code couldn’t properly interact with form fields. After troubleshooting, the developer discovered that the automation was still operating on the previous browser tab instead of the newly opened one. This required implementing explicit tab switching in the code.
By day two, the solution could open temporary emails, retrieve addresses, navigate to the target website, enter credentials, and refresh inboxes. However, a persistent issue remained with capturing the OTP codes sent during authentication. Despite consulting various resources including peers, seniors, and AI assistants like ChatGPT and BlackBox, this problem consumed a day and a half of the tight schedule.
With submission day approaching, the developer pivoted their approach after discovering the MailDM library, which creates temporary emails within the system itself. This breakthrough allowed the code to create temporary credentials, navigate the authentication process successfully, and finally access the target data.
The final challenges involved pagination and handling server response limitations. The developer implemented pagination logic to avoid scraping the same data repeatedly, and added error handling with try-except blocks to manage server response issues, allowing the code to refresh automatically when encountering errors.
The successful implementation ultimately enabled the extraction of data from E-Drocks and storage in CSV format, showcasing how persistence and adaptability can overcome web scraping challenges even under tight deadlines.