How to Build a Web Scraping Bot for Extracting Email Addresses and Phone Numbers

Building an automated solution for extracting contact information from websites can save significant time and effort. This step-by-step guide demonstrates how to create a specialized bot in Octopus that scrapes email addresses and phone numbers from any website efficiently.

Getting Started with Octopus

The process begins with preparing a list of target URLs in an Excel file. These can be any websites you need to extract contact information from. Once your list is ready, follow these steps:

1. Open Octopus and click on “New” then “Custom Task”
2. Paste your list of URLs into the designated field
3. Save the task, which will open your configuration panel
4. Name your bot something descriptive like “Email and Phone Bot”

Setting Up the Loop Parameters

In the loop settings, you can configure how many URLs to process. Octopus has a limit of 10,000 URLs per set. You can import URLs directly from a file, task, or batch as needed.

Creating Custom Experts

The power of this scraping approach lies in using “experts” – specialized code snippets that pinpoint specific elements on a webpage. For this bot, you’ll need three custom experts:

• An email extraction expert
• A phone number extraction expert
• A click button expert (to navigate to contact pages)

Configuring the Workflow

Follow these configuration steps to set up your bot properly:

1. Add the email expert by clicking the plus button, selecting “X-Ted data”
2. Go to options and set the wait reproduction time to 2 seconds
3. Add a custom field for page-level data to track the URL being processed
4. Add another custom task for capturing data and implement the email expert
5. Repeat the process for the phone number expert
6. Add a click button function with the appropriate expert code to automatically navigate to contact pages

Running and Exporting Results

Once your workflow is configured:

1. Click on “Run” and select “Standard Mode”
2. The bot will automatically navigate through your URLs, locate contact pages, and extract email addresses and phone numbers
3. When complete, export the results as an Excel file
4. Review your extracted data, which will be organized by URL with corresponding email addresses and phone numbers

This automated approach significantly reduces the manual effort required to collect contact information from multiple websites and provides a structured dataset for your outreach or database needs.