Creating a Simple Web Scraper with ChatGPT in N8N

Web scraping doesn’t always require complex tools or coding knowledge, especially when you’re targeting specific information rather than massive datasets. This article outlines a straightforward approach to creating a web scraper using ChatGPT within the N8N workflow automation platform.

Understanding the Process

The web scraper we’re building follows a three-step process:

Make an HTTP request to retrieve a website’s HTML content
Convert the HTML to readable Markdown format
Use AI to transform the Markdown into structured, usable data

Step-by-Step Implementation

1. Set Up Your Environment

Begin by creating a new workflow in N8N and giving it an appropriate name such as “web scraper”.

2. Create an HTTP Request Node

Add an HTTP request node configured with the following settings:

Request Method: GET
URL: Enter the website URL you want to scrape

When tested, this node will return the raw HTML of the target website as a single string, which isn’t immediately useful in its raw form.

3. Add a Markdown Node

Next, add a Markdown node that will convert the HTML into readable text. Connect this to your HTTP request node by dragging the output data into the HTML field of the Markdown node.

The Markdown conversion makes the content more readable for humans, but it’s still a large chunk of unstructured text.

4. Implement OpenAI Processing

Add an OpenAI node using the “Message a Model” function. This will transform the Markdown content into structured data. Configure it with:

Model: Choose a cost-effective option like Omini
System Role: “You are a helpful, intelligent, web scraping assistant”
First User Message: Include instructions for converting the Markdown into structured data with your desired format
Second User Message: Drag in the output from the Markdown node

The OpenAI node will process the Markdown and return structured data that can be used in subsequent workflow steps.

Benefits of This Approach

This method offers several advantages:

No coding required
Targeted data extraction
Structured output that can be exported to Excel or other formats
Flexibility to extract different types of information (links, phone numbers, etc.)

Once processed, the data appears in a tabular format with each piece of information in its own row, making it easy to manipulate and utilize within N8N workflows.

Conclusion

For single-website scraping tasks that don’t require massive data collection, this ChatGPT-powered approach provides an elegant solution without the complexity of traditional web scraping tools. The structured data output can be further processed, exported, or integrated with other systems as needed.