Building a Powerful Web Scraping System with CROFLA AI and N8N

by

Building a Powerful Web Scraping System with CROFLA AI and N8N

Web scraping and crawling are essential techniques for data collection in modern applications. This comprehensive guide explores how to set up a robust system that leverages CROFLA AI for local scraping and N8N for workflow automation, creating a powerful data ingestion pipeline.

Introduction to CROFLA AI

CROFLA AI is a local web scraper that can effectively replace services like APFI. It functions using playwright to crawl web pages directly on your local system. While you can install it using pip, this guide focuses on using Docker for deployment.

Setting Up CROFLA AI

Begin by cloning the repository to your server using git. The repository contains several important files, including an .nv.txt file that serves as a template for your API configurations.

Copy this template to create a .llm.nv file, which will store your actual API keys. CROFLA AI supports multiple language models including:

  • GROC (which offers free API keys)
  • OpenAI
  • Gemini
  • DeepSeek
  • Mistral

The Docker Compose file defines all these services, exposing the port 11235 for access. Running docker-compose up -d downloads and starts the CROFLA AI service.

Creating the N8N Workflow

The N8N workflow consists of several connected components:

1. Webhook Entry Point

The workflow begins with a webhook that accepts URL submissions via POST requests. This serves as the entry point for any web page you want to scrape.

2. Scraping Request

The URL from the webhook is sent to the CROFLA AI server with appropriate authentication headers. The request includes parameters like priority level and output format preferences, as well as options to follow redirects.

3. Task Processing

CROFLA AI returns a task ID, which the workflow uses to poll for results. A wait node pauses for a few seconds (configurable based on your machine’s performance) before checking if the scraping task is complete.

4. Data Formatting

Once complete, the scraped content is formatted as markdown with citations. This format is particularly AI-friendly and ensures the data can be effectively utilized downstream.

5. Database Integration

The final step inserts the scraped and formatted content into a QGround database, making it available for retrieval and analysis.

Configuring N8N Agents

To make the scraping functionality available programmatically, you can configure an N8N agent with an HTTP node. The agent description should clearly explain that it scrapes web pages and ingests content into the database.

Since the workflow runs locally, the agent’s HTTP node should be configured to connect to your internal network (e.g., 172.17.0.1 instead of an external IP). Proper authentication using header credentials ensures secure communication between the agent and the webhook.

Security Considerations

Implementing header authentication between your N8N workflows and agents is crucial, especially if your webhook is exposed to the internet. This prevents unauthorized triggering of your scraping workflow.

Enhancing the System

The basic workflow can be extended to:

  • Scrape entire websites by retrieving sitemaps and looping through all URLs
  • Configure the QGround database as a tool for agents to access scraped data
  • Integrate with different embedding models (like OpenAI’s text-embed-3-small) for improved data retrieval

This setup creates a complete system where agents can both scrape new content and query previously scraped information, building a continuously expanding knowledge base.

Conclusion

This web scraping system combines the power of local scraping through CROFLA AI with the workflow automation capabilities of N8N. The result is a flexible, secure, and efficient data collection pipeline that can be integrated into various applications.

Leave a Comment