Integrating QAI and Oxalabs WebScraper API for Automated Research Reports

Python developers looking to automate research and content generation now have a powerful new combination of tools at their disposal. By integrating an open-source AI agent framework with specialized web scraping capabilities, it’s possible to create a system that autonomously researches topics and generates structured reports.

The project combines QAI, an open-source framework for AI agents, with Oxalabs WebScraper API to create an automated research workflow. This integration allows for scraping Google search results and generating structured markdown reports with minimal human intervention.

Project Structure

The implementation relies on several key components working together:

Agent Configuration

The system uses two specialized agents defined in the agents.yaml file:

• Researcher Agent: Generates appropriate search terms based on user questions, scrapes Google results, and identifies the most relevant information
• Analyst Agent: Processes the collected data to create comprehensive, well-structured reports

Task Definition

The tasks.yaml file outlines two primary tasks:

• Search Task: Configures how the Researcher agent interacts with the WebScraper API, including expected output formats
• Report Task: Specifies how the Analyst agent should use the search results as context to generate the final markdown report

Custom Tool Implementation

A crucial component is the customtool.py file, which contains the Oxalabs search class. This class handles:

• Communication with the WebScraper API
• Google search result retrieval
• Automatic parsing of the scraped data into a format usable by the QAI agents

System Integration

The Qudat.py file initializes both the agents and their assigned tasks, then assembles them into a functional crew. The main.py file serves as the entry point, accepting user inputs and triggering the workflow.

Workflow Process

When a user submits a question, the system follows these steps:

1. The Researcher agent formulates appropriate search terms based on the question
2. The WebScraper API retrieves relevant Google search results
3. The Researcher agent filters and selects the most valuable information
4. The Analyst agent processes this information to create a comprehensive report
5. The system returns a clean, structured markdown document containing the findings

Applications and Benefits

This integration offers significant advantages for researchers, content creators, and data analysts:

• Automated information gathering from across the web
• Consistent formatting and structure in research reports
• Significant time savings compared to manual research
• Scalable research capabilities for handling multiple topics

By leveraging OpenAI as the LLM provider, the system benefits from advanced language processing capabilities, resulting in high-quality content generation based on the scraped data.

The combination of QAI’s agent framework with Oxalabs’ specialized scraping tools represents a powerful approach to automated research and content generation, streamlining what would otherwise be a time-consuming manual process.