Building Adaptable AI Agents with RISA for Reliable Web Scraping and Data Analysis

Web scraping has traditionally been plagued by fragility – even minor website changes can break scrapers entirely. Now, a powerful new approach combines RISA’s code interpreter capabilities with LLMs to create highly adaptable data extraction workflows.

RISA offers a flexible solution for building resilient AI agents that can handle dynamic web content without constant maintenance. When paired with LANGRAF for orchestration and browser-based tools for navigation, this approach creates powerful data analysis systems.

The Power of Code Generation for Web Scraping

The key innovation in this approach is using LLMs to generate extraction code on the fly, which is then executed by RISA’s code interpreter. This two-step process makes scraping significantly more adaptable to website changes:

1. The LLM analyzes HTML structure and generates appropriate extraction code
2. RISA executes this code against the target HTML

This approach means that even if a website changes its structure, headers, or column order, the system can adapt automatically without manual intervention.

Building a Fuel Price Analysis Agent

A practical implementation of this technology is an agent that tracks and analyzes fuel price changes across US states by scraping data from the AAA website. The workflow includes:

1. Scraping the AAA website using browser-based tools to avoid blocking
2. Extracting data from HTML to CSV using RISA and LLM-generated code
3. Checking for data changes from previous runs
4. Analyzing notable changes when detected
5. Generating appropriate visualization charts based on the analysis

The entire workflow is orchestrated through LANGRAF, which manages the sequence of operations and conditionally executes steps based on detected data changes.

Flexible Data Visualization

One of the most powerful aspects of this approach is the ability to generate appropriate visualizations dynamically. Since the agent doesn’t know in advance what patterns might emerge in the data, it leverages the LLM to:

1. Analyze the latest data changes
2. Determine the most appropriate chart type
3. Generate visualization code that RISA executes

This results in visualizations that adapt to the current data context. For example, initial runs might focus on absolute values with color-coded ranges, while subsequent runs might highlight changes over time.

Technical Implementation

The implementation uses several key technologies:

• RISA for code interpretation and execution
• LANGRAF for workflow orchestration
• Browser-based tools to avoid scraping blocks
• LLMs (like Claude) for code generation
• Beautiful Soup for HTML parsing
• Plotly for chart generation

Custom runtimes in RISA ensure that necessary libraries like Beautiful Soup and Plotly are available for the generated code to use.

The Future of Web Scraping

This approach represents a significant advancement in web scraping reliability. Traditional scrapers required constant maintenance as websites evolved, but this LLM-powered approach can adapt to changes without developer intervention.

The pattern of using LLMs to write code that is then executed by interpreters like RISA creates highly adaptable systems that can be applied to many data processing challenges beyond web scraping.

As these technologies mature, we can expect to see more resilient data pipelines that maintain functionality even as their data sources evolve and change.