Building a Smart Shopping Assistant with Web Scraping and LLMs

Creating an intelligent shopping assistant that can provide personalized product recommendations is now within reach for developers thanks to the powerful combination of web scraping and large language models (LLMs). This revolutionary approach allows users to ask natural language questions about products while receiving answers based on real-time, locally-relevant data.

The Power of Combining Scraped Data with LLMs

When properly implemented, this technology enables incredibly specific queries like “What is the best programming computer for under $700 that’s available at my local store?” The system then searches for the best options by analyzing scraped product information and leveraging semantic understanding to deliver personalized recommendations.

How the System Works

The solution consists of several key components working together:

1. A scraping system that collects product data via a single API endpoint
2. A semantic vector database for storing and searching the product information
3. An LLM interface that can interpret user questions and generate helpful responses

Technical Implementation

The backend is built using FastAPI and includes several crucial components:

Data Collection

The system uses Scraper API to fetch product data from retail websites. This provides structured JSON data that’s easy to process and store. The implementation includes functions to:

• Extract product IDs from URLs
• Handle different product conditions (new, refurbished, etc.)
• Collect product reviews and ratings

Vector Database Integration

All product information is stored in ChromaDB, a vector database that enables semantic searching. Using OpenAI’s embedding function, the system converts both user queries and product descriptions into vectors for accurate matching.

Query Processing

When a user asks a question, the system:

1. Converts the query into a vector representation
2. Searches the database for the most relevant products
3. Formats the matched products into a context-rich prompt
4. Sends this prompt to an LLM to generate a natural language response

User Interface

While the backend runs on FastAPI, a Streamlit interface provides a user-friendly way to interact with the system. Users can ask natural language questions and receive detailed responses based on the scraped product data.

Real-World Applications

This technology demonstrates impressive versatility. In testing, the system successfully answered diverse queries such as:

• “Which PC has the best reviews for under $1,000?”
• “What are the top reviews for this computer?”
• “Are there any affordable options for light gaming?”
• “Are there any affordable options for light gaming with an i7 processor?”
• “What product would be best for someone who travels often?”

For each question, the system provided relevant recommendations based on current product availability, specifications, prices, and user reviews.

Future Implications

This implementation represents the growing convergence of web scraping and artificial intelligence. As these technologies continue to mature, we can expect even more sophisticated applications that blend real-time data collection with intelligent analysis.

For developers and businesses, mastering these techniques opens new possibilities for creating highly personalized user experiences that leverage both public web data and advanced AI capabilities.