Building a Website Analyzer: Scraping and Content Assessment Tool

A powerful new website analysis tool has been developed that can automatically scrape websites and provide in-depth content assessment. This tool offers a streaming interface that displays results in real-time while processing website content.

The analyzer traverses through websites page by page, detecting all URLs and extracting data from each page it discovers. One of its key features is the ability to categorize content – distinguishing between articles, blog posts, and other page types.

For each page analyzed, the tool evaluates several important metrics:

• Commercial content ratio – how promotional versus educational the content is
• Technical depth – assessing whether content answers technical questions
• Educational value – determining the level of informational content
• Content categorization – identifying if pages are articles, blog posts, or other content types

The developer noted that the project took approximately two and a half hours to build, with most of the time invested in creating the scraper functionality. A particularly challenging aspect was implementing screenshot capabilities, as the tool captures a full-page screenshot of each analyzed page.

Under the hood, the analyzer employs BAML functions with jc4o mini to determine if content is a blog post, article, or other type of content. It analyzes various content attributes including commercial elements, educational depth, and key insights. The system handles errors gracefully by capturing exceptions when content is too large or when other issues arise.

Currently, the tool uses the local disk as a cache system, though the developer mentioned it could be modified to save results to a database. The streaming UI required significant development effort, implemented using an in-memory dictionary that acts like a real-time database, constantly updating as new pages are discovered and analyzed.

The interface displays status updates for all pages, showing which are pending, loading, or completed. This provides users with clear visibility into the analysis process as it unfolds.

This website analyzer represents a powerful tool for content assessment, SEO analysis, and website auditing that could be integrated with large language models or other analysis systems for even deeper insights.