FireCrawl: Extracting Structured Web Data With The New Extract Function
FireCrawl has introduced a powerful new feature that makes web scraping more efficient and structured. The open-source tool, which provides 500 free credits to new users, now offers an extract function that can generate structured data from websites based on specific schemas.
The dashboard provides easy access to an API key, comprehensive documentation, and a playground featuring different scrapers including single-wale, crawl, map, and search functionalities. The newly added extract function enables users to scrape websites with specific prompts to gather targeted information.
How The Extract Function Works
The extract function transforms unreadable HTML code into structured, AI-friendly formats. When used properly, it can scrape not just individual pages but entire domains including all sub-sites. This makes it particularly valuable for comprehensive data collection projects.
FireCrawl’s approach generates data in Markdown format, which serves as an excellent starting point for knowledge spaces in chatbots or voice agents. The structured format makes it easier for LLM (Large Language Model) systems to process and utilize the information.
Implementing The Extract Function
To use the extract function effectively:
- Begin by defining your schema in JSON format
- Set up an HTTP request with your API key
- Include your target URL and the schema in the request body
- Execute a second HTTP request to retrieve the processed data
- Implement logic to handle processing time (using if/wait nodes)
The process may take 30-40 seconds to complete as FireCrawl processes the website data according to your schema. Once completed, you’ll receive structured data that matches your defined format.
Practical Applications
This functionality opens numerous possibilities for data automation workflows. You can:
- Export the data to Google Sheets
- Create automated loops to process multiple URLs
- Build AI agents that use the extracted data
- Generate stories, reports, or analyses based on the collected information
The structured approach makes the data immediately usable for various applications, eliminating the need for additional processing steps.
With FireCrawl’s extract function, web scraping becomes more targeted and efficient, allowing users to extract exactly the information they need in a format that’s ready for immediate use.