How to use Firecrawl?

Leon Chase

• 4 min read

Firecrawl is a powerful web scraping and crawling tool designed to extract structured data from websites. It allows users to automate the process of collecting data from web pages, even those that are dynamically generated or require interaction (e.g., clicking buttons, filling forms). Firecrawl is particularly useful for developers, data analysts, and businesses that need to gather large amounts of data from websites for tasks like price monitoring, lead generation, content aggregation, and more.

Here’s a step-by-step guide on how to use Firecrawl:

1. Getting Started with Firecrawl

Step 1: Sign Up

  • Visit the Firecrawl website and sign up for an account.
  • Once registered, log in to access the platform.

Step 2: Install Firecrawl CLI (Optional)

If you prefer to use Firecrawl programmatically, you can install the Firecrawl CLI via npm (Node.js package manager):

npm install -g @firecrawl/firecrawl

After installation, log in to your Firecrawl account using the CLI:

firecrawl login

2. Creating a New Crawl

Step 1: Start a New Crawl

You can start a new crawl either through the web interface or via the CLI/API.

Using the Web Interface
  1. Go to the Crawls section in your Firecrawl dashboard.
  2. Click "New Crawl".
  3. Enter the starting URL of the website you want to crawl (e.g., https://example.com).
Using the CLI

To start a crawl via the CLI, run:

firecrawl crawl https://example.com

Step 2: Configure Crawl Settings

Before starting the crawl, you can configure various settings to control how the crawler behaves:

  • Depth: Specify how deep the crawler should go (e.g., how many levels of links it should follow).
  • Max Pages: Set a limit on the number of pages to crawl.
  • Selectors: Define CSS selectors to extract specific data from the pages.
  • Exclusions: Exclude certain URLs or sections of the site from being crawled.
  • Dynamic Content: Enable JavaScript rendering if the site uses dynamic content (e.g., SPAs built with React or Angular).
Example: Configuring Selectors

If you want to extract product names and prices from an e-commerce site, you can specify CSS selectors for those elements:

{
  "selectors": {
    "product_name": ".product-title",
    "product_price": ".product-price"
  }
}

3. Running the Crawl

Once you’ve configured the crawl, you can start it.

Using the Web Interface

  • Click "Start Crawl" in the Firecrawl dashboard.
  • The crawler will begin navigating the website and extracting data based on your configuration.

Using the CLI

Run the following command to start the crawl:

firecrawl crawl https://example.com --depth 2 --max-pages 50

This command will crawl up to 50 pages, following links up to 2 levels deep.

4. Monitoring the Crawl

While the crawl is running, you can monitor its progress in real-time.

Using the Web Interface

  • In the Firecrawl dashboard, you’ll see a live status update of the crawl, including the number of pages visited, data extracted, and any errors encountered.

Using the CLI

The CLI will display logs as the crawl progresses, showing which pages are being visited and what data is being extracted.

5. Extracting Data

Once the crawl is complete, you can access the extracted data.

Using the Web Interface

  • After the crawl finishes, you can view the extracted data in the Firecrawl dashboard.
  • The data will be displayed in a structured format (e.g., JSON, CSV), and you can download it directly from the dashboard.

Using the CLI

You can export the data to a file using the CLI:

firecrawl export <crawl_id> --format json

This will export the data from the specified crawl ID in JSON format.

6. Automating Crawls

Firecrawl allows you to schedule crawls to run automatically at regular intervals.

Step 1: Schedule a Crawl

  • In the Firecrawl dashboard, go to the Scheduling section.
  • Set up a recurring crawl by specifying the frequency (e.g., daily, weekly) and the target URL.

Step 2: Use Webhooks (Optional)

You can configure Firecrawl to send data to a webhook endpoint after each crawl. This is useful for integrating with other systems or triggering downstream processes.

7. Advanced Features

Handling Dynamic Content

Firecrawl supports headless browsing, which allows it to render JavaScript-heavy websites. If the site you’re crawling relies on JavaScript to load content, ensure that JavaScript rendering is enabled in your crawl settings.

Custom Headers and Authentication

If the website requires authentication (e.g., login credentials), you can configure custom headers or cookies in your crawl settings:

{
  "headers": {
    "Authorization": "Bearer YOUR_TOKEN"
  }
}

Proxy Support

Firecrawl supports proxy servers to avoid IP bans or rate limits. You can configure proxies in your crawl settings:

{
  "proxies": ["http://proxy1.com", "http://proxy2.com"]
}

8. Example Use Cases

E-commerce Price Monitoring

  • Goal: Track product prices across multiple e-commerce sites.
  • Steps:
    1. Set up a crawl targeting product pages.
    2. Use CSS selectors to extract product names and prices.
    3. Schedule the crawl to run daily and export the data to a database or analytics tool.

Lead Generation

  • Goal: Scrape contact information from business directories.
  • Steps:
    1. Crawl business directory websites.
    2. Extract fields like company name, email, phone number, and address.
    3. Export the data to a CRM or spreadsheet for further processing.

Content Aggregation

  • Goal: Collect articles, blog posts, or news from multiple sources.
  • Steps:
    1. Crawl news websites or blogs.
    2. Extract article titles, authors, publication dates, and content.
    3. Aggregate the data into a central repository or CMS.

SEO Analysis

  • Goal: Analyze website structure, meta tags, and backlinks for SEO optimization.
  • Steps:
    1. Crawl the target website.
    2. Extract meta tags, headings, and internal/external links.
    3. Analyze the data to identify SEO opportunities or issues.

9. Tips for Using Firecrawl Effectively

Tip 1: Respect Website Policies

  • Always check the website’s robots.txt file and terms of service before crawling. Some websites prohibit automated scraping, and violating these policies could result in legal consequences.

Tip 2: Optimize Selectors

  • Use precise CSS selectors to extract only the data you need. Avoid overly broad selectors that might capture unnecessary information.

Tip 3: Handle Rate Limits

  • If the website has rate limits or blocks frequent requests, use proxies or slow down the crawl speed to avoid getting banned.

Tip 4: Test Small Crawls First

  • Before running a large-scale crawl, test your configuration on a small subset of pages to ensure that the data is being extracted correctly.

Tip 5: Automate Data Processing

  • Integrate Firecrawl with other tools (e.g., Zapier, Airtable, Google Sheets) to automate data processing and analysis after the crawl.

10. Conclusion

Firecrawl is a versatile and user-friendly tool for web scraping and crawling. Whether you’re extracting product details from e-commerce sites, aggregating content from blogs, or monitoring competitors’ pricing, Firecrawl provides the flexibility and power to handle complex crawling tasks with ease.

By leveraging Firecrawl’s web interface or CLI/API, you can automate data extraction, schedule recurring crawls, and integrate the data into your workflows. With proper configuration and respect for website policies, Firecrawl can become an essential tool for gathering valuable insights from the web.