Firecrawl Guide: Turn Websites into LLM-Ready Data

Introduction

Building AI applications often requires feeding them high-quality data from the web. However, traditional web scraping tools output messy HTML that confuses Language Models (LLMs).

Enter Firecrawl: a developer-first tool designed to turn any website into clean, LLM-ready markdown or structured data with a single API call.

What is Firecrawl?

Firecrawl is an API service that simplifies the process of gathering data from the web. Unlike traditional scrapers that require you to parse DOM elements, Firecrawl does the heavy lifting of rendering pages and converting them into formats that AI agents and LLMs can understand easily.

Diagram: From URL to LLM-Ready Markdown

Core Features

Installation & Setup

Firecrawl offers SDKs for both Python and Node.js. First, sign up at firecrawl.dev to get your API key.

pip install firecrawl-py

npm install @mendable/firecrawl-js

Usage Guide

1. Scraping a Single Page

from firecrawl import FirecrawlApp

app = FirecrawlApp(api_key="YOUR_API_KEY")

# Scrape a website and get Markdown
response = app.scrape_url(
    'https://docs.firecrawl.dev', 
    params={'formats': ['markdown']}
)

print(response['markdown'])

2. Crawling an Entire Site

# Crawl a website with a limit
status = app.crawl_url(
    'https://docs.firecrawl.dev',
    params={
        'limit': 100,
        'scrapeOptions': {'formats': ['markdown']}
    }
)

print(status['jobId'])

Use Cases