Firecrawl v3 landed this month, and it’s the first scraping release in a while that changes what a solo builder can pull off. The headline: an agentic crawler that handles login walls, infinite-scroll catalogs, and JavaScript-heavy SPAs without a single Playwright selector — plus a JSON extraction mode that returns schema-typed objects instead of a wall of markdown you re-parse with another LLM call. This isn’t a frontier-lab announcement, so it slid past most feeds, but it’s arguably more useful to anyone shipping agents than the last three model releases combined. If you’ve been duct-taping Puppeteer scripts to feed a RAG pipeline, the duct tape just became optional.
What’s new in Firecrawl v3
The agentic crawler
Previous versions of Firecrawl were excellent at one job: point it at a URL, get clean markdown back. That works beautifully for documentation sites and blogs, and falls apart the moment a page requires a click, a scroll, a cookie banner dismissal, or a session. v3 puts a model in the loop of the browsing itself. You describe the goal in natural language — “get every product on this catalog including paginated results” — and the crawler decides what to click, when to scroll, and when it’s done. Under the hood it’s still a real browser session; the difference is that the navigation policy is generated per page instead of hardcoded by you.
Structured extraction as a first-class output
The Firecrawl extract API now accepts a JSON schema and returns data typed against it. Instead of scraping markdown and then paying for a second model call to turn that markdown into rows, you declare the shape up front and get objects back. For anyone doing structured data extraction with an LLM in a two-step pipeline, this collapses it into one. Fewer tokens, fewer parse failures, and a schema violation becomes an error you can retry rather than a hallucinated field you discover three weeks later in production.
Session and auth handling
Less glamorous, but it saves the most grief. v3 persists cookies across a crawl and accepts credentials or a stored session, which means gated content — internal wikis, customer portals, member-only forums you have legitimate access to — is reachable through the same API as public pages. Combined with agentic navigation, that’s the difference between “scrapes public blogs” and “scrapes the thing you actually needed.”
Why it matters
- RAG pipelines stop needing a bespoke scraper per source. The biggest hidden cost in building an AI web scraper for RAG is that every site breaks differently. An agentic crawler absorbs that variance instead of pushing it into your codebase.
- One API call replaces scrape-then-extract. Schema-typed output removes an entire LLM hop. On a 10,000-page crawl that’s a material cost difference, not a rounding error.
- Lead-gen and market research scraping gets accessible to non-specialists. Paginated directories and JS-rendered listings were the two things that reliably defeated hobbyist scrapers. Both are now table stakes.
- Enterprise scraping vendors lose their moat on the mid-market. The pitch for Zyte or Bright Data was always “we handle the hard sites.” A lot of “hard” just got commoditized.
- Web scraping for AI agents becomes a tool call, not a subsystem. Hand an agent a Firecrawl tool and a schema, and it fetches its own grounding data mid-task.
- Failure modes shift from brittle to probabilistic. That’s the tradeoff, and it’s real. Your selector no longer breaks loudly on a redesign; instead the agent quietly returns slightly different data. Validation stops being optional.
How to use Firecrawl v3 today
-
Install and set your key. Grab a key from the dashboard; the free tier covers everything below.
pip install firecrawl-py export FIRECRAWL_API_KEY="fc-your-key-here" -
Start with a plain scrape to confirm auth works. Don’t debug an agentic crawl before you’ve confirmed the boring path.
from firecrawl import Firecrawl app = Firecrawl() # reads FIRECRAWL_API_KEY from env doc = app.scrape("https://ailearningguides.com", formats=["markdown"]) print(doc.markdown[:500]) -
Switch to schema-typed extraction. Define the shape you want with Pydantic (or raw JSON Schema) and let the extract API fill it. This is the highest-leverage change in v3.
from pydantic import BaseModel from typing import List class Product(BaseModel): name: str price: str in_stock: bool product_url: str class Catalog(BaseModel): products: List[Product] result = app.scrape( "https://example-shop.com/collections/all", formats=[{ "type": "json", "schema": Catalog, "prompt": "Extract every product listed on the page." }] ) for p in result.json["products"]: print(p["name"], p["price"]) -
Turn on agentic navigation for multi-step pages. Give it a goal in plain language and a bound on how far it can wander. Always set a limit — an unbounded agentic crawl on a large site is how you burn credits at 3am.
job = app.crawl( "https://example-shop.com/collections/all", prompt="Visit every product page, including all paginated results. " "Skip blog posts, reviews and category landing pages.", limit=200, scrape_options={ "formats": [{"type": "json", "schema": Product}] } ) print(job.status, len(job.data)) -
Handle gated content with a session. Only do this on sites you own or have explicit permission to access — credentials in a scraper are a compliance decision, not a technical one.
curl -X POST https://api.firecrawl.dev/v2/scrape \ -H "Authorization: Bearer $FIRECRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://portal.example.com/reports", "actions": [ {"type": "write", "selector": "#email", "text": "me@example.com"}, {"type": "write", "selector": "#password", "text": "'"$PORTAL_PW"'"}, {"type": "click", "selector": "button[type=submit]"}, {"type": "wait", "milliseconds": 2000} ], "formats": ["markdown"] }' -
Wire it into an agent as a tool. Once extraction is schema-typed, the output is safe to hand straight to a model — no markdown babysitting.
{ "name": "fetch_structured_page", "description": "Fetch a URL and return typed data matching the given schema.", "input_schema": { "type": "object", "properties": { "url": {"type": "string"}, "goal": {"type": "string", "description": "What to extract, in plain language"} }, "required": ["url", "goal"] } } -
Validate everything on the way out. Agentic scraping fails softly. Assert on row counts and required fields, and alert when a crawl returns 40% of what it did last week.
rows = [Product(**p) for p in result.json["products"]] assert len(rows) >= EXPECTED_MIN, f"Only got {len(rows)} rows — site likely changed"
How it compares: Firecrawl vs Apify and the rest
The Firecrawl vs Apify question comes up constantly, and the honest answer is that they solve adjacent problems. Apify is a marketplace and orchestration platform; Firecrawl is an API with opinionated defaults for LLM consumption. Here’s the practical breakdown for a builder deciding this week.
| Tool | Best for | LLM-ready output | Agentic navigation | Setup cost |
|---|---|---|---|---|
| Firecrawl v3 | Feeding RAG and agents clean, typed data fast | Native — markdown and JSON schema | Yes, built in | Minutes |
| Apify | Pre-built actors for specific sites at scale | Varies by actor; usually raw JSON | Per-actor, hand-coded | Hours to days |
| Bright Data | High-volume enterprise with heavy anti-bot targets | No — raw HTML or custom | No | Days plus sales call |
| Playwright / Puppeteer | Full control, deterministic flows you own | No | No — you write it | Days per site |
| Jina Reader | Quick single-URL to markdown, free and simple | Markdown only | No | Seconds |
The decision rule: if your bottleneck is engineering time per source, Firecrawl wins. If your bottleneck is volume against hostile targets, you still want a proxy-heavy enterprise vendor. And if you’re scraping one site whose shape you fully control, Playwright is still cheaper and more predictable than paying per page.
What’s next
The obvious roadmap direction is deeper agent integration — MCP servers, native tool definitions for the major frameworks, and crawls an agent can steer mid-flight rather than configure up front. Firecrawl already ships an MCP server, and the gap between “agent calls a scrape tool” and “agent runs a persistent browsing session it can reason about across turns” is the interesting territory. Watch for stateful crawl sessions an agent can pause, inspect, and resume.
The pressure point to watch is cost and rate limits. Agentic navigation means an LLM call per navigation decision, and that math gets ugly fast on large crawls. Expect a tiered model where cheap deterministic crawling handles the 90% of pages that are simple and the agent only engages when a page defeats the fast path. If Firecrawl doesn’t ship that, someone will build it as a wrapper.
The third thing worth tracking is the legal and robots.txt conversation. Agentic crawlers that log in and navigate like humans sit in murkier territory than a polite markdown fetcher, and the norms around AI training data and scraping are still being written. Build with rate limits, respect robots.txt by default, and don’t scrape anything you couldn’t defend in an email to the site owner. That’s not just ethics — it’s how you keep your pipeline from getting blocked.
Frequently Asked Questions
Is Firecrawl v3 free to use?
There’s a free tier with a monthly credit allowance that’s genuinely enough to build and test a pipeline. Agentic crawls consume more credits per page than plain scrapes, because each navigation decision costs a model call. Budget accordingly, and always set a limit on crawl jobs.
Can it get past Cloudflare and other anti-bot systems?
Sometimes, not reliably. Firecrawl handles standard JS rendering, cookie walls and basic bot checks well. Aggressive fingerprinting and CAPTCHA-gated sites are still a proxy-vendor problem. If your target list is heavily protected, budget for a fallback.
How does the extract API compare to scraping markdown and running my own LLM pass?
The Firecrawl extract API does roughly the same thing, but co-located with the page fetch and validated against your schema before it returns. You save a round trip, save tokens, and get typed errors instead of malformed JSON. If you already have a heavily tuned extraction prompt, keep it — pass it in as the prompt field alongside your schema.
Does agentic web scraping work on single-page apps?
Yes — that’s one of the main reasons it exists. SPAs that render content only after interaction were the classic failure case for markdown-oriented scrapers. The agent clicks, scrolls and waits for content to appear. It’s slower than a static fetch, so don’t use it where you don’t need it.
Should I switch off Playwright entirely?
No. Keep Playwright for flows you run constantly against a site you know, where determinism matters and a silent data change would be expensive. Use Firecrawl for breadth — the long tail of sources where a bespoke scraper would never pay for itself.
What’s the biggest mistake people make with this?
Trusting the output without validation. Deterministic scrapers break loudly; agentic ones degrade quietly. Add row-count assertions, schema validation, and a diff against yesterday’s run before anything hits your RAG index or your CRM.
Go deeper than this article
This article covers the essentials. Our Technical & Coding eguide collection gives you the full step-by-step playbooks — prompts, workflows, and copy-paste recipes built for exactly this work.
