Black Sparrow is an open-source, local-first website crawler and technical SEO audit engine I built in Rust for developers, SEO professionals, and AI agents.

I started the project to explore how a modern crawler could remain fast, private, and automation-friendly without requiring cloud accounts or heavyweight infrastructure. Its asynchronous engine uses Tokio for concurrency, streams HTML through lol_html, adapts crawl speed with an AIMD congestion controller, persists audit history in SQLite, and builds website link graphs using petgraph.

Black Sparrow currently performs 120 technical checks covering HTTP behavior, metadata, headings, canonicalization, indexing directives, internal links, structured data, security headers, hreflang, and AI-search readiness. Post-crawl analysis detects issues such as orphan pages and canonical loops while calculating internal PageRank across the site.

Audit results can be exported as interactive HTML reports, Screaming Frog-compatible CSV files, Markdown for language models, structured JSON, or terminal output.

A native Model Context Protocol server allows AI coding agents such as Cursor, Claude Desktop, and Windsurf to start audits, monitor crawl progress, inspect individual pages, and query findings directly. The project is distributed as a cross-platform command-line tool under the MIT and Apache 2.0 licenses.