Stateless Document Ingestion & Semantic Search API

ContextFlow AI is a robust, multi-tenant document ingestion utility built to abstract the engineering math and background queue management required to get clean, structurally sound document data into vector stores. Instead of spending weeks wrestling with file parsers, race conditions, and serverless execution timeouts, you can spin up a production-ready RAG pipeline with two API calls.

Architectural Core:

• Decoupled Processing Pipeline: Ingress routes are built on Next.js 16 to handle immediate request validation, pushing jobs to an Upstash Redis cluster. Heavy lifting is offloaded to standalone, persistent Node.js workers running BullMQ on Railway to guarantee large files never trip arbitrary serverless execution limits.

• 100% Stateless Pass Through Mode: Built specifically to solve corporate data privacy and GDPR compliance hurdles. By passing passthrough: true, the background workers stream files from Cloudflare R2, execute semantic text splitting, fetch OpenAI text-embedding-3-small arrays, fire the complete payload back to your app via a secure webhook, and instantly flush the server RAM. Zero retention, zero third-party database lock in.

Technical Capabilities:

• Dynamic, Structural Chunking: Leverages character windows that respect paragraph breaks, double newlines, and sentence boundaries rather than blindly slicing tokens mid word.

• Token Efficiency & Deduplication: Employs deterministic SHA-256 file fingerprinting to detect duplicate uploads instantly, preventing redundant embedding costs.

• Concurrent Quota Safeguards: Quotas are tracked in memory across batch ingestion routes (up to 20 files/URLs) to eliminate race conditions before they can hit your Postgres instance.

• Sub-500ms Vector Search: A dedicated /search endpoint executing cosine similarity over pgvector (Neon), optimized with Upstash Redis caching that limits API key validation database hits to once every 60 seconds.