Software Engineer & Builder. Creator of Observyze.

devto

github

Rahul singh • Peerlist

<h3>What is Observyze?</h3>Observyze is an open, high-throughput LLM Observability &amp; Runtime Reliability engine engineered for production AI systems, agent runtimes, and multi-model infrastructure.Most LLM logging tools operate as passive sinks that record traces post-execution. Observyze is built as an active reliability layer — combining sub-10ms asynchronous telemetry ingestion with inline runtime circuit breakers and local memory-bounded PII sanitization.<h3>Key Technical Capabilities:</h3><h3>1. High-Throughput, Non-Blocking Ingestion (&lt;10ms SLA)</h3><ul><li>Zero-Block Telemetry: Telemetry payloads are accepted by a Fastify ingestion tier and staged into a Redis-backed durable write queue via atomic LPUSH. The API responds with 201 Created in under 10ms, decoupling client latency from database write locks.</li><li>Batched Persistence &amp; Deduplication: A decoupled background worker drains the queue in micro-batches, deduplicates traces by unique hash, and persists to MongoDB with compound time-series indexes.</li><li>Client Buffer Resilience: The SDK maintains a local FIFO ring buffer (1000 items max) with exponential backoff retries (1s, 2s, 4s, 8s, 16s, 30s) to survive network partitions without leaking memory or dropping spans.</li></ul><h3>2. In-Memory Client-Side PII Sanitization</h3><ul><li>Pre-Network Zero-Trust Masking: Pure, non-mutating sanitization executed locally in memory before payload transmission over TLS.</li><li>Pattern Coverage: Vectorized pattern matchers covering emails, JWT signatures, Bearer tokens, AWS access keys (AKIA/ASIA), OpenAI/Anthropic API keys, US SSNs, credit cards (Luhn-compliant), E.164 phone numbers, and IP addresses.</li><li>Key-Aware Redaction: 27 sensitive JSON dictionary keys (password, secret, api_key, private_key, cvv) are redacted with bounded recursion (depth &lt;= 16) to guarantee zero stack-overflow risk on deep agent execution graphs.</li></ul><h3>3. Inline Autonomous Circuit Breakers</h3><ul><li>Active Execution Interception: Wrap arbitrary async agent execution blocks to synchronously check semantic hallucination and safety thresholds with a 5-second abort controller.</li><li>Configurable Fault Tolerance: Supports deterministic fail-closed (blocks execution if evaluation service is unreachable) and fail-open operational modes.</li><li>Confidence-Weighted Gating: Evaluates confidence intervals (&lt;0.40) to alert without blocking, preventing false-positive interruptions on ambiguous classifications.</li></ul><h3>4. Exact Token Accounting &amp; Cost Engine</h3><ul><li>Dynamic Pricing Table: Real-time per-token pricing sync for 40+ foundation models (OpenAI, Anthropic Claude 3.5, Google Gemini 2.5/1.5, Groq Llama 3.1, Mistral, DeepSeek V3/R1).</li><li>Longest-Prefix Matching: Resolves versioned model strings (e.g. gpt-4o-2024-08-06, claude-3-5-sonnet-20241022) to exact input/output costs per span.</li><li>Multi-Dimensional Attribution: Queries aggregate costs across organization, project, user ID, and session ID with linear regression burn-rate forecasting.</li></ul>5. Universal Transparent Proxy Gateway<ul><li>Zero-Code Instrumentation: Drop-in reverse proxy compatible with standard OpenAI and Anthropic SDKs by simply overriding baseURL.</li><li>Upstream Credential Vaulting: API keys are encrypted at rest using AES-256-GCM with authenticated data tags and dynamically injected upstream.</li></ul>Engineering Stack:<ul><li>Core API &amp; Gateway: Fastify, TypeScript, Node.js runtime, Zod validation schemas</li><li>Ingestion &amp; Streaming: Redis (LPUSH/RPOPLPUSH write queue + Pub/Sub), Server-Sent Events (SSE)</li><li>Database &amp; Storage: MongoDB with compound time-series indexes, AES-256-GCM encrypted vaults</li><li>Evaluation &amp; Cost Service: Python, FastAPI, LiteLLM pricing registry, HuggingFace NLI models</li><li>Web Dashboard: Next.js 14 (App Router), TailwindCSS, Framer Motion, TanStack Query</li></ul>Peerlist Community Access:We are providing 90 Days of Free Early Access Pro Tier (100,000 monthly traces, in-memory PII scrubbing, and real-time guardrails) for engineering teams on Peerlist.Documentation &amp; Live App: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://observyze.com/docs">https://observyze.com/docs</a>