What is Observyze?

Observyze is an open, high-throughput LLM Observability & Runtime Reliability engine engineered for production AI systems, agent runtimes, and multi-model infrastructure.

Most LLM logging tools operate as passive sinks that record traces post-execution. Observyze is built as an active reliability layer — combining sub-10ms asynchronous telemetry ingestion with inline runtime circuit breakers and local memory-bounded PII sanitization.

Key Technical Capabilities:

1. High-Throughput, Non-Blocking Ingestion (<10ms SLA)

• Zero-Block Telemetry: Telemetry payloads are accepted by a Fastify ingestion tier and staged into a Redis-backed durable write queue via atomic LPUSH. The API responds with 201 Created in under 10ms, decoupling client latency from database write locks.

• Batched Persistence & Deduplication: A decoupled background worker drains the queue in micro-batches, deduplicates traces by unique hash, and persists to MongoDB with compound time-series indexes.

• Client Buffer Resilience: The SDK maintains a local FIFO ring buffer (1000 items max) with exponential backoff retries (1s, 2s, 4s, 8s, 16s, 30s) to survive network partitions without leaking memory or dropping spans.

2. In-Memory Client-Side PII Sanitization

• Pre-Network Zero-Trust Masking: Pure, non-mutating sanitization executed locally in memory before payload transmission over TLS.

• Pattern Coverage: Vectorized pattern matchers covering emails, JWT signatures, Bearer tokens, AWS access keys (AKIA/ASIA), OpenAI/Anthropic API keys, US SSNs, credit cards (Luhn-compliant), E.164 phone numbers, and IP addresses.

• Key-Aware Redaction: 27 sensitive JSON dictionary keys (password, secret, api_key, private_key, cvv) are redacted with bounded recursion (depth <= 16) to guarantee zero stack-overflow risk on deep agent execution graphs.

3. Inline Autonomous Circuit Breakers

• Active Execution Interception: Wrap arbitrary async agent execution blocks to synchronously check semantic hallucination and safety thresholds with a 5-second abort controller.

• Configurable Fault Tolerance: Supports deterministic fail-closed (blocks execution if evaluation service is unreachable) and fail-open operational modes.

• Confidence-Weighted Gating: Evaluates confidence intervals (<0.40) to alert without blocking, preventing false-positive interruptions on ambiguous classifications.

4. Exact Token Accounting & Cost Engine

• Dynamic Pricing Table: Real-time per-token pricing sync for 40+ foundation models (OpenAI, Anthropic Claude 3.5, Google Gemini 2.5/1.5, Groq Llama 3.1, Mistral, DeepSeek V3/R1).

• Longest-Prefix Matching: Resolves versioned model strings (e.g. gpt-4o-2024-08-06, claude-3-5-sonnet-20241022) to exact input/output costs per span.

• Multi-Dimensional Attribution: Queries aggregate costs across organization, project, user ID, and session ID with linear regression burn-rate forecasting.

5. Universal Transparent Proxy Gateway

• Zero-Code Instrumentation: Drop-in reverse proxy compatible with standard OpenAI and Anthropic SDKs by simply overriding baseURL.

• Upstream Credential Vaulting: API keys are encrypted at rest using AES-256-GCM with authenticated data tags and dynamically injected upstream.

Engineering Stack:

• Core API & Gateway: Fastify, TypeScript, Node.js runtime, Zod validation schemas

• Ingestion & Streaming: Redis (LPUSH/RPOPLPUSH write queue + Pub/Sub), Server-Sent Events (SSE)

• Database & Storage: MongoDB with compound time-series indexes, AES-256-GCM encrypted vaults

• Evaluation & Cost Service: Python, FastAPI, LiteLLM pricing registry, HuggingFace NLI models

• Web Dashboard: Next.js 14 (App Router), TailwindCSS, Framer Motion, TanStack Query

Peerlist Community Access:

We are providing 90 Days of Free Early Access Pro Tier (100,000 monthly traces, in-memory PII scrubbing, and real-time guardrails) for engineering teams on Peerlist.

Documentation & Live App: https://observyze.com/docs