Backend & AI Engineer | Built AegisFlow (LLM compliance proxy) | VoiceToNotes backend (150K users)

github

Jhanvi Nagori • Peerlist

<h3>AegisFlow: High-throughput, low-latency distributed compliance proxy</h3>AegisFlow acts as a crucial intermediary between internal corporate applications and third-party Large Language Model (LLM) providers. It intercepts outgoing prompt traffic, locally strips or masks Personally Identifiable Information (PII) in real-time using lightweight Machine Learning (ML) models, and then forwards the sanitized data to the cloud LLM. Upon receiving the LLM's response, AegisFlow transparently rehydrates it before returning it to the user. To ensure SOC 2 compliance, all transactional metadata is archived via an asynchronous Kafka pipeline.<ul><li>Idempotency: Ensures exactly-once processing through an Idempotency-Key header and Redis tracking.</li><li>PII Masking Pipeline: Utilizes ML models (Presidio and Hugging Face NER) for PII detection and masking, with fallback to regex-based methods if ML inference fails.</li><li>Async Audit Trail: Logs all transactions to Kafka and then to PostgreSQL for auditable records, meeting SOC 2 requirements.</li><li>Observability: Integrated with Prometheus and Grafana for comprehensive monitoring of latency, throughput, error rates, and system health.</li><li>High Performance: Designed for low-latency and high-throughput, with benchmarks demonstrating sub-25ms internal overhead and the capability to handle thousands of requests per second.</li><li>Flexible Deployment: Supports local development with Docker Compose and production deployments on AWS using managed services like ECS, MSK, ElastiCache, and RDS.</li></ul>