Google Cloud Platform Big Data and Machine Learning Fundamentals

Kafka Essential Training

AWS Concepts online course

Advanced SQL for Data Scientists

high secondary school

post graduation diploma

Bachelor of Engineering - BE

- Built high-volume production data pipelines processing streams from 1,000+ connected sources — owned ingestion from Oracle and MySQL into HDFS with schema validation and deduplication at source using Hadoop, Hive, and Sqoop
- Optimised distributed query performance by 35% through strategic partitioning, bucketing, and cluster tuning, same performance engineering principles applied in modern Spark/Databricks workloads
- Designed data flows supporting predictive maintenance AI models — engineered feature-rich datasets enabling early-failure detection and real-time operational monitoring dashboards for engineering stakeholders
- Implemented pipeline observability: completeness, freshness, and referential integrity checks across all ingestion jobs; established SLA monitoring for operational analytics consumers

Big Data Engineer

Joined Real Madrid's business data team as an AI engineering intern — not a passive academic placement. Owned the problem end-to-end: understood the business challenge, designed the architecture, and shipped ClubOS — a standalone AI SaaS tool delivered directly to senior club stakeholders.

✦ Identified core business problem: disconnected social media signals, commercial performance data, and match data with no unified decision-support layer for non-technical leadership
✦ Built ClubOS end-to-end as AI engineer — integrated YouTube, Instagram, historical commercial data, and match performance into a single platform; LLM-generated scenario narratives delivered via a React-based simulator
✦ Designed and implemented the full data layer: ingestion pipelines, domain data products, Gold-layer transformation on Databricks, and retrieval architecture for LLM context
✦ Delivered to senior club stakeholders operating to professional football department standards — not a prototype, a working tool 

AI Engineer (Internship) | Real Madrid

- Architected cloud data lakehouse on AWS (S3, Glue, EMR, Lake Formation) using Apache Iceberg — multi-zone design (raw → curated → semantic) serving 6+ cross-functional teams; column-level security via Lake Formation
- Designed semantic data layer on AWS and GCP — star schemas, wide tables, aggregation layers for QuickSight and Power BI; dbt-governed transformation with full lineage tracking
- Implemented CDC ingestion using Kafka and AWS Glue Streaming — replaced daily batch loads with sub-hour SLA delivery across critical reporting domains
- Built domain-oriented data products aligned with data mesh principles — defined data contracts, domain ownership, SLAs, and cataloguing standards
- Led Databricks/PySpark transformation workloads on Delta Lake — optimised cluster configs and partition strategies; reduced job execution time 40%
- Deployed RAG pipeline using Amazon Bedrock for internal knowledge Q&A — prompt engineering, retrieval architecture, guardrails; first LLM system in prod

Senior Data Engineer

Senior Data Engineer → AI Engineer|Built AI SAAS@RealMadrid Python,LLm,RAG,Agentic AI, Databricks Ex-@Deloitte@Honewell

github

Know about Divyansh Shrivastava's work experience, education, bootcamps, certifications, and much more. Divyansh Shrivastava