Designed the data flows that pull production data out of the MES and deliver it to downstream systems, running on the NiFi cluster and central Kafka platform I set up.

• NiFi reads from the Oracle MES databases over CDC/JDBC and produces raw events into Kafka

• Spark Structured Streaming consumes the raw topics and writes cleaned, deduplicated gold topics for downstream apps

• Airflow workflows schedule the Spark jobs every 15 to 30 minutes, with retries and run history