Running Kafka Inside Snowflake Before Snowflake Did: Part 2 of 4
The Demo I Built by Accident
The Demo I Built by Accident
Let me tell you about the time I built something that technically worked but was still kind of wrong.
Not wrong in a "it crashed and burned" way. Wrong in a "you solved a different problem than the one you were asked to solve" way. Which, if you have spent any time in engineering, you know can feel just as bad.
This is the story of Demo 1: the accidental demo that taught me more than I expected, even if it did not hit the actual target.
What I Actually Built
The setup for this demo was straightforward, at least on paper.
I had three containers running inside a single SPCS service:
The Kafka broker. This is the engine of the whole thing. It receives messages, stores them in topics, and hands them out to consumers. I used Apache Kafka 3.8.0 in KRaft mode, which means no Zookeeper. Zookeeper was Kafka's old way of managing coordination between nodes. KRaft replaces it with a built-in mechanism, making the setup significantly simpler. One less moving part, which is always welcome.
The producer. A Python container that generates fake but realistic-looking events every second: logins, clicks, purchases, page views, logouts. Each event gets a unique ID, a timestamp, a random user ID, and a small payload. It sends these to the broker over port 9092.
The consumer. Another Python container that reads those events from the broker, and then writes each one as a row into a Snowflake table called EVENTS. It authenticates to Snowflake using an OAuth token that SPCS automatically mounts inside the container at a known file path. No hardcoded passwords, no manual credential management. Clean.
All three containers live inside the same SPCS service, which means they share the same internal network. They all talk to each other over localhost. The broker is at localhost:9092, the producer sends to localhost:9092, the consumer reads from localhost:9092.
On top of that, I wired up a Streamlit dashboard in Snowflake that queries the EVENTS table every few seconds and shows you a live bar chart of event types plus the last ten rows as they come in. Watching events land in real time while Kafka is running inside Snowflake felt genuinely satisfying the first time I saw it work.
The data flow looked like this:
Producer container generates event > sends to Kafka broker > consumer picks it up > consumer writes to Snowflake table > Streamlit dashboard updates.
Everything inside SPCS. No traffic leaving the boundary at any point.
Why JP Pointed Out It Was Wrong
When I showed JP what I had built, his response was essentially: "This is good work, but the producer is in the wrong place."
And he was right.
Think about what Kafka is actually used for in the real world. It sits in the middle of a larger system. Producers are the applications, services, or devices that generate data: a web app tracking user behaviour, an IoT sensor reporting temperature, a payment service emitting transaction events. These producers exist outside Kafka, and almost always outside whatever infrastructure Kafka is running on.
In my demo, the producer was co-located with the broker inside the same SPCS service. They were neighbours. Of course they could talk to each other. That is not a meaningful test of anything.
What JP needed to see was: an external system, running completely outside of SPCS, reaching the broker and successfully publishing messages. That is the real-world scenario. That is what proves the architecture is actually usable.
My demo proved that Kafka can run inside SPCS. That is not nothing. But it did not prove that anything outside can reach it.
So the closed-loop demo, while functional, was a bit like testing whether a bridge holds weight by putting the car and the other side of the bridge in the same room. Technically, the car crossed. But it did not cross anything.
Why the Demo Was Still Worth Building
Here is the thing though: this was not wasted work. Not even close.
It proved the baseline. Before anything else, we needed to know that Kafka could run inside SPCS at all. Container services have limits, quirks, and constraints. Getting a Kafka broker to start, stay running, and accept connections inside SPCS was a real question mark at the start. This demo answered it with a working system.
It established the foundation. The broker configuration, the Python consumer, the OAuth token authentication pattern, the Snowflake table setup, the Streamlit dashboard: all of that carried forward into Demo 2 unchanged. The second demo was built on top of the first, not from scratch.
It produced the ARM versus AMD64 lesson. This one is important enough to get its own section, coming right up.
Productive detours exist in engineering. Sometimes you build the wrong thing and it teaches you exactly what you need to know to build the right thing. This was one of those times.
The ARM vs. AMD64 Lesson (Do Not Skip This)
This is the part I want every engineer reading this to absorb before they ever touch SPCS. It will save you a frustrating afternoon.
My development machine is an Apple MacBook Pro with an M1 chip. The M1 is ARM-based, which means it uses a different processor architecture than the Intel and AMD chips that have traditionally powered servers and cloud infrastructure. ARM is arm64. Most cloud compute is amd64 (also called x86_64).
When you build a Docker image on your local machine, Docker builds it for the architecture of the machine you are on. On an M1 Mac, that means arm64 by default.
Snowpark Container Services runs on amd64 compute. It does not support arm64 images.
Here is what happens if you build on an M1 Mac and push without specifying the platform: the image looks perfectly fine. It pushes without errors. You deploy your service in SPCS. The container starts. And then it just... fails. It either hangs in a pending state or crashes with an error message that says exec format error, which is not exactly the most descriptive thing you have ever read.
The fix is one flag added to your build command:
docker build --platform linux/amd64 -t your-image-name .That flag tells Docker to build for amd64 even though you are on an arm64 machine. Docker emulates the target architecture during the build. It is slower than a native build, especially the first time, because Docker has to translate every instruction. But it produces an image that actually runs in SPCS.
If you are on a Mac with an M1, M2, or M3 chip and you are building images for SPCS: every single image needs that flag. The official Kafka image you pull from Docker Hub needs it too, at the pull step:
docker pull --platform linux/amd64 apache/kafka:3.8.0Miss it once and you will spend a while staring at logs wondering why a container that worked perfectly on your laptop refuses to start in the cloud.
Consider this your early warning. You are welcome.
What This Demo Proved and What It Did Not
To be clear about where things stood after Demo 1:
It proved that Kafka can run inside SPCS in a stable, functional configuration. It proved that a consumer running inside SPCS can authenticate to Snowflake and write data to a table. It proved that the OAuth token mount works cleanly. It proved the full internal pipeline from event generation to Snowflake table to Streamlit dashboard.
It did not prove that anything outside SPCS can reach the broker. The service spec had no public endpoint declared. The broker advertised localhost:9092 as its address, which is meaningless to any client outside the service. There was no path in, and no mechanism to create one, in this version of the architecture.
Demo 1 closed the internal loop. Demo 2 needed to open it.
That is exactly what Part 3 is about: the two stacked problems you have to solve to make external access work, the three options on the table, and the architecture that finally made the round trip real.
See you there.
About Authors
Author: Shrinivas Vishnupurikar [ Snowflake Data Engineer at ArisData ]
Co-author: JayaPrakash (JP) Nellore [ Data and Analytics Principal & Co-Founder at ArisData ]
Co-author: Swami Addagalla [ Co-Founder at ArisData ]
Join Shrinivas on Peerlist!
Join amazing folks like Shrinivas and thousands of other builders on Peerlist.
0
0
0