Running Kafka Inside Snowflake Before Snowflake Did: Part 3 of 4
The Real Solution
The Real Solution
Alright. This is the one you came for.
If Part 2 was the honest confession, Part 3 is where we actually solve the problem. And the first thing I want to tell you is this: what looks like one problem is actually two problems wearing a trench coat.
Miss that, and you will build something that appears to work, fails in a confusing way, and leaves you wondering what went wrong. I am going to make sure that does not happen to you.
Two Problems, Not One
When JP asked me to get an external producer talking to a Kafka broker inside SPCS, the instinct is to think: "I just need to expose a port." Open the right door, point the producer at it, done.
That instinct is half right. And half right, in this case, means it still does not work.
Here are the two problems you have to solve, at the same time, both of them:
Problem 1: Network reachability. There has to be an actual path from the external producer to the broker inside SPCS. By default, SPCS does not expose anything to the outside world. You have to explicitly declare a public endpoint in your service configuration. Without that, the broker is completely unreachable from outside. No door exists.
Problem 2: The advertised listener trap. This one is sneakier. Kafka has a specific behaviour: when a client connects to the broker for the first time (the bootstrap connection), the broker responds with metadata. That metadata includes the address the client should use for all future communication. This address is called the advertised listener.
In Demo 1, the advertised listener was localhost:9092. That is fine when the client is inside the same service, because localhost does resolve to the broker from there. But for an external client, localhost resolves to the client's own machine. Not the broker. The client's laptop. Which very definitely does not have a Kafka broker running on port 9092.
So what happens? The bootstrap connection actually succeeds. The client reaches the broker, gets the metadata back, and then tries to use that metadata to send messages. The address in the metadata says localhost:9092. The client dutifully connects to localhost:9092 on its own machine. Nothing is there. Every message send fails.
This is why the problem is so tricky. The first connection works. Then everything after it silently breaks. And the error you get is not "wrong address in advertised listener." It is something vague about connection timeouts, which sends you in completely the wrong direction.
The fix is not just opening a door. You also have to make sure the broker is telling the truth about its own address. The Golden Rule of Kafka networking: whatever Kafka advertises must be reachable from wherever the client is sitting.
Both problems. Same time.
The Wall We Hit First: SPCS is HTTPS-Only
Once I understood both problems, the next question was: what kind of endpoint can SPCS even give me?
SPCS public endpoints are HTTPS-only. Full stop.
HTTPS is great for web traffic. REST APIs, dashboards, anything that speaks the HTTP request/response pattern works perfectly through an SPCS public endpoint.
Kafka does not speak HTTP. It has its own wire protocol that runs over raw TCP. It is a persistent, binary, stateful connection. Not a request, not a response, not a URL. Just a stream of bytes over an open TCP socket.
You cannot route raw TCP through an HTTPS-only gateway. It is like trying to mail a package through a fax machine. The infrastructure physically does not support the thing you are trying to send through it.
So the naive approach of "expose port 9092 via a public SPCS endpoint and point the producer at it" does not work. The endpoint exists, but Kafka's protocol cannot travel through it.
This meant I needed to find a different path.
Three Options on the Table
I considered three architectural approaches for bridging this gap. Here is what each one actually means:
Option 1: Confluent REST Proxy.
Add a REST Proxy container inside the same SPCS service as the broker. The external producer speaks HTTPS to the REST Proxy, which is the thing with the public endpoint. The REST Proxy translates the incoming HTTP requests into Kafka's native binary protocol and forwards them to the broker over localhost:9092. The broker never sees HTTP. It only ever speaks its own language, which is fine because the REST Proxy is sitting right next to it on the same internal network.
Option 2: TCP-aware ingress.
Investigate whether SPCS has any mechanism for exposing raw TCP traffic, for example through a private link or some non-HTTPS connectivity option. This would let the external producer speak native Kafka protocol directly, without any translation layer.
Option 3: TCP-tunnel sidecar.
Run a sidecar container that maintains an authenticated tunnel between the external client and the broker. The external producer connects to the tunnel, the tunnel carries the traffic through to the broker. This adds a custom piece of infrastructure to manage, but it keeps the native protocol intact.
Each option has a different trade-off profile. Option 1 adds a translation layer but works cleanly over HTTPS. Option 2 would be ideal but depends on SPCS supporting something it may not. Option 3 gives you protocol purity at the cost of operational complexity.
Why REST Proxy Was the Right Choice
I went with Option 1, and here is the reasoning.
SPCS public endpoints are HTTPS-only. That is a hard constraint, not a soft preference. Option 2 (TCP-aware ingress) would require SPCS to support something it does not, which means it was off the table unless I found evidence of a workaround. I did not.
Option 3 (the tunnel sidecar) is genuinely interesting, but it means building and maintaining a custom piece of infrastructure that has no precedent in this environment. For a production system where you have security, reliability, and operational requirements, maybe that trade-off makes sense. For a demo where the goal is a clean, explainable architecture: it adds complexity that the solution does not need.
Option 1 fits the constraint neatly. SPCS can only do HTTPS. The REST Proxy speaks HTTPS on its public-facing side and Kafka's native protocol on its internal side. It is a purpose-built translation layer for exactly this situation. Confluent built and maintains it. It is well-documented. And because it sits inside the same SPCS service as the broker, the internal communication between the proxy and the broker stays on localhost, which is fast, simple, and requires no additional configuration.
The REST Proxy is not a workaround. It is the right tool for this constraint.
The Multi-Listener Pattern: Serving Two Clients at Once
With the REST Proxy handling the external path, I still needed to solve the advertised listener problem. And here is where it gets interesting, because the solution has to serve two different clients at the same time.
The consumer container is still inside SPCS, using localhost:9092. That works fine and I did not want to break it.
The external producer is now outside SPCS, using the REST Proxy's public HTTPS endpoint as its entry point. The REST Proxy connects to the broker on the internal network.
From the broker's perspective, it is receiving connections from two different directions: internal ones from the consumer and the REST Proxy (both on localhost), and the REST Proxy itself is acting as the protocol bridge for the external producer.
The key insight is that the REST Proxy handles the advertised listener problem on behalf of the external producer. The external producer never talks directly to Kafka. It only ever talks to the REST Proxy over HTTPS. The REST Proxy then connects to the broker internally, using localhost:9092, where the advertised listener is already correct.
This is why the architecture works cleanly. The REST Proxy insulates the external producer from Kafka's wire protocol and from the advertised listener issue entirely. The broker keeps advertising localhost:9092. The internal consumer keeps working exactly as before. And the external producer reaches the REST Proxy over HTTPS without needing to know anything about Kafka's internal addressing.
The two-listener problem is solved not by configuring two listeners on the broker, but by putting the REST Proxy in the middle and letting it handle the translation.
What the Final Architecture Looks Like
Here is the complete picture of Demo 2:
Inside SPCS, three containers share one service:
The Kafka broker, configured exactly as in Demo 1, advertising localhost:9092 for internal clients.
The consumer, unchanged from Demo 1, reading from localhost:9092 and writing to the Snowflake EVENTS table.
The Confluent REST Proxy, the new addition. It listens on port 8082 inside the service, connects to the broker on localhost:9092, and handles the translation between HTTP and Kafka's binary protocol.
The SPCS service spec now includes a public endpoint declaration pointing to the REST Proxy's port. Snowflake assigns a stable HTTPS URL to this endpoint.
Outside SPCS, on a developer laptop, runs a Python producer. Instead of using a native Kafka client library, it sends HTTP POST requests to the REST Proxy's public HTTPS URL. Each request carries the message payload in JSON format. The REST Proxy receives it, translates it into a Kafka produce request, and forwards it to the broker over localhost.
The broker stores the message in the events topic. The consumer picks it up, writes it to the Snowflake table. The Streamlit dashboard updates.
The full round trip: external laptop sends HTTPS request to REST Proxy endpoint > REST Proxy forwards to broker over localhost > consumer reads from broker > consumer writes to Snowflake > Streamlit dashboard reflects the new row.
Validating the Round Trip
"Demo complete" had a specific definition. It was not just "the pipeline runs." It was four conditions, all true at the same time:
A producer on a laptop, configured with the Snowflake-assigned REST Proxy URL, successfully publishes messages.
The in-SPCS consumer receives those messages and writes them to the EVENTS table.
The Streamlit dashboard shows the new rows arriving in real time.
The producer can be stopped and restarted from outside Snowflake without any change needed inside Snowflake.
All four conditions were met. The external producer sent messages. They showed up in the table. The dashboard updated. Stopping and restarting the producer from the laptop had no effect on the service running inside SPCS. It just picked back up when the producer reconnected.
That is the moment this went from an architecture on paper to a working system.
What This Actually Demonstrates
Step back for a second and look at what this architecture proves beyond just "Kafka works."
It proves that you can run a stateful, non-HTTP workload inside SPCS and make it reachable from outside using a protocol bridge pattern. The REST Proxy is one instance of that pattern, but the underlying principle applies more broadly: if your workload speaks a protocol that SPCS cannot expose directly, wrap it in something that does speak HTTPS, put that wrapper inside the same service, and expose the wrapper.
It proves that SPCS's shared-network-namespace property is genuinely useful. The fact that all containers in a service share localhost is what makes the internal communication between the REST Proxy and the broker so clean. No service discovery, no internal DNS, no additional networking configuration. They are just neighbours.
And it proves that the "two stacked problems" framing matters. If you had only solved the network reachability problem (exposed the endpoint) without thinking about the advertised listener, you would have a system that bootstraps and then fails silently. The REST Proxy architecture sidesteps the second problem entirely by making it so the external producer never talks to Kafka directly.
Both problems, solved together. That is the real solution.
Part 4 is the reflection: what Snowflake's Data Stream announcement actually means for all of this, what I learned from building it by hand, and a genuine thank-you to the person who pushed me into this problem in the first place.
See you there.
About Authors
Author: Shrinivas Vishnupurikar [ Snowflake Data Engineer at ArisData ]
Co-author: JayaPrakash (JP) Nellore [ Data and Analytics Principal & Co-Founder at ArisData ]
Co-author: Swami Addagalla [ Co-Founder at ArisData ]
Join Shrinivas on Peerlist!
Join amazing folks like Shrinivas and thousands of other builders on Peerlist.
0
0
0