DevOps & cloud engineer. CKA, CKAD. Author of DebugBox, a Kubernetes debugging container in three size-scoped variants.

devto

github

Muhammad Ibtisam Iqbal • Peerlist

<p>SilverStack is an open, ongoing project of mine: complete, preconfigured machines packaged as custom OCI rootfs images and published as playgrounds anyone can launch in a browser on iximiuz Labs, which boots each one as a Firecracker microVM. Seven images so far, five live as playgrounds. systemd runs as PID 1 in every one, and machines serving a public URL reach it through an outbound-only Cloudflare Tunnel from behind CGNAT, no inbound port ever opened. Each image is built and pushed to GHCR by its own GitHub Actions workflow, and every service image must pass a build-time health check first. The current set is developer and CI/CD focused because that was needed first.</p><h3>What is published</h3><ul><li><p><strong>Dev Machine</strong>: a fully provisioned Ubuntu 24.04 DevOps workstation, every tool pre-installed and every alias pre-wired. My primary daily machine.</p></li><li><p><strong>Jenkins Server</strong>: Jenkins LTS, systemd-booted, Nginx-proxied, Cloudflare-ready.</p></li><li><p><strong>SonarQube Server</strong>: SonarQube 26.2 CE (LTA), PostgreSQL-backed.</p></li><li><p><strong>Nexus Server</strong>: Nexus Repository Manager 3 Community Edition.</p></li><li><p><strong>CI/CD Stack</strong>: all three services plus a jump host, four machines in one playground on a shared private network.</p></li></ul><p>Two further images exist: the shared ubuntu-24-04 base, and an Apache Guacamole desktop (MariaDB, guacd, Tomcat 10, XRDP, Nginx) for browser-based RDP. Guacamole is built and CI-published, not yet released as a playground.</p><h3>Why a custom rootfs</h3><p>iximiuz Labs runs playgrounds as full microVMs, not containers, booting from a root filesystem mounted at /. The default approach runs install scripts on every start, so every visitor waits. A custom rootfs removes that wait: everything is already in the image, pushed to a public OCI registry and referenced as the VM drive source. Zero install time, instant prompt.</p><h3>Both are an OCI image. Only one is a machine.</h3><p>A container borrows the host's kernel and gets one process started for it. A microVM is handed a kernel and has to bring itself up from there. A playground root drive has to be linux/amd64, carry no kernel, ship an init system, run sshd on 0.0.0.0:22 at boot, and already contain its users. No CMD or ENTRYPOINT is read at all, since the platform boots /lib/systemd/systemd itself.</p><p>Services are enabled at build time, never started: no init system runs during docker build, so systemctl enable writes a symlink into multi-user.target.wants/ that systemd reads on the next real boot.</p><p>/etc/machine-id is emptied so each VM generates its own systemd identity. SSH host keys are deleted too: left in, every VM booted from that image would share one identity. iximiuz Labs regenerates host keys at first boot anyway, so this is defence in depth here, but not optional on my own Firecracker or Cloud Hypervisor.</p><p>/run is tmpfs and wiped on every reboot, so a lab-init oneshot rebuilds what sshd and nginx need on each boot rather than once at instance creation. That same unit does the runtime work an image should not carry: creating PostgreSQL and MariaDB databases, importing schemas, injecting credentials, and tuning kernel parameters such as vm.max_map_count for SonarQube's Elasticsearch. Every image takes Docker build arguments, so usernames, ports and domains are set at build time.</p><h3>No public IP, no inbound ports</h3><p>The machines sit behind Carrier-Grade NAT, by design rather than as a flaw to work around. A service needing a persistent custom domain with SSL gets two components baked in: Nginx, reverse-proxying port 80 to the service's internal port, and cloudflared, opening an outbound tunnel at boot with automatic SSL. Images with no public URL carry neither. Nginx also runs a /health endpoint and hardened security headers.</p><h3>Publishing and access</h3><p>GHCR is a requirement, not a preference: the drive image must be publicly pullable and cannot live on Docker Hub, which rate-limits pulls. A manifest pointing at Docker Hub, or a private image, fails at playground start rather than at build.</p><p>Every service image runs a build-time health check before it can be pushed: nginx -t parses the real config, systemd units are verified by their multi-user.target.wants symlink, and required binaries and directories are asserted present. A nonzero exit fails the build.</p><p>Access is least-privilege throughout: strict sudoers profiles, daemon user isolation, ephemeral SSH keys per microVM instance, and third-party binaries pinned to specific versions and verified by SHA256 checksum.</p><h3>Interactive tutorial</h3><p>A tutorial on iximiuz Labs builds a container image and a bootable rootfs from the same Dockerfile: where the two diverge, what a plain ubuntu:24.04 is missing, and how to prove an image would boot without booting it.</p><h3>Links</h3><ul><li><p><strong>Images</strong>: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/ibtisam-iq/silver-stack/tree/main/iximiuz/rootfs">https://github.com/ibtisam-iq/silver-stack/tree/main/iximiuz/rootfs</a></p></li><li><p><strong>Playgrounds</strong>: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://labs.iximiuz.com/a/ibtisam-iq">https://labs.iximiuz.com/a/ibtisam-iq</a></p></li><li><p><strong>Build notes</strong>: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://runbook.ibtisam-iq.com/self-hosted/ci-cd/iximiuz/">https://runbook.ibtisam-iq.com/self-hosted/ci-cd/iximiuz/</a></p></li><li><p><strong>Tutorial</strong>: <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://labs.iximiuz.com/tutorials/build-custom-rootfs-image-for-microvm-playgrounds-d1d0eba9">https://labs.iximiuz.com/tutorials/build-custom-rootfs-image-for-microvm-playgrounds-d1d0eba9</a></p></li></ul>

SilverStack

<p>DebugBox is an open-source Kubernetes diagnostics toolkit: a debugging container image in three size-scoped Alpine variants, 15 MB to 91 MB, the smallest 93% smaller than netshoot. Every variant builds from a digest-pinned base, ships multi-arch for amd64 and arm64, and reaches a registry only after Hadolint linting, in-container smoke tests, and Trivy gating that fails the job on HIGH or CRITICAL. Documented on a versioned MkDocs site, MIT licensed, 22 tags per release.</p><h3><strong>The problem</strong></h3><p>You need to debug a pod. You run <em>kubectl debug my-pod --image=netshoot</em> and wait for 202 MB to download. On an edge cluster or a metered connection, every megabyte costs time. And you only needed to check DNS. netshoot is all or nothing.</p><h3><strong>Three variants</strong></h3><ol><li><p><strong>Lite (15 MB)</strong>: curl, dig, nslookup, netcat, iproute2, iputils, jq, yq, vi. DNS and connectivity.</p></li><li><p><strong>Balanced (47 MB)</strong>: adds tcpdump, socat, mtr, openssl, strace, lsof, htop, git, vim, kubectx, kubens. The daily driver.</p></li><li><p><strong>Power (91 MB)</strong>: adds tshark, nmap, ngrep, iperf3, ethtool, iftop, iptables, nftables, conntrack-tools, ltrace. Packet analysis and forensics.</p></li></ol><p>Lite is roughly 13x smaller than netshoot. Even Power is 111 MB smaller, a 55% reduction.</p><p><em>kubectl debug my-pod -it --image=ghcr.io/ibtisam-iq/debugbox</em></p><p><em>kubectl run debug --rm -it --image=ghcr.io/ibtisam-iq/debugbox --restart=Never</em></p><p><em>docker run -it ghcr.io/ibtisam-iq/debugbox</em></p><p>kubectl run works on any Kubernetes version. kubectl debug requires 1.23 or later.</p><h3>What it is not for</h3><p>Not for persistent sidecars, production workloads, or control plane access. It runs as root and is meant for ephemeral debugging only.</p><h3>Shell helpers</h3><ul><li><p>json(), yaml() and ll() ship in every variant. </p></li><li><p>Balanced and Power add ports(), connections(), routes(), k8s-info(), sniff(), sniff-http(), sniff-dns() and cert-check(). </p></li><li><p>Power also adds conntrack-watch().</p></li></ul><h3>Container design</h3><p>Dockerfiles stay declarative and installation stays procedural. Package lists live in <em>.packages </em>files, and complex installs such as yq, with architecture detection and SHA256 verification, get dedicated scripts. Every tool version is pinned, and each variant builds from its base by SHA256 digest rather than by tag, so a rebuilt base cannot silently change what ships.</p><h3>Nothing reaches a registry until it has passed</h3><p>Every commit builds 6 images: 3 variants across 2 architectures, using Docker Buildx and QEMU. Each is loaded into the local Docker daemon. None are pushed.</p><p>Then each one has to earn it. Compressed size is measured and printed in the job log, because size is the entire point of this project and a regression should be visible before release rather than in a user's slow pull. A smoke test runs inside the container rather than against it, verifying it works as a runtime: identity resolves, /tmp is writable, procfs is mounted, the network stack answers, core binaries resolve on PATH. Trivy scans with HIGH and CRITICAL set to exit code 1, and the only exceptions are tracked, with reasons, in a committed <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/ibtisam-iq/debugbox/blob/main/.trivyignore">.trivyignore</a>. Hadolint lints every Dockerfile.</p><p>Publishing is a separate workflow, triggered only by a version tag, that scans again before it builds and pushes multi-arch images (amd64 and arm64) to GHCR and Docker Hub with OCI metadata labels injected at build time. Tags come in two forms: floating per variant, and pinned semantic versions for production use.</p><p>A registry should be a record of things that already passed, not the place you find out.</p><h3>Documentation with one source of truth</h3><p>The docs started with the same fact written down in too many places. One image's compressed size appeared in 10 separate files, and a runnable pull command in 16. Then the image changed, and correctness depended on finding all of them.</p><p>Now every fact has exactly one home and everything else links to it. The tool inventory lives in a single <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/ibtisam-iq/debugbox/blob/main/docs/manifest.yaml">manifest.yaml</a> and renders into one <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://debugbox.ibtisam-iq.com/latest/reference/manifest/">reference page</a>, so adding a tool is a one line change. An entire installation page turned out to be a longer restatement of the quick start, so it stopped existing. That cleanup added 117 lines and removed 474.</p><p>Two checks run on every push: a strict docs build that fails on a broken internal reference, and a link checker that crawls every rendered page and fails on dead external URLs.</p><h3>Interactive tutorial</h3><p>A hands-on tutorial on <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://labs.iximiuz.com/a/ibtisam-iq">iximiuz Labs</a> walks all three variants end to end in a live multi-node Kubernetes playground, no local cluster required.</p><p>Published with <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/ibtisam-iq/debugbox/blob/main/CONTRIBUTING.md">CONTRIBUTING.md</a>, <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/ibtisam-iq/debugbox/blob/main/CODE_OF_CONDUCT.md">CODE_OF_CONDUCT.md</a> and <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/ibtisam-iq/debugbox/blob/main/SECURITY.md">SECURITY.md</a>. One command runs the whole quality loop locally: make check does lint, build, test and scan in that order.</p><h3>Links</h3><ul><li><p><strong>Docs:</strong> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://debugbox.ibtisam-iq.com">https://debugbox.ibtisam-iq.com</a></p></li><li><p><strong>Repo:</strong> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://github.com/ibtisam-iq/debugbox">https://github.com/ibtisam-iq/debugbox</a></p></li><li><p><strong>Tutorial:</strong> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://labs.iximiuz.com/tutorials/kubernetes-debugging-with-debugbox-74e481c8">https://labs.iximiuz.com/tutorials/kubernetes-debugging-with-debugbox-74e481c8</a></p></li><li><p><strong>Docker Hub:</strong> <a target="_blank" rel="noopener noreferrer nofollow ugc" class="break-all link-break text-gray-500 hover:underline" href="https://hub.docker.com/r/mibtisam/debugbox">https://hub.docker.com/r/mibtisam/debugbox</a></p></li></ul>