I built a persistent key-value storage engine in Go based on Bitcask, the log-structured storage system used in Riak. I wanted to understand how databases actually work at a low level, the tradeoffs between performance and durability, how crash recovery works and why different storage engines exist for different workloads.

Why I Did This:

Bitcask has this beautifully simple design, just append everything to a log file and keep an in-memory index. No fancy B-trees, no complicated balancing algorithms. It's literally just "write goes brrr" and you maintain a HashMap to find stuff. Perfect for actually learning instead of drowning in theory.

How I Built It:

The basic idea:

• Writes just append to a file (super fast, sequential I/O)

• In-memory HashMap tells you where each key lives

• Reads are just HashMap lookup -> seek to file offset -> read value

• Eventually you compact old files to reclaim space

The interesting parts:

1. Different write modes:

I implemented three modes because I wanted to understand the tradeoffs everyone talks about:

• Async writes: Basically YOLO mode. Writes go to a buffer in memory and flush whenever. Got like 10k+ writes/sec which was honestly satisfying to watch. But yeah, crash and you lose recent writes. Not ideal for your bank account balance.

• Batch sync: The "let's be reasonable" option. Buffer a bunch of writes, then call `fsync()` once. You get decent throughput (5-6k writes/sec) and only risk losing the current batch if things go sideways.

• Direct sync: Full paranoia mode. Every single write calls `fsync()` immediately. Super safe but painfully slow. This is what you'd use for data you absolutely cannot lose.

Implementing these really drove home why database docs are always like "configure your durability settings based on your use case" - the performance difference is wild.

2. Checksums because data corruption is real:

Added CRC32 checksums to every record because apparently disks lie to you sometimes? Each entry is [checksum][timestamp][key_size][value_size][key][value] and I verify the checksum on every read and during recovery.

3. Crash recovery:

When you start up, you rebuild the in-memory HashMap by scanning all the data files. But here's the thing - you have to validate checksums as you go because the last file might have partial writes from a crash.

4. Compaction (aka garbage collection for databases):

Since you never update in place, old values just pile up. I wrote a background process that goes through old files, keeps only the latest value for each key and writes them to new files. Getting this right was lowkey the hardest part.

What I Learned:

The performance differences between write modes were eye-opening. Async mode was flying while direct sync was struggling at a few thousands. You really see why databases let you tune this stuff.

Also, fsync() is expensive in ways I didn't fully appreciate. And checksums aren't just paranoia - they're necessary because hardware fails in weird ways.

Plus it's just deeply satisfying to write db.Put("key", "value") and know exactly what's happening - the bytes being written to disk, the HashMap being updated, whether fsync() is being called.