Oct 23 • 🎙️ Share
Storing Data Efficiently with JS
As I’m building Gextron, one of the most expensive operations is generating a full option chain.
I actually wrote about the bottleneck here:
Moving this work to Node workers helped a lot — but there was still another hidden cost: how data was stored.
So I completely changed the model of how Redis is used inside Gextron.
🧩 Before — Redis as a Key-Value JSON Cache
I was just storing option-chain data as raw JSON strings. [IMG 1]
Redis stored this as plain UTF-8 text, which came with heavy costs:
- 🔸 Lots of heap allocation
- 🔸 CPU-heavy serialization (JSON.stringify / JSON.parse)
- 🔸 Huge Redis memory usage (3 – 5 MB per chain)
- 🔸 Slow iteration and high GC pressure
⚡ Now — Redis as a Binary Data Store (Columnar Cache)
Now the data is serialized into typed arrays (Float32Array, Uint32Array) and packed into one binary block, then Brotli-compressed. [IMG 3]
Redis now stores a compact numeric binary snapshot, not text.
This change alone brought massive wins:
- ⚙️ Pure numbers, no strings
- 💾 ~10× smaller payloads (≈ 300 KB vs 3 MB)
- 🚀 Fast transfers
- ⚡ Decompression 2–4 ms vs 20–40 ms for JSON.parse()
🧠 Lets breakdown a simple analogy:
📚 Bookshelf
- JSON: Thousands of loose pages (repeated titles, redundant text)
- Columnar: One clean binder with tabbed sections
🧮 CPU Access
- JSON: Skims ever page to find a number
- Columnar: Reads one tight stack of numbers
📦 Compression
- JSON: Random text
- Columnar: Predictable patterns (tiny)
🧰 The Takeaway
By switching from JSON blobs to binary columnar storage, Redis went from being a text cache to an in-memory analytics engine. These are the same principles behind systems like Parquet and ClickHouse
4
1 comment
Storing Data Efficiently with JS
skool.com/javascript
Chat about javascript and javascript related projects. Yes, typescript counts.
Leaderboard (30-day)
1
+20
2
+6
3
+2
4
+2
5
+2
Powered by