We're excited to announce Bichon v2.0, a major milestone that replaces the Fjall blob engine with bichon-blob — a purpose-built, log-structured storage engine designed specifically for email archiving workloads.
Why a New Storage Engine?
Fjall served us well through the v1.x series, but as our users scaled to hundreds of gigabytes and millions of messages, we encountered fundamental limitations:
- Compression: Fjall uses LZ4, which prioritizes speed over ratio. Email archives are highly compressible — plain text, HTML, and attachments with repetitive structures. Zstd delivers 2-3x better compression on real-world email corpora.
- Deduplication granularity: Fjall operates at the key-value level with no content-aware dedup. We needed BLAKE3 content hashing to avoid storing identical emails and attachments across accounts.
- Compaction overhead: LSM trees require periodic compaction that can stall writes during heavy ingestion. A log-structured design with online GC avoids this entirely.
What's New
bichon-blob Storage Engine
| Feature | v1.x (Fjall) | v2.0 (bichon-blob) |
|---|---|---|
| Compression | LZ4 | Zstd (3-22) |
| Dedup | None | BLAKE3 content hash |
| Structure | LSM tree | Log-structured segments (1GB) |
| Index | Built-in | redb-backed O(1) lookup |
| Recovery | WAL-based | Crash-safe append-only |
Performance Improvements
In our benchmarks with a 50GB mixed email corpus (200,000 messages across 5 accounts):
- Storage reduction: 50GB → 18GB (64% savings from Zstd + dedup)
- Ingestion throughput: 85 msg/s → 210 msg/s (2.5x faster)
- Search latency (P99): 450ms → 180ms (2.5x faster)
- Memory usage: 1.8GB → 900MB (50% reduction)
Migration from v1.x
We've made migration as smooth as possible. The bichon-admin tool includes a one-step blob migration:
./bichon-admin
# Select "Migrate v1.x Storage to v2.x"The migration is non-destructive — your v1.x data is never modified. Tantivy indexes and memdb metadata remain untouched. Only blobs are copied from Fjall to bichon-blob.
Note: The migration reads all existing blobs and re-compresses them with Zstd. Depending on your data size, this can take a while. Plan for ~10GB/hour on typical HDD storage.
What's Next
We're already working on:
- S3-compatible storage backend for cloud-native deployments
- MCP Server for LLM-powered email search and analysis
- Enterprise SSO (OIDC / SAML)
As always, feedback and contributions are welcome on GitHub and Discord.