Design Rationale
Three decisions shape everything else in Bichon.
One statically-linked binary. The server, REST API, WebUI, full-text engine, and blob store all ship inside a single executable. Deployment is download and run; upgrading is replace the file. There is nothing to install alongside it — no PostgreSQL, no Elasticsearch, no Redis, no Node runtime.
Rust end to end. An email archive is a decade-scale workload: one process, running for years, ingesting and indexing continuously. Rust gives memory safety without a garbage collector — no GC pauses mid-index, predictable tail latency, and a memory footprint small enough for a NAS or a 2 GB VPS.
Purpose-built storage instead of a general-purpose database. Archiving has three distinct workloads — metadata lookups, full-text queries, and large-blob reads — with almost nothing in common. So each gets its own engine: memdb for metadata, Tantivy for search, bichon-blob for content. None of them is asked to do the other two's job.
How the Alternatives Store Email — and Where It Breaks
| Approach | What happens at scale |
|---|---|
| One EML file per message (typical legacy open-source archivers) | A million emails means a million inodes. Backups and antivirus scans crawl, filesystems degrade, and there is no search — "find that 2009 invoice" becomes a grep job. |
| Every message in PostgreSQL | Raw email bloats TOAST tables; VACUUM and backup times grow with the archive; full-text search via database extensions lags far behind a real search engine. You now operate a production database — migrations, tuning, replication — just to read old mail. |
| Multi-component stacks | Separate database, indexer, and web components to deploy, version-match, monitor, and back up. Every component adds a failure mode and an upgrade constraint; a 15-minute restore becomes a stack-rebuild exercise. |
| Bichon | One binary, three embedded storage layers, zero external dependencies. |
What This Buys You
- Deployment — copy a binary or pull a Docker image. No dependency matrix, no config sprawl across five services.
- Maintenance — nothing to patch around the archiver. Backups go straight to an S3-compatible store or a local directory natively; there is no separate database to back up correctly.
- Performance — 10M+ messages on a single instance: Tantivy serves millisecond search, BLAKE3 dedup means identical bodies and attachments are stored once, Zstd keeps TB-level archives compact.
- Durability — append-only segment files with crash-safe recovery and a WAL-backed metadata store; a power cut mid-write never corrupts the archive.
The rest of this page describes how those pieces actually fit together.
Workspace Crates
bichon/
├── crates/
│ ├── memdb/ Embedded key-value database layer (WAL, transactions)
│ ├── core/ Library — IMAP sync, search, storage, auth, models
│ ├── server/ Binary — Poem web server + embedded WebUI (rust-embed)
│ ├── cli/ Binary — bichon-cli import/export CLI
│ └── admin/ Binary — bichon-admin password reset & migration
└── web/ React + TypeScript + Vite + ShadCN UI frontendThree-Layer Storage
Request Layer
REST API (Poem) │ WebUI (React)
─────────────────────┼────────────────────
Storage Layer │
│
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ memdb │ │ Tantivy │ │ bichon-blob │
│ (metadata) │ │ (full-text) │ │ (blobs) │
│ │ │ │ │ │
│ • accounts │ │ • envelope │ │ • raw emails │
│ • users │ │ • attachment │ │ • attachments│
│ • roles │ │ • tags │ │ Zstd compr.│
│ • config │ │ • contacts │ │ │
│ • proxies │ │ Zstd compr.│ │ BLAKE3 hash │
└──────────────┘ └──────────────┘ └──────────────┘- memdb: Key-value metadata store. Houses accounts, users, roles, OAuth2 configs, proxy settings, and system configuration.
- Tantivy: Full-text search indices with Zstd compression support. Two separate indices: envelope (email metadata + body text) and attachment (file metadata + extracted text). Batch-committed every 1,000 documents or 60 seconds.
- bichon-blob: Zstd-compressed log-structured storage engine. Append-only segment files (1 GB each) with a redb-backed index for O(1) key lookup. Content-hash addressed (BLAKE3) with insert-time deduplication. Supports global dedup, online GC, and crash-safe recovery.
IMAP Download Pipeline
Schedule tick (every 10s)
│
▼
reconcile_mailboxes()
Compare local vs. remote
│
┌────┴────┐
▼ ▼
UID OK UID changed / new
(incremental) (full rebuild)
│ │
▼ ▼
fetch new fetch all
(max+1:*) (1:* batched)
│ │
└────┬────┘
▼
extract_envelope_and_store_it()
│
┌────┼────┐
▼ ▼ ▼
Tantivy bichon-blob memdb- Per-account background tasks managed by a global download-task singleton
- Concurrency controlled by semaphore (default:
num_cpus × 2) - Manual sync via
POST /api/v1/accounts/:id/start-download; cancel withcancel-download - Busy-check prevents overlapping manual and automatic syncs on the same account
Content Deduplication
Every ingested email is hashed with BLAKE3. Attachments are detached from the MIME tree, hashed independently (decoded content), and stored as raw undecoded bytes in bichon-blob. The email body is patched with hash-based placeholders and stored separately.
Both email and attachment blobs are deduplicated by content hash — identical content is never stored twice, regardless of which account or folder it arrives in.
A periodic index dedup task (every 12 hours) scans Tantivy for duplicate (account, mailbox, content_hash) tuples, keeps the most recently ingested copy, and cascade-deletes orphaned attachment entries so UID-based incremental sync remains accurate.
The original EML reconstructs byte-for-byte by swapping placeholders back with their attachment blobs.
See Content Deduplication for the full mechanism — hash computation, the ingest cache, the content-addressed blob store, and the periodic index cleanup task.