Every email Bichon ingests is hashed, and identical content is stored only once. Deduplication happens at three active layers (ingest cache, blob store, index cleanup) plus a sync layer that avoids downloading the same message twice in the first place. Each layer has a different key, scope, and strictness — this page explains what each one does and how they interact.

                    ┌────────────────────────────────────┐
  Raw EML arrives   │ L1  Ingest cache (memory)         │  hit ───► skipped entirely
  IMAP / EML / PST  │     key: (account, mailbox,       │          (no parse, no index, no blob)
  / SMTP ingest     │          BLAKE3(raw EML))         │
                    └──────────────┬─────────────────────┘
                                   │ miss
                                   ▼
                    ┌────────────────────────────────────┐
                    │ L2  bichon-blob (global)            │  hash exists ───► no write
                    │     key: BLAKE3 → 32-byte address   │
                    └──────────────┬─────────────────────┘
                                   ▼
                    ┌────────────────────────────────────┐
                    │ L3  Index dedup task (every 12 h)  │
                    │     backstop: removes duplicates   │
                    │     that slipped past L1           │
                    └────────────────────────────────────┘

  (L4 — sync layer: UID / UIDVALIDITY tracking avoids re-downloading
   messages that are already archived)

What Gets HashedCopy Anchor Link

All hashes are BLAKE3, computed over raw bytes:

• Email content hash — computed over the complete original EML bytes, exactly as received. It is not derived from the Message-ID header (which is missing or forged in spam) nor from the parsed body.
• Attachment hash — computed over the decoded attachment content, while the blob store keeps the raw undecoded MIME bytes. Two attachments that decode to the same content share one blob, even if their transfer encodings differ.

After hashing, attachments are detached from the MIME tree: their bytes inside the body are replaced with a placeholder (<<BICHON_DETACH_HASH:…>>), and the remaining stripped EML is stored as a blob keyed by the email content hash. The original EML reconstructs byte-for-byte by swapping placeholders back with their attachment blobs.

Layer 1 — Ingest Cache (Best-Effort Skip)Copy Anchor Link

The first check happens synchronously, before parsing, on every ingestion path: IMAP download, EML import, PST import, and SMTP live ingestion all pass through the same envelope extraction core.

1. Compute BLAKE3(raw EML).
2. If (account_id, mailbox_id, content_hash) is in the in-memory dedup cache → the message is skipped entirely. It is not parsed, not indexed, and not written to the blob store. Imports count it as a duplicate, not an error.
3. Otherwise the message is parsed, queued for indexing, and the key is inserted into the cache.

Properties of the cache:

• Scope is (account, mailbox) — the same message appearing in INBOX and Sent is archived twice at the index level, once per folder. Cross-account duplicates are likewise kept as separate envelopes. (The blob layer, below, is what makes this cheap.)
• Bounded: 300,000 entries (~45 MB). When full, the oldest quarter is evicted.
• Lazily populated on first use after startup, and only with messages ingested in the last 7 days. After a restart, an older message that is re-downloaded will pass this layer and enter the index — layer 3 cleans it up later, and layer 2 guarantees it still costs no additional blob storage.
• Invalidated on delete: removing an account, mailbox, or message also clears its cache entries, so a deleted message can be re-archived later.

This layer is deliberately best-effort: a cache miss never means "not a duplicate", only "not known to be one".

Layer 2 — Content-Addressed Blob Store (Strict, Global)Copy Anchor Link

bichon-blob is a content-addressable store: every blob is keyed by its 32-byte BLAKE3 hash, in a single global key space shared by email bodies and attachments. Identical content is stored exactly once, regardless of which account or folder it arrived in — a mailbox with 50 copies of the same 10 MB PDF stores the PDF once.

• There is no chunking: one stripped EML is one blob, one attachment is one blob.
• Before writing, the blob manager checks exists(hash) and skips the write if the content is already present.
• Blobs live in append-only segment files (1 GB each) with a redb-backed index for O(1) lookup; deletes are tombstones, and a background GC compacts segments once more than 30% of a segment is dead. GC and the dedup task pause while a backup window is open.
• Shared-key safety: because email blobs and attachment blobs share one key space (e.g., an embedded .eml attachment that is also archived on its own), deletion goes through a reference barrier — a blob is tombstoned only when both its email-hash and attachment-hash reference counts in the index are zero.
• Self-healing: if a blob is missing on read, Bichon re-fetches the original message from IMAP, verifies the BLAKE3 hash matches, and re-stores it.

One nuance: the attachment hash is computed over decoded content while the blob stores raw encoded bytes. If the same decoded content later arrives with a different raw encoding (e.g., different base64 line wrapping), the first stored raw form wins — reconstruction of the second message may differ in raw bytes while the decoded content is identical. This is by design; integrity checks treat only a genuinely missing blob as a defect.

Layer 3 — Periodic Index Dedup Task (Backstop)Copy Anchor Link

A background task runs every 12 hours (the first run is 12 hours after server start) and reconciles the search index:

1. Scans only the fast column store — (account, mailbox, ingest_at, content_hash, id) — no stored-field reads.
2. Groups documents within an account by (mailbox, content_hash).
3. In each group, keeps the most recently ingested copy and soft-deletes the rest. Keeping the newest matters: after a UIDVALIDITY reset the server reassigns UIDs, and keeping the fresh copy is what lets UID-based incremental sync continue without regressing.
4. Cascade-deletes the attachment index entries belonging to the deleted envelopes, so attachment search never returns hits from removed duplicates.

This task only deletes index documents — never blobs. The surviving copy references the same content hash, so nothing in the blob store is orphaned by a dedup pass. Rounds are skipped entirely while a backup window is open, so cleanup never races a snapshot.

The end state it enforces: for every (account, mailbox), the index converges to exactly one document per content hash, no matter what slipped past layer 1.

Layer 4 — Sync-Layer Dedup (Avoiding the Download)Copy Anchor Link

Deduplication also works before ingestion, to avoid wasted bandwidth:

• UID incremental sync: per mailbox, Bichon persists UIDVALIDITY and the highest UID seen; the next sync fetches from highest_uid + 1 only.
• UIDVALIDITY changes (server rebuild, folder rename): local data is never deleted. Bichon fetches the remote UID list plus Message-ID headers, compares against the local index, and downloads only genuinely missing messages. Header comparison is the coarse filter — the ingest cache catches whatever it misses.

The Pro built-in IMAP server maintains its own stable UID mapping ((mailbox, envelope) → uid) so that handed-out UIDs do not shift when the underlying index changes (dedup cleanup, re-ingest, etc.).

Restore, Import, and BackupCopy Anchor Link

• Restore does not re-ingest: backups are snapshots of the index and blob segments, restored as-is. Dedup state comes back exactly as it was — restore neither re-deduplicates nor breaks deduplication.
• Imports (EML/MBOX/PST) go through the same pipeline as IMAP downloads, so they are deduplicated with the same (account, mailbox, content_hash) scope. Re-importing the same archive is idempotent.
• S3 backup is itself content-addressed: identical content produces an identical object key, so unchanged blobs are skipped on upload.

ConfigurationCopy Anchor Link

Deduplication is always on and has no user-facing switches. The relevant built-in parameters are:

For the storage engine itself, see Architecture and Storage & Backup.