A Simple State-Based Synchronization Algorithm

This post is inspired by experiences writing vdirsyncer and explores a common synchronization challenge. It’s aimed at developers needing to implement synchronization, especially when journal-based approaches aren’t feasible due to API limitations.

Synchronizing data between two locations (e.g., a local client and a cloud service, or two cloud services) is a common problem. How do you ensure that additions, deletions, and modifications on one side are correctly reflected on the other, especially when you can’t rely on a central transaction log or journal?

This article outlines a state-based algorithm, drawing inspiration from how tools like OfflineIMAP handle synchronization, and extending it to cope with mutable items.

Recap: The Problem and OfflineIMAP’s Approach (Immutable Items)

E.Z. Yang’s article “[How OfflineIMAP works]"(http://lists.alioth.debian.org/pipermail/offlineimap-devel/2012-January/0014 Offlineimap-devel Offlineimap-devel/0014 Offlineimap-devel 8070.html) provides excellent background. The core problem OfflineIMAP addresses (for immutable items like emails initially) can be stated as:

You are given two “sets” A and B of “items”, where the items have immutable content and globally unique IDs. Define a function sync(A, B).

  • If items are added to one set, sync(A, B) should copy those items to the other set.
  • If items are deleted from one set, those deletions should also be performed on the other set.

The key insight in OfflineIMAP’s approach is maintaining a third set, let’s call it status. This set doesn’t store item content, only the IDs of items that were present (in both A and B) after the previous successful synchronization. On the very first sync, status is empty.

By comparing the presence or absence of an item ID across all three sets (the current state of A, the current state of B, and the status from the last sync), we can deduce what happened:

  1. [Case: Item only in A] (ID in A, but not B, not status)

    • Deduction: Item was created on side A since the last sync.
    • Action: Copy item from A to B. Add ID to status.
  2. [Case: Item only in B] (ID in B, but not A, not status)

    • Deduction: Item was created on side B since the last sync.
    • Action: Copy item from B to A. Add ID to status.
  3. [Case: Item in A and status, but not B] (ID in A, ID in status, but not B)

    • Deduction: Item existed previously but was deleted from side B.
    • Action: Delete item from A. Remove ID from status.
  4. [Case: Item in B and status, but not A] (ID in B, ID in status, but not A)

    • Deduction: Item existed previously but was deleted from side A.
    • Action: Delete item from B. Remove ID from status.
  5. [Case: Item in A and B, but not status] (ID in A, ID in B, but not status)

    • Deduction: Item was created independently on both sides since the last sync (or this is the first sync). Since content is assumed immutable, they should be identical if truly the same item.
    • Action: Add ID to status. (No copy needed if items are truly immutable and identical).
  6. [Case: Item only in status] (ID in status, but not A, not B)

    • Deduction: Item existed previously but was deleted from both sides (or one side deletion wasn’t synced before the other side deleted it too). The status entry is now orphaned.
    • Action: Remove ID from status.

(Note: The original text uses images for these cases; descriptive text is used here for broader compatibility.)

This algorithm works well for immutable items. Trying to adapt it directly to mutable items by treating modifications as a “delete” followed by a “create” has significant drawbacks:

  • Conflict Duplication: If an item is modified differently on both sides, this approach results in both modified versions appearing on both sides after the sync. This might be okay for generic file sync but is often unhelpful for structured data.
  • API Inefficiency: Deleting and then creating an item is often more resource-intensive (more API calls, potential data loss window) than simply updating an existing item.

The Problem with Mutable Items

Let’s refine the problem statement for mutable content:

You are given two “sets” A and B of “items”, where each item has a globally unique ID and mutable content. Define a function sync(A, B).

  • If items are added to one set, copy them to the other.
  • If items are deleted from one set, delete them from the other.
  • If items on one side are modified, update the corresponding item on the other side.
  • Handle cases where an item is modified on both sides (conflict).

To handle modifications, we need a way to detect if an item’s content has changed since the last sync. Let’s assume we have access to a reasonably cheap way to get a checksum or “etag” for each item’s current state on each side (e.g., a content hash like MD5/SHA1, or perhaps a reliable last-modified timestamp like mtime). Importantly, the etag value for the same logical item might differ between side A and side B even if the content is identical (e.g., different filesystem metadata, different API representations).

The Algorithm for Mutable Items

The core change required is upgrading our status store. It can no longer be just a set of IDs. It needs to become a mapping that stores, for each known item ID, the etags it had on both sides after the last successful sync.

Example status Mapping:

Item ID ETag A (Previous Sync) ETag B (Previous Sync)
item1 771627ff caf893af
item2 dc165135 07b92840
item3 e4d1c3b2 a5f0e9d8

Now, when we examine an item ID present in the current state of A, B, and in the status mapping, we compare the current etags from A and B with the stored etags in the status map:

  1. [Case: Item in A, B, status - ETag Changed on A Only]

    • Condition: current_etag_A != status[ID].etag_A AND current_etag_B == status[ID].etag_B
    • Deduction: Item was modified only on side A since the last sync.
    • Action: Copy item content from A to B. Update status[ID] with (current_etag_A, current_etag_B).
  2. [Case: Item in A, B, status - ETag Changed on B Only]

    • Condition: current_etag_A == status[ID].etag_A AND current_etag_B != status[ID].etag_B
    • Deduction: Item was modified only on side B since the last sync.
    • Action: Copy item content from B to A. Update status[ID] with (current_etag_A, current_etag_B).
  3. [Case: Item in A, B, status - ETag Changed on Both]

    • Condition: current_etag_A != status[ID].etag_A AND current_etag_B != status[ID].etag_B
    • Deduction: Item was modified independently on both sides since the last sync. This is a conflict.
    • Action: Invoke conflict_resolution(ID, itemA, itemB). The resolution function should decide the outcome and potentially update status[ID] accordingly.

Additionally, the original case 5 needs modification:

  1. [Case: Item in A and B, but not status] (ID in A, ID in B, but not status)
    • Deduction: Item was created independently on both sides since the last sync, or this is the first time syncing this item. The content might be different. This is potentially a conflict.
    • Action: Invoke conflict_resolution(ID, itemA, itemB). The resolution function should decide the outcome (e.g., merge, duplicate, choose one) and add the resulting state to status (e.g., status[ID] = (current_etag_A, current_etag_B) after resolution).

The other cases (1, 2, 3, 4, 6 from the immutable algorithm) generally remain the same, though the action must now also include updating the status map with the relevant etags upon creation/copy (case 1 & 2) or removing the entry upon deletion (case 3, 4, 6).

Conflict Resolution Strategies

Handling conflicts (cases 9 and 10) is highly application-specific. The conflict_resolution routine needs to decide what to do when changes clash or when new items with the same ID appear. Options include:

  • Ask the User: Prompt the user interactively to choose which version (A or B) to keep, or if they want to merge.
  • Duplicate: Create both versions on both sides, possibly renaming one or assigning a new ID. (Similar to the flawed immutable approach, but potentially made explicit).
  • Last-Modified Wins: If reliable, high-resolution timestamps are available for actual content modification (not just metadata changes), use the timestamp to pick the “winner”. (See caveats below).
  • Three-Way Merge: If you stored the item’s content (or etag) from the previous sync (the “base” version), you might perform a three-way merge between base, A, and B. This requires storing more state.
  • Format-Specific Merge: If the data format inherently supports merging (e.g., it tracks change history like Git, or uses CRDTs), use that mechanism.
  • Predetermined Winner: Always prefer changes from A over B, or vice-versa (simple, but can lead to data loss).

Crucial Step: Before declaring a conflict based solely on differing etags (especially in case 10 or on the first sync), conflict_resolution should always check if the actual content of the item on side A is different from the content on side B. If the content is identical despite differing etags, it’s not a real conflict; simply update the status map with the current etags and proceed.

ETag Considerations (Especially mtime)

If using file modification times (mtime) as etags:

  • Precision: Filesystem mtime precision varies (e.g., FAT has 2-second resolution). Changes made within that resolution window might be missed if mtime is the only check. POSIX systems offer st_mtime_ns for nanosecond precision, but support varies.
  • Bogus Changes: Metadata changes (like touching a file) can update mtime without altering content. Relying solely on mtime can lead to unnecessary sync operations or false conflict reports.

As noted, vdirsyncer uses mtime primarily as a hint that a file might have changed, then confirms by comparing content hashes to avoid false positives. This hybrid approach is often practical.

Assumptions:

  • Running in a Node.js environment (for the crypto module for stable hashing). If running in a browser, you might need a different hashing library or the built-in SubtleCrypto API.
  • Similar simplifications as the Python version (in-memory stores, basic conflict resolution).
import crypto from 'crypto'; // For stable hashing

// --- Data Structures ---

class Item {
    // Make id readonly after construction
    public readonly id: string;
    public content: string;

    constructor(id: string, content: string) {
        this.id = id;
        this.content = content;
    }

    toString(): string {
        // Simple representation for logging
        return `Item(id='${this.id}', content='${this.content}')`;
    }
}

// Use Map for stores: { item_id => Item }
type Store = Map<string, Item>;

// Use Map for status: { item_id => [etag_A | null, etag_B | null] }
// Tuple represents etags from the *last* sync
type StatusMap = Map<string, [string | null, string | null]>;

// --- Helper Functions ---

function getEtag(item: Item | null | undefined): string | null {
    """Calculates a stable SHA1 etag for an item's content."""
    if (!item) {
        return null;
    }
    // Use crypto for stable hashing
    const hash = crypto.createHash('sha1');
    hash.update(item.content, 'utf-8');
    return hash.digest('hex');
}

function copyItem(sourceStore: Store, destStore: Store, itemId: string): void {
    """Copies item content from source to destination."""
    const sourceItem = sourceStore.get(itemId);
    if (sourceItem) {
        // Create a new Item object to avoid shared references
        destStore.set(itemId, new Item(sourceItem.id, sourceItem.content));
        console.log(`SYNC: Copied item '${itemId}' from source to destination.`);
    } else {
        console.log(`SYNC WARNING: Tried to copy non-existent item '${itemId}' from source.`);
    }
}

function deleteItem(store: Store, itemId: string): void {
    """Deletes an item from a store."""
    if (store.has(itemId)) {
        store.delete(itemId);
        console.log(`SYNC: Deleted item '${itemId}' from store.`);
    }
}

function resolveConflict(
    itemId: string,
    itemA: Item | undefined,
    itemB: Item | undefined,
    storeA: Store,
    storeB: Store
): [string | null, string | null] {
    """
    Handles conflicts based on a predefined strategy.
    Returns the etags [etag_a, etag_b] that should be stored in the status map
    after resolution.
    """
    console.log(`CONFLICT DETECTED for item '${itemId}'!`);

    // --- Implement your conflict resolution strategy here ---
    // Strategy: Simple "A wins" (if A exists)
    if (itemA) {
        console.log(`RESOLUTION: '${itemId}' - Applying 'A wins'. Copying A -> B.`);
        copyItem(storeA, storeB, itemId);
        // Both sides now match A's content. Need to get the potentially updated item B
        const resolvedItemA = storeA.get(itemId); // Should be itemA
        const resolvedItemB = storeB.get(itemId); // Now matches itemA
        const etagA = getEtag(resolvedItemA);
        const etagB = getEtag(resolvedItemB);
        return [etagA, etagB];
    } else if (itemB) {
        // A doesn't exist, but B does (e.g., Case 10 where A was deleted before sync)
        console.log(`RESOLUTION: '${itemId}' - A missing, keeping B. Copying B -> A.`);
        copyItem(storeB, storeA, itemId);
        const resolvedItemA = storeA.get(itemId); // Now matches itemB
        const resolvedItemB = storeB.get(itemId); // Should be itemB
        const etagA = getEtag(resolvedItemA);
        const etagB = getEtag(resolvedItemB);
        return [etagA, etagB];
    } else {
        // Should not happen if called correctly, but handle defensively
        console.error(`RESOLUTION ERROR: '${itemId}' - Both items missing during conflict resolution.`);
        return [null, null];
    }

    // --- Other possible strategies ---
    // Strategy: "B wins"
    // Strategy: "Duplicate" (create item_a_conflict, item_b_conflict) - More complex
    // Strategy: "Ask User" (requires interactive input)
    // Strategy: "Merge" (requires understanding item content)
}


// --- The Synchronization Algorithm ---

function synchronize(storeA: Store, storeB: Store, status: StatusMap): StatusMap {
    """
    Performs a two-way sync between storeA and storeB using the status map.
    Modifies storeA and storeB in place (as Maps are reference types).
    Returns the updated status map.
    """
    console.log("\n--- Starting Synchronization ---");
    const newStatus: StatusMap = new Map();

    // Get current states efficiently
    const currentItemsA = new Map(storeA); // Shallow copy items for iteration safety if needed
    const currentItemsB = new Map(storeB);
    const currentEtagsA: Map<string, string | null> = new Map();
    currentItemsA.forEach((item, id) => currentEtagsA.set(id, getEtag(item)));
    const currentEtagsB: Map<string, string | null> = new Map();
    currentItemsB.forEach((item, id) => currentEtagsB.set(id, getEtag(item)));

    // Combine all unique IDs from current state and previous status
    const allIds = new Set([...currentItemsA.keys(), ...currentItemsB.keys(), ...status.keys()]);

    // Process in a consistent order (optional but good for debugging)
    const sortedIds = Array.from(allIds).sort();

    for (const itemId of sortedIds) {
        console.log(`\nProcessing ID: '${itemId}'`);
        const inA = currentItemsA.has(itemId);
        const inB = currentItemsB.has(itemId);
        const inStatus = status.has(itemId);

        const currentEtagA = currentEtagsA.get(itemId) ?? null;
        const currentEtagB = currentEtagsB.get(itemId) ?? null;
        const [prevEtagA, prevEtagB] = status.get(itemId) ?? [null, null];

        const itemA = currentItemsA.get(itemId);
        const itemB = currentItemsB.get(itemId);

        // Determine the case based on presence in A, B, and Status
        if (inA && inB && inStatus) {
            // Item exists everywhere - check for modifications
            console.log(" State: In A, In B, In Status");
            const changedA = currentEtagA !== prevEtagA;
            const changedB = currentEtagB !== prevEtagB;

            if (changedA && !changedB) { // Case 7: Modified on A only
                console.log(" Action: Modified on A. Copying A -> B.");
                copyItem(storeA, storeB, itemId);
                // Etag B might change representationally, recalculate for status
                newStatus.set(itemId, [currentEtagA, getEtag(storeB.get(itemId))]);
            } else if (!changedA && changedB) { // Case 8: Modified on B only
                console.log(" Action: Modified on B. Copying B -> A.");
                copyItem(storeB, storeA, itemId);
                newStatus.set(itemId, [getEtag(storeA.get(itemId)), currentEtagB]);
            } else if (changedA && changedB) { // Case 9: Modified on both (Conflict)
                console.log(" Action: Modified on both sides.");
                const resolvedEtags = resolveConflict(itemId, itemA, itemB, storeA, storeB);
                newStatus.set(itemId, resolvedEtags);
            } else { // No change
                console.log(" Action: No change detected.");
                newStatus.set(itemId, [currentEtagA, currentEtagB]);
            }

        } else if (inA && inB && !inStatus) { // Case 10: Exists on A, B, but new to status (Potential conflict)
            console.log(" State: In A, In B, NOT In Status");
            console.log(" Action: New on both sides or first sync.");
            // Before declaring conflict, check if content is actually different
            // Note: Direct etag comparison is sufficient if etags are reliable content hashes
            if (currentEtagA === currentEtagB) { // Assuming reliable etags represent content
                console.log("  Content matches based on etag. Adding to status.");
                newStatus.set(itemId, [currentEtagA, currentEtagB]);
            } else {
                 // Could add an explicit content check here if etags might differ for same content
                 // if (itemA && itemB && itemA.content === itemB.content) { ... } else { resolve...}
                const resolvedEtags = resolveConflict(itemId, itemA, itemB, storeA, storeB);
                newStatus.set(itemId, resolvedEtags);
            }

        } else if (inA && !inB && inStatus) { // Case 3: Deleted on B
            console.log(" State: In A, NOT In B, In Status");
            console.log(" Action: Deleted on B. Deleting from A.");
            deleteItem(storeA, itemId);
            // Removed from status implicitly (not added to newStatus)

        } else if (!inA && inB && inStatus) { // Case 4: Deleted on A
            console.log(" State: NOT In A, In B, In Status");
            console.log(" Action: Deleted on A. Deleting from B.");
            deleteItem(storeB, itemId);
            // Removed from status implicitly

        } else if (inA && !inB && !inStatus) { // Case 1: Created on A
            console.log(" State: In A, NOT In B, NOT In Status");
            console.log(" Action: Created on A. Copying A -> B.");
            copyItem(storeA, storeB, itemId);
            newStatus.set(itemId, [currentEtagA, getEtag(storeB.get(itemId))]);

        } else if (!inA && inB && !inStatus) { // Case 2: Created on B
            console.log(" State: NOT In A, In B, NOT In Status");
            console.log(" Action: Created on B. Copying B -> A.");
            copyItem(storeB, storeA, itemId);
            newStatus.set(itemId, [getEtag(storeA.get(itemId)), currentEtagB]);

        } else if (!inA && !inB && inStatus) { // Case 6: Deleted on both sides
            console.log(" State: NOT In A, NOT In B, In Status");
            console.log(" Action: Deleted on both sides. Removing from status.");
            // Removed from status implicitly

        } else if (!inA && !inB && !inStatus) {
            // This case should ideally not happen if allIds is constructed correctly
            console.log(` State WARNING: Item '${itemId}' somehow not found anywhere?`);
        }
    } // End for loop

    console.log("\n--- Synchronization Complete ---");
    return newStatus;
}


// --- Example Usage ---

// Initial State
let storeA: Store = new Map([
    ["item1", new Item("item1", "Content A v1")],
    ["item2", new Item("item2", "Shared Content v1")],
    ["item3", new Item("item3", "Only A Content")],
]);
let storeB: Store = new Map([
    ["item1", new Item("item1", "Content B v1")], // Different content initially
    ["item2", new Item("item2", "Shared Content v1")],
    ["item4", new Item("item4", "Only B Content")],
]);
// Start with an empty status for the first sync
let status: StatusMap = new Map();


console.log("=== Initial State ===");
console.log("Store A:", storeA);
console.log("Store B:", storeB);
console.log("Status :", status);

// --- First Sync ---
status = synchronize(storeA, storeB, status);

console.log("\n=== State After First Sync ===");
console.log("Store A:", storeA);
console.log("Store B:", storeB);
console.log("Status :", status); // Should now contain etags for item1, item2, item3, item4


// --- Make some changes before the second sync ---
console.log("\n=== Making Changes Before Second Sync ===");
// Modify item on A
storeA.get("item1")?.content = "Content A v2";
// Modify item on B
storeB.get("item2")?.content = "Shared Content v2";
// Delete item from A
storeA.delete("item3");
// Add new item to B
storeB.set("item5", new Item("item5", "New B Content"));
// Modify item on both sides (will cause conflict)
// Note: Item 4 was copied from B to A in the first sync. Now modify both.
storeA.get("item4")?.content = "Only A Content Now";
storeB.get("item4")?.content = "Only B Content v2";


console.log("Store A (Changes):", storeA);
console.log("Store B (Changes):", storeB);


// --- Second Sync ---
status = synchronize(storeA, storeB, status);

console.log("\n=== State After Second Sync ===");
console.log("Store A:", storeA);
console.log("Store B:", storeB);
console.log("Status :", status);

// Expected outcome after second sync (with "A wins" conflict resolution):
// item1: B should now have "Content A v2"
// item2: A should now have "Shared Content v2"
// item3: Should be deleted from B (as it was deleted from A)
// item4: Conflict resolved ("A wins"), B should have "Only A Content Now"
// item5: Should be copied from B to A

To run this:

  1. Save the code as a .ts file (e.g., sync.ts).
  2. Make sure you have Node.js and TypeScript installed (npm install -g typescript ts-node).
  3. Compile and run: tsc sync.ts && node sync.js or run directly using ts-node sync.ts.

Conclusion

This state-based synchronization algorithm, extending the ideas used in OfflineIMAP, provides a reliable way to synchronize mutable items between two stores without relying on transaction journals. It hinges on maintaining a status map tracking item IDs and their etags from the previous sync, comparing current etags to detect changes, and having a well-defined strategy for conflict resolution. While more complex than simple copying, it handles additions, deletions, modifications, and conflicts in a structured manner, making it suitable for various “cloud sync” scenarios where simpler methods fall short.