Wino-Mail/OUTLOOK_QUEUE_SYNC_IMPLEMENTATION.md

Outlook Queue-Based Synchronization Implementation

Overview

This document describes the implementation of the queue-based, metadata-only synchronization system for Outlook, mirroring the approach used in Gmail but adapted for Outlook's per-folder synchronization model.

Key Differences from Gmail

1. Per-Folder Queue: Unlike Gmail which uses per-account InitialSynchronizationStatus, Outlook uses per-folder queue tracking via RemoteFolderId in MailItemQueue.
2. Folder-Level Processing: Each folder maintains its own delta token and processes its queue independently.
3. No Account-Level Status: Outlook doesn't use Account.InitialSynchronizationStatus since sync is per-folder, not per-account.

Architecture Changes

1. MailItemQueue Entity Enhancement

File: Wino.Core.Domain\Entities\Mail\MailItemQueue.cs

Added RemoteFolderId property to support per-folder queue tracking:

public string RemoteFolderId { get; set; }  // For Outlook per-folder sync

2. Service Layer Updates

Files:

• Wino.Core.Domain\Interfaces\IMailService.cs
• Wino.Services\MailService.cs
• Wino.Core\Integration\Processors\DefaultChangeProcessor.cs

Added new methods for folder-specific queue operations:

Task<List<MailItemQueue>> GetMailItemQueueByFolderAsync(Guid accountId, string remoteFolderId, int take);
Task<int> GetMailItemQueueCountByFolderAsync(Guid accountId, string remoteFolderId);

3. OutlookSynchronizer Redesign

File: Wino.Core\Synchronizers\OutlookSynchronizer.cs

New Methods

1. QueueMailIdsForFolderAsync

   • Queues all mail IDs for a specific folder using Delta API
   • Only retrieves message IDs (minimal data transfer)
   • Stores delta token for future incremental syncs
   • Creates queue entries with RemoteFolderId for folder tracking

2. ProcessMailQueueForFolderAsync

   • Processes queued mail IDs in batches
   • Downloads metadata only (no MIME content)
   • Handles failures with retry logic
   • Updates queue item status (IsProcessed, FailedCount)

3. DownloadMessageMetadataBatchAsync

   • Downloads metadata for a batch of messages concurrently
   • Uses semaphore to limit concurrent downloads (10 max)
   • Calls CreateMailCopyFromMessage for metadata extraction
   • Creates NewMailItemPackage with null MimeMessage

4. CreateMailCopyFromMessage (Centralized)

   • REPLACES scattered AsMailCopy() and CreateMinimalMailCopyAsync() calls
   • Single source of truth for converting Graph Message to MailCopy
   • Extracts all required fields from metadata
   • Sets FolderId, UniqueId, and FileId

Modified Methods

1. DownloadMailsForInitialSyncAsync

   • Now orchestrates queue-based sync
   • Step 1: Queue all mail IDs via Delta API
   • Step 2: Process queue in batches

2. ProcessDeltaChangesAndDownloadMailsAsync

   • Downloads delta changes with metadata only
   • Uses DownloadMessageMetadataBatchAsync instead of full MIME download

3. CreateNewMailPackagesAsync

   • Still downloads MIME for specific scenarios (search results, drafts)
   • Uses CreateMailCopyFromMessage for consistency
   • Not called during normal sync operations

Removed Methods

• DownloadMailsConcurrentlyAsync - Replaced by queue system
• DownloadSingleMailAsync - Replaced by queue batch processing
• Scattered CreateMinimalMailCopyAsync implementations

Synchronization Flow

Initial Sync (Per Folder)

1. SynchronizeFolderAsync
   ├─ Check: !folder.IsInitialSyncCompleted
   └─ DownloadMailsForInitialSyncAsync
      ├─ QueueMailIdsForFolderAsync
      │  ├─ Use Delta API with Select=["Id"]
      │  ├─ Iterate all pages
      │  ├─ Create MailItemQueue entries with RemoteFolderId
      │  └─ Store delta token
      └─ ProcessMailQueueForFolderAsync
         ├─ Get queue items by folder (100 at a time)
         ├─ Process in chunks of 20
         ├─ DownloadMessageMetadataBatchAsync
         │  ├─ Concurrent download (10 max)
         │  ├─ GetMessageByIdAsync (metadata fields only)
         │  ├─ CreateMailCopyFromMessage
         │  └─ CreateMailAsync (package with null MIME)
         └─ Update queue status

Delta Sync (Per Folder)

1. SynchronizeFolderAsync
   ├─ Check: folder.IsInitialSyncCompleted
   └─ ProcessDeltaChangesAndDownloadMailsAsync
      ├─ Use Delta API with existing token
      ├─ Collect new mail IDs
      ├─ DownloadMessageMetadataBatchAsync
      │  ├─ Download metadata only
      │  └─ Create MailCopy entries
      └─ Update delta token

On-Demand MIME Download

User Reads Mail
└─ DownloadMissingMimeMessageAsync
   ├─ Download full MIME via /messages/{id}/$value
   └─ SaveMimeFileAsync

Benefits

1. Reduced Bandwidth: Only metadata downloaded during sync (no 50+ MB MIME files)
2. Faster Sync: Parallel processing with controlled concurrency
3. Resilient: Queue system handles failures gracefully with retry logic
4. Consistent: Centralized CreateMailCopyFromMessage method
5. Scalable: Per-folder processing allows independent folder syncs

Code Consolidation

Before (Scattered Approach)

// Multiple places creating MailCopy
var mailCopy = message.AsMailCopy();
mailCopy.FolderId = folder.Id;
// ... repeat in 3+ locations

// Mixed MIME downloading
var package = await CreateNewMailPackagesAsync(...); // Downloads MIME
var minimal = await CreateMinimalMailCopyAsync(...); // No MIME

After (Centralized Approach)

// Single method for all scenarios
var mailCopy = CreateMailCopyFromMessage(message, folder);

// Clear separation
// Sync: metadata only
var package = new NewMailItemPackage(mailCopy, null, folder.RemoteFolderId);

// On-demand: full MIME
await DownloadMissingMimeMessageAsync(mailCopy, ...);

Graph API Fields Used

Only essential fields are requested during sync:

private readonly string[] outlookMessageSelectParameters =
[
    "InferenceClassification",
    "Flag",
    "Importance",
    "IsRead",
    "IsDraft",
    "ReceivedDateTime",
    "HasAttachments",
    "BodyPreview",
    "Id",
    "ConversationId",
    "From",
    "Subject",
    "ParentFolderId",
    "InternetMessageId",
];

NOT downloaded during sync:

• Body content (HTML/Text)
• Raw MIME message
• Attachment content
• Extended properties

Migration Path for IMAP (Future)

The base WinoSynchronizer class has been updated with:

protected virtual Task QueueMailIdsForInitialSyncAsync(MailItemFolder folder, CancellationToken cancellationToken = default);
protected virtual Task<List<string>> DownloadMailsFromQueueAsync(MailItemFolder folder, int batchSize, CancellationToken cancellationToken = default);
protected virtual Task<MailCopy> CreateMinimalMailCopyAsync(TMessageType message, MailItemFolder assignedFolder, CancellationToken cancellationToken = default);

IMAP can override these methods to implement similar queue-based sync:

• TODO: Add folder-based queue support to IMAP
• TODO: Implement metadata-only header parsing
• TODO: Centralize IMAP MailCopy creation

Testing Checklist

• Initial sync queues all mail IDs per folder
• Queue processing downloads metadata only
• Delta sync uses metadata-only approach
• Failed queue items retry correctly
• Concurrent download respects semaphore limits
• Delta token stored and used correctly per folder
• Search results still download MIME when needed
• Draft handling works with MIME headers
• On-demand MIME download functions correctly
• Large folders (1000+ messages) sync efficiently
• Network interruption recovery

Performance Expectations

Before (With MIME):

• 100 messages ≈ 100-500 MB download
• Sync time: 5-10 minutes
• API calls: 100+ individual message downloads

After (Metadata Only):

• 100 messages ≈ 1-5 MB download (metadata)
• Sync time: 30-60 seconds
• API calls: Batched requests (10-20 concurrent)
• MIME downloaded only when user reads (lazy loading)

Notes

• CreateNewMailPackagesAsync is now marked as only for special cases
• DefaultChangeProcessor no longer needed for basic operations
• All synchronizers can benefit from this pattern (Gmail, IMAP, Outlook)
• The InitialSyncMimeDownloadCount property is now obsolete