Azure Documentation

Azure Cosmos DB Change Feed

The Change Feed is a persistent record of changes that have occurred within your Azure Cosmos DB container. It provides ordered, immutable representations of every document that has been modified or deleted within the container. This feature is invaluable for building real-time applications, data synchronization, auditing, and various other scenarios that require reacting to data modifications.

What is the Change Feed?

The Change Feed is an append-only log of changes. Each entry in the Change Feed represents a version of a document at a specific point in time. It includes metadata such as the operation type (create, update, delete) and the timestamp of the change. You can process these changes using various methods, including:

• Change Feed Processors: A common and recommended way to consume the Change Feed. They provide a distributed, scalable, and fault-tolerant mechanism for processing changes.
• Direct API Access: For simpler use cases or custom processing logic, you can directly query the Change Feed using the Cosmos DB SDKs.

Key Concepts

• Lease Container: When using Change Feed Processors, a lease container is required. This container stores lease information that helps the processor maintain state and distribute the workload across multiple instances.
• Partition Key: The Change Feed is processed based on the container's partition key. For efficient processing, ensure your partition key design supports even distribution of operations.
• Consistency Levels: The Change Feed reflects changes based on the consistency level configured for your Cosmos DB account.

Common Use Cases

• Real-time Data Synchronization: Propagate changes to other databases, caches, or services.
• Auditing: Maintain a historical record of all data modifications for compliance and security.
• Data Archiving: Archive older versions of documents.
• Building Event-Driven Architectures: Trigger downstream processes based on data changes.
• Data Analytics: Stream data changes to analytical systems for near real-time insights.

Getting Started with Change Feed Processors

The most robust way to work with the Change Feed is by using the Change Feed Processor library. Here's a simplified conceptual example using C# (the actual implementation might vary based on SDK version and language):

using Microsoft.Azure.Cosmos;
using Microsoft.Azure.Cosmos.ChangeFeedProcessor;
using System;
using System.Threading.Tasks;

// Assuming you have initialized your CosmosClient and obtained container references
CosmosClient client = new CosmosClient("YOUR_COSMOS_DB_CONNECTION_STRING");
Container sourceContainer = client.GetDatabase("yourDatabase").GetContainer("yourSourceContainer");
Container leaseContainer = client.GetDatabase("yourDatabase").GetContainer("yourLeaseContainer"); // Lease container

async Task StartChangeFeedProcessorAsync()
{
    ChangeFeedProcessor processor = sourceContainer.GetChangeFeedProcessorBuilder(
            processorName: "myChangeFeedProcessor",
            onChangesDelegate: HandleChangesAsync)
        .WithLeaseContainer(leaseContainer)
        .Build();

    await processor.StartAsync();

    Console.WriteLine("Change Feed Processor started. Press Ctrl+C to stop.");
    // Keep the application running to listen for changes
    await Task.Delay(Timeout.Infinite);
}

async Task HandleChangesAsync(
    ChangeFeedProcessorContext context,
    IReadOnlyCollection<dynamic> changes,
    CancellationToken cancellationToken)
{
    Console.WriteLine($"Processing {changes.Count} changes.");
    foreach (var change in changes)
    {
        // Process each change here
        // For example, log the change or send it to another service
        Console.WriteLine($"Operation: {context.OperationKind}, Document ID: {change.id}");
        // Example: Access document properties
        // Console.WriteLine($"Document Content: {change.ToString()}");
    }
    // You can also access context.OwnerResourceId, context.OwnerName etc.
}

// To start the processor:
// await StartChangeFeedProcessorAsync();
            

Monitoring the Change Feed

You can monitor the health and performance of your Change Feed Processor through Azure Monitor, including metrics like:

• Throughput (RU/s) consumed by the processor.
• Latency of change processing.
• Number of processed changes.
• Error rates.

Considerations

• Processing Guarantees: The Change Feed Processor provides "at-least-once" processing guarantees. This means you might process the same change twice in rare failure scenarios. Design your handlers to be idempotent.
• Order of Changes: Changes are guaranteed to be ordered within a logical partition key. Across different partition keys, the order is not guaranteed.
• Deleted Documents: The Change Feed includes entries for deleted documents. If you need to retrieve the content of a deleted document, you should use the soft delete feature or versioning if your application requires it.

By leveraging the Azure Cosmos DB Change Feed, you can build dynamic, responsive, and data-driven applications that react instantaneously to data modifications.