Wednesday, August 22, 2018

Cosmos Change Feed and Lease Collection Use

The change feed functionality in Azure Cosmos DB is a great feature with a wide variety of uses. The change feed subscribes to changes that occur in a collection and keeps its state in another collection, called the Lease Collection. Because the minimum RUs in a collection are 400 and a single lease hardly uses 50 of the RUs, we had decided to store additional meta data documents in the same lease collection, because we could not use the Continuation Token and Sequence Number data to determine where in the stream the subscriber was (for our own monitoring purposes). Don’t do this! The change feed library will regularly scan and pull all documents in this collection and use it to balance the subscribers and therefore the more documents are in this collection, the greater the RU requirements. Our RUs were bumping against 4,000 when we finally realized what it was doing:


After removing our additional metadata documents, the lease collection RUs are much better (note that we have several subscribers using the same lease).


Now we store our meta data documents in another collection, which happily sits at around 100 RUs, while leases, as you can see above, hovers around the same.

Bottom line: don’t store anything custom in the Lease Collection of the change feed because it exponentially increases the RU requirements.