Cursor-Scoped Event Aggregation Pattern – for high performance aggregation queries

Posted on Posted in Uncategorised

Often, stateful applications executing enterprise scenarios are not just interested in the current state of the system, but also in the events which caused the transition that resulted in the particular state. This requirement has exponentially increased the popularity of the Event Sourcing Pattern.

Event Sourcing Patterns ensure that all changes/events that caused a transition in application state are stored as a sequence of events in an append-only store. These events can then be queried for tracking history or even be used to reconstruct system state in case of an unexpected application crash.

The Event Sourcing design pattern is best suited to decouple commands and queries to improve efficiency of the system. An implicit dimension which is recorded in this pattern along with the events of interest is the ‘time’ when the event occurred. This post talks about a design pattern which enhances event sourcing pattern making it capable of portraying views based on a time range.

Problem

Apart from the requirement of replaying events to recreate state, some systems also have requirements around aggregating information drawn from events which occurred during a specific timeframe.

Running a query against the event source table to retrieve this information can be an expensive operation. The event source table has a tendency to grow quickly if it is designated to capture events related to multiple entity types. In this case the performance of querying this table to retrieve this information will not be very performant.

Apart from the requirement of replaying events to recreate state, some systems also have requirements around aggregating information drawn from events which occurred during a recent time frame. Running a query against the event source table to retrieve this information will be an expensive operation. The event source table also has a tendency to grow very fast if it is designated to capture events related to multiple entity types. In which case the efficiency of querying the event store to retrieve this information will be sub-optimal.

Event Sourcing Pattern

Solution

A solution to this problem is to introduce a Cursor-Scoped Event Aggregation Pattern on top of the Event Sourcing Pattern. The cursor here will be associated with a configurable time range which will define the scope of events the query is interested in. The pattern replicates the tuples of relevance (based on the cursor) in an in-memory data dictionary based on the filtering criteria defined in the query. The Aggregation function will collate the properties of interest within this dictionary to produce results for the query.

Cursor-Scoped Event Aggregation Pattern

The following flowchart captures the activities which are performed on the cursor data store when a new event is appended in the store or when a new query is fired against it.

CA-flowchart

 

Related Cloud Design Patterns
  • Event Sourcing Pattern – This pattern assumes an event sourcing pattern to be implemented by the system.
  • Index Table Pattern – Index table pattern is an alternative to improve performance around querying event sources.
Considerations
  • Eventual consistency – Event Sourcing Pattern is usually implemented in scenarios which can support eventual consistency. As we are querying the event source directly rather than the entities, there is a chance of inconsistency between the state of the entities and the result of the query.
  • Immutability – The event store is immutable, the only way to reverse a transaction is to introduce a compensation event. The Aggregator should be capable of handling compensation events.
Usage Scenarios
  • High performance systems – Systems which required real time response on aggregation queries.

8 thoughts on “Cursor-Scoped Event Aggregation Pattern – for high performance aggregation queries

  1. Have you ever thought about publishing an e-book or guest authoring on other sites? I have a blog based upon on the same topics you discuss and would really like to have you share some stories/information. I know my audience would enjoy your work. If you’re even remotely interested, feel free to shoot me an email.

  2. Excellent weblog right here! Additionally your website a lot up very fast! What host are you using? Can I am getting your associate link to your host? I desire my web site loaded up as fast as yours lol

  3. Hi, Neat post. There is a problem with your site in web explorer, might test this? IE still is the market leader and a huge component to other folks will pass over your excellent writing due to this problem.

  4. Howdy! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us valuable information to work on. You have done a wonderful job!

Leave a Reply

Your email address will not be published. Required fields are marked *