Dealing with transactions in a distributed microservice environment
As far as any other technology, microservices can solve numerous architectural problems, but also introduces others.
One of the big questions here is how to deal with ACID properties of a transaction in a distributed environment. Since each microservice deals with its own domain, take care of the transaction completeness by all theirs dependencies is a huge problem.
To illustrate it take into account this situation. Imagine a customer in a bakery asking the waiters if its order is already done. Since the waiters make orders and the cookers cooks the orders, understand whether a order is already done or how much time until its finished is out of its domain. In order to answer these question the waiters must ask the cookers for the answer.
Everybody can remember this situation in real life, however, how to deal with this in distributed environments?
Saga Pattern
Since 1987 the Saga Pattern deals with business transactions that span into multiple services. Here a saga is a sequence of transactions where the first initiates a call to other service and the completeness of the first transaction depends on the entire chain. The figure below illustrates this scenario.
As one can see, to ensure that an order has been concluded (i.e. the events 2, 3 and 4 were finished), it is necessary to “wait” for the other services conclusion in the sequence.
This pattern has different types of implementation which are not cover here. Follow the references for more details.
A Saga Pattern implementation
In order to illustrate our example we reintroduce our bakery scenario applying the Event Oriented implementation of Saga Pattern.
Here we have three personas named Waiter, Cooker and Bartender. Waiters can show the menu items, make orders and check orders status. Cookers and Bartenders receive theirs respective kind of orders (i.e. foods and drinks) and make them. Each item has different preparation time.
After a order has been created, the customers can ask the Waiters for theirs order status. Since we are in a distributed environment the Waiters must be flagged when orders are finished.
We choose the Python 3.8+ language with Flask web framework to write our microservices and kafka-python library to deal with Kafka. Mongodb as theirs database and Kafka as messenger manager. This code can be find at https://github.com/victoramsantos/saga-pattern-example (v1.0.0). In the follow we shows our architecture.
Waiters has a database schema for menu, with all available items, and other for order’s status. Both Cooker and Bartender has databases with each item and its respective preparation time.
Customers can get the menu through the /menu API which will returns something like:
After selected their items, customers can make orders. The POST /order API starts this transaction. Since only Waiters interfaces with customers, they collect the customers orders and produces messages to one of foods or drinks topics. Each message has the orderId and the items’ type and id. Here the request must have a body like this:
Here we already started our saga transaction. Now if a customer ask for an order status, the GET /order API should answer something like:
Note the different values for status here. Since each item are build by the Cookers or Bartenders and the items has different preparation times, the Waiters must be listening the balcony topic to retrieve the items status. Messages in this topic are in the format:
Conclusion
Here we presented a Saga Pattern implementation. We exemplified the pattern with a spanned transaction. Our bakery orders were created by Waiters but until the Cookers and Bartenders processes the orders we can not delivery it to customers.
As next steps, we intend to create other articles presenting our code and drill down in more details our implementation.
