This text was reconstructed from the slides, outline, announcement and source code. No recording is available, so it is not a verbatim transcript. The topics follow the 23 slides of the 2024 talk.
The original slide images remain in Russian. The transcript is available in the selected language; code and API identifiers are not translated.
Hello! I’m Pavel Simonov, a systems architect at Citeck. Today I’ll explain how we implemented distributed transactions, the difficulties involved and how to handle them.
We’ll start with two-phase commit, then move on to transactional actions and event listeners. The goal is to make operations across microservices easier to write while keeping the data consistent.
We’re all used to ordinary transactions. Put an annotation on a method, and you expect either all its changes to be saved or everything to roll back on error. With one database, that’s straightforward.
Here we begin a transaction, make several changes and finish with Commit. Those changes need to be saved together.
What if something goes wrong halfway through? Say the first two changes succeed, but the next one fails. We call Rollback, and the database returns to the state before those changes. We don’t have to work out which writes succeeded and undo them ourselves.
Rollback has limits, though. If we’ve already sent an email, rolling back the database won’t bring it back. We’ll return to those actions later.
Now imagine the data is in two microservices, each with its own database. We’ve started transactions in DB1 and DB2 and made our changes. What next?
If we commit them one after another, the first may save its changes while the second fails. The first Commit is already complete. An ordinary Rollback won’t undo it, and the data will disagree.
One option is to keep a single database and return to a monolith. Everything would fit into one transaction, just as in the earlier diagrams.
That can work well for some tasks. But when an operation already involves independent services, we still need to agree how they’ll save their changes together.
Another option is a saga. Each service saves its own changes. If a later step fails, we run compensation: a separate operation to correct the result of an earlier step.
We have to write that compensation ourselves. It depends on the business logic: undoing a document creation, restoring a status or fixing other consequences. That’s useful for long-running processes, but here we want the familiar convenience of completing an operation or rolling back its changes.
Where does this come up? Camunda might manage a process while the document’s business data lives in another service. Completing a task also updates the document. We want both changes to be saved together.
The same issue arises with integrations or a custom microservice using the platform’s business data. One request may touch several databases even though the user sees a single operation.
We use two-phase commit, or 2PC, for this. XA is a standard for working with transactional resources. Let’s first look at the participants.
The transaction manager, TM, manages the transaction. An XA resource can prepare its changes, then commit or roll them back; a database with XA support is one example. We also need TM Repo, where the manager records information so it can work out what to do with an unfinished transaction after a failure.
First we make changes in both resources. When it’s time to finish, we don’t commit immediately. We ask each resource whether it’s ready to save its part. That’s Prepare.
If anyone isn’t ready, we roll back. If everyone is ready, we store the commit decision and tell the resources to Commit. Once finished, we can remove the coordination records.
Failures matter here. Once the commit decision has been recorded, recovery must carry it through. We can’t simply choose rollback anymore. Prepared resources may hold locks while waiting, which is a trade-off we need to account for.
Here’s an example with our services. A request enters through ecos-gateway and involves ecos-process and ecos-model. Each has its own PostgreSQL database containing business data.
The gateway delegates commit coordination to ecos-model, which also holds the transaction manager’s repository. ecos-model tracks the participants, checks their readiness and coordinates the decision they need to carry out.
How does another service know it was called within a transaction? We pass the transaction identifier along with the HTTP request. It tells the service which work the call belongs to.
We also pass instance routes. Here emodel maps to wbag7dya54yq, and eproc to za7iyygilrr4. With several running instances, the next call needs to reach the one that already holds our transaction’s state.
We need to learn routes from responses too. Suppose App A-0 calls App B-0, which then calls App C-1. A doesn’t yet know which C instance took part.
B-0 returns the route App C → C-1. If A later needs to call C in the same transaction, it can reach the right instance. Picking any instance through a load balancer could reach one that knows nothing about this transaction.
We return three things: transactional resources, routes and transactional actions. We’ve covered routes. The resource list tells the caller who needs to participate in completing the transaction.
Actions let a service say that some code must run at a particular moment, such as before or after commit. Let’s look at why we need that.
Suppose we need to recalculate permissions before saving changes, then publish an event or send an email afterward. If we send the email immediately and the transaction rolls back, someone gets notified about a change that never took effect.
So we register these actions beforehand and execute them in the right phase: permissions before commit, notifications after a successful commit.
What do we need to know about an action? The application where it’s registered, its identifier, when it should run and its order.
There are three timing types: before commit, after commit and after rollback. We can collect actions from different services and execute them in the required sequence.
Threads are another challenge. Application A calls B and waits for a response. But B may synchronously call back into A while working, in the same transaction.
A already has work in progress, with an occupied thread and its context. That matters particularly when the call returns to the same business process. Let’s look at an example.
We use WebApiThreadChain for this. While the original thread waits for an HTTP response, it can process nested calls from a queue. When a callback reaches A in the same transaction, we can hand it to that waiting thread. It does the work with the existing process context, then resumes the original call once B responds.
There are two subprocesses here. In the first, we take a task and set its status to “In progress at support tier 1”. We then cancel the SLA 1 notifications and complete SLA 1.
That status change also starts a second branch: checking transition conditions, notifying the initiator and handling the request. A simple status update takes us back into the process engine before the first branch has finished.
Across services, the call chain looks like this. ecos-process calls ecos-model to set the status. ecos-model sends a “Status changed” event, and we return to ecos-process to continue on the same thread.
The diagram shows a failed transition check. Here we need to know which process is running and which transaction we’re in. A transaction identifier alone isn’t enough if we lose the process execution context.
These cases need transactional event listeners. When another service must react to data creation, update or deletion as part of our operation, we call it synchronously over HTTP(S) in the current transaction.
To tell the sender whom to call, we keep a subscription registry in Zookeeper. An application registers the events it needs, its filter and the attributes to pass to its handler. The subscription also marks the listener as transactional. Services watch the registry and update their recipient lists when it changes.
The handler can then do its work before the transaction finishes. A notification that everything has been saved can be sent by a separate action after commit.
Now let’s look at the load test. We used JMeter and counted a non-200 response or a request taking more than 10 seconds as an error. Each thread used a separate user.
The system started with 100 000 users and 10 000 000 contracts. The workload ran as 10 000 users for one hour.
Users performed different tasks: logging in and opening the home page, 20%; browsing contracts, 30%; opening a contract page with all widgets, 30%.
They also viewed active tasks, 30%; created documents, 20%; started processes, 10%; and completed tasks, 50%. These are shares of users participating in each scenario. One user could perform several actions, so the percentages don’t need to add up to 100.
Over the hour we handled 20 447 914 requests. Of these, 221 failed or exceeded the time limit. The slide rounds this to 0.00%; more precisely, it’s about 0.00108%.
Mean response time was 21.88 ms, with a median of 6 ms. 90% of requests took less than 18 ms.
If you’d like to see how this works in code, the distributed transaction library’s source is available at the link.
We started with a simple goal: saving changes across several services together. That required two-phase commit, context and route propagation, then actions and listeners that work within that setup. It helps us write microservice code with the convenience we’re used to from a single database. Thank you!
