Compensation Protocols

This section explores simpler protocols that operate at a higher level of abstraction, without relying on low-level concepts.

Compensation Protocols enable transactions to commit data in a reversible manner. If an issue arises later, these transactions can roll back the system by compensating for the previously committed changes.

Designing these protocols is challenging because they don’t inherently meet the Isolation requirement. The Commit and Compensate phases often behave as if they are parts of separate transactions. If the design is flawed, other operations might update data before the Compensate phase. This can lead to operations using inconsistent data and producing further inconsistencies. Furthermore, this separation makes it impossible to provide Strong Consistency, a strict requirement in some critical systems.

Try-Confirm/Cancel (TCC)

Try-Confirm/Cancel (TCC) is an approach similar to Two-Phase Commit (2PC) but operates without locking mechanisms. It also involves two phases coordinated by a central entity:

1. Try: The coordinator instructs participants to perform tentative actions, such as reserving resources. Data is actually committed during this phase, not merely marked as dirty data.

2. Confirm or Cancel:

   • Confirm: If all participants successfully complete their Try actions, the coordinator asks them to confirm and finalize the remaining operations.
   • Cancel: If any participant fails to prepare, other participants will revert their changes.

Consider a transaction transferring money between accounts located on different servers:

You might wonder why Account A’s balance is updated in the Try phase, but Account B’s is not. The Try phase must not introduce harmful effects into the system. It would be a poor design to increase Account B’s balance initially, as these temporarily credited funds shouldn’t be available for use until the transaction is validated.

Similar to 2PC, the primary advantage of TCC is its support for parallel processing, allowing steps to be performed simultaneously. However, TCC creates tight coupling between services because they need to expose low-level Try-Confirm-Cancel interfaces. This requires a deep understanding of the participating services.

Saga Pattern

Saga can be considered the most straightforward solution for distributed transactions. In a saga, transactions are broken down into a series of steps executed sequentially. If any step fails, the process moves backward to compensate for the preceding successful steps.

For instance, imagine an e-commerce system with three services. When a user places an order with the Order Service:

1. The Order Service creates the order record.
2. The Stock Service reduces the available stock of the ordered items.
3. The Payment Service completes the payment.

There are two primary ways to implement this workflow using Saga:

Orchestration Saga

In this approach, a central coordinator is established, much like in TCC or 2PC. The coordinator executes the actions in sequence. If any action fails, it coordinates compensation for previous actions in reverse order.

When all steps succeed, the process might look like this:

If, for example, the payment step fails, other services will be instructed to compensate for the previous steps:

This seems straightforward, as it mirrors real-life sequential processes without partial operations or locking.

Again, why is the payment processed last? Payment processing often relies on third-party solutions. Executing the payment earlier would make compensation more difficult. Therefore, internal tasks are typically completed first.

However, this sequential nature means the Saga pattern does not inherently support parallelism. A transaction’s operations must be performed in order, even if some could benefit from parallel execution.

Choreography Saga

Choreography Saga implements the pattern asynchronously. A transaction is completed as services react to events exchanged throughout the workflow.

Continuing with the e-commerce example, services communicate implicitly through an event stream:

In case of a failure, compensation events are generated. For example, if the payment step fails, it triggers a chain of compensation events for the preceding steps:

Orchestration requires a central coordinator and direct communication, which can degrade availability and introduce a Single Point of Failure . However, it conveniently centralizes the transaction logic within the coordinator. In contrast, Choreography distributes the transaction logic across different services, potentially making the source code difficult to understand and maintain.

Although Choreography can enhance availability and decouple the system, its complexity can be overwhelming and may outweigh the benefits. Moreover, Saga is inherently slower due to its sequential nature, and the Choreography approach can increase this overhead through indirect communication.

Saga Modelling

As stated earlier, Saga is a compensation protocol and does not offer the Isolation property. Before compensation occurs, dirty data can create vulnerabilities in the system. Therefore, the most critical aspect of Saga is defining a safe and reliable workflow.

1. First, assume that any step can fail. When a step fails, we must ensure it doesn’t jeopardize the system.

2. Second, not every step in a workflow can be compensated, especially when external factors are involved. Imagine a step that transfers money to a user’s bank account. Reverting this step would require retrieving the amount from the bank, which is nearly impossible as a bank typically won’t permit withdrawals without user consent.

In a saga, actions should be categorized into three groups:

1. Compensable Action: Can be rolled back using a corresponding compensation action if necessary. These are usually internal operations whose effects can readily be reversed.
2. Pivot Action: Represents a point of no return in a workflow. Once it succeeds, all subsequent actions must be completed, and no compensation can be applied. It’s typically the last Compensable action in a sequence.
3. Retryable Action: Can be safely retried multiple times without causing inconsistencies. This action is irreversible and is expected to be retried until successful.

Saga workflows are designed as follows: After the pivot step is completed, compensation is no longer an option.

Returning to the e-commerce example, we’ll arrange the steps as follows:

1. The Order Service creates an order using CreateOrder().
2. The Stock Service reserves the ordered items using ReserveItems().
3. The Payment Service processes the payment using ProcessPayment().
4. Finally, the Delivery Service creates a delivery request using CreateDeliveryRequest().

The first two steps are internal processes, so they are safe to compensate. Payment and delivery operations usually depend on third-party solutions, so it may not be feasible to reverse them. After the payment step succeeds, we must guarantee completion of the final step.

However, this workflow varies based on the system. If the delivery task is handled internally, we may place it before the payment step.

Saga Serialization Anomalies

Multiple sagas can execute concurrently and modify the same data. This can lead to serialization anomalies, like those discussed in the Concurrency Control topic.

We’ll briefly consider some common anomalies and how to resolve them.

Dirty Read

The first anomaly is Dirty Read. Similar to a rollback, the compensation phase can cause data to become dirty for transactions that read it before compensation.

Consider an example: when a user places a large order, they receive a discount voucher.

Unfortunately, the final step (payment) fails. The system needs to revert the second step by deleting the voucher. However, before the deletion occurs, another saga might use that voucher.

The result is unexpected because an invalid voucher was used. There are several ways to resolve this:

1. Rearrange the flow: Move the voucher creation to the Retryable phase. Although it’s an internal and compensable operation, its potential for causing issues makes it safer in the Retryable phase.

   This solution is known as the Pessimistic View, which assumes that actions prone to causing harm are likely to be compensated and should thus be deferred to a later point.

2. Lock data: Use a status field that acts as a lock. For example, when a new voucher is created, its state is set to Pending, marking it as unavailable for use. When the saga that created the voucher completes successfully, it changes the state to Completed, allowing the voucher to be used.

   This solution is called Semantic Locking, where application-level locking is implemented to prevent anomalies.

The second approach may add unnecessary overhead in this example. The next anomaly illustrates its benefits more clearly.

Lost Update

The next anomaly is Lost Update, where updates are unexpectedly overwritten by other concurrent operations.

For example, the OrderService creates an order, which is marked Completed at the end of its saga. However, in the meantime, a CancelOrder Saga executes and sets the order’s state to Cancelled. Consequently, the CreateOrder Saga might overwrite the change from the CancelOrder saga, resulting in a cancelled order still being processed by the system.

Semantic Locking is an effective approach for preventing Lost Update. As before, orders are locked with a Pending state until they are successfully created.

Semantic Locking is nothing but a lock, reducing parallelism and negatively affecting performance. We should prioritize designing a reliable workflow and minimizing locking.

Saga Transaction Recovery

Transaction State

Both the coordinator and participating services can crash at any time. Therefore, they must persist the state of transactions to enable retries if necessary.

Making transactions idempotent (where each transaction is marked with a unique identifier) is an effective way to prevent duplicate processing and support transaction recovery after a failure.

Coordinator Recovery

The coordinator service may periodically scan its state store to complete any unfinished transactions. Continuing with the previous example, the Order Service keeps track of the current transaction step:

Order Service (coordinator):
  transaction-1234:
    Step: Payment Service
    State: Processing

For added safety, the transaction state is also duplicated in the participating services:

Order Service (coordinator):
  transaction-1234:
    Step: Payment Service
    State: Processing
Stock Service:
  transaction-1234: Complete
Payment Service:
  transaction-1234: Complete

For example, if the Order Service (coordinator) fails before receiving a response from the Payment Service, this can lead to mismatched states (Processing and Complete) across different services.

After restarting, the Order Service will observe that the transaction is pending and will attempt to call the Payment Service again to complete it. Fortunately, duplicating the state ensures that the Payment Service will not reprocess the transaction.

Choreographer Recovery

Consume-process-produce Pipeline

This pattern was discussed in the Event Streaming section. The Choreography Saga helps to avoid duplication without relying on idempotency in this specific pipeline.

The Consume-Process-Produce Pipeline is applicable when a service’s changes are confined to the event stream and do not affect other datasets.

Transactional Outbox Pattern

Another challenge with Choreography Saga is ensuring that database changes are reliably published as events. For example, when a transaction is executed in an internal data store, we want to publish an associated event indicating its completion.

The Transactional Outbox Pattern addresses this by storing external calls (event publications) as part of the internal database transaction.

For example, the transaction that creates an order also creates an associated record in the Outbox table, instead of immediately publishing the event.

A separate process (called a Relay Process) periodically scans the Outbox table and publishes any unprocessed events.

If the relay process crashes before updating the outbox record in the third step, it might publish the event twice. Fortunately, consumers can use an idempotency key (e.g., OrderId) to detect and ignore duplicate events.