Service Cluster

In the previous topic, we discussed some fundamental aspects of Microservice . In this topic, we’ll explore how to operate microservices effectively to build a reliable system.

Cluster

Traditionally, we build and run a service as a single process. This approach may work initially. However, if the process or its host machine crashes, the entire service becomes unavailable.

Therefore, to improve resilience, we need to deploy the service as a cluster of multiple instances (processes), ideally distributed across different machines. This setup allows the service to remain operational even if some instances or machines fail.

For example, consider a service cluster with multiple instances distributed across two machines. If one instance, or even its entire machine, fails, the remaining instances can continue providing the service.

From this point onward, when we refer to a service, we mean a cluster comprising multiple instances.

Service State

To operate a service reliably, we need to understand its state. Two key metrics help us assess it: Health and Availability .

Health

Health refers to an instance’s ability to perform its intended tasks. An instance generally reports one of two health states:

• Healthy (Up): It is ready to accept and handle requests.
• Unhealthy (Down): The instance has encountered a problem, such as a database disconnection or hardware failure, and can no longer serve requests.

Health Interface

Typically, an instance exposes a health interface that reports its current status:

• Consumers, typically other services, can perform Health Check to ensure they are communicating with a healthy instance.

• The system can also use this interface to isolate unhealthy instances.

Heartbeat Mechanism

A common technique for isolating unhealthy instances is the heartbeat mechanism. Essentially, a health checker, such as a Load Balancer or DNS Server, periodically checks the health interfaces of service instances to identify and exclude faulty ones.

For example, suppose a health checker verifies the health of every instance every 5 seconds. If an instance is found to be unhealthy, the checker stops forwarding traffic to it.

Service Availability

Availability is a critical metric that represents the accessibility of a service from the user’s perspective.

For example, suppose a service depends on another service that is currently unavailable. From a technical perspective, the dependency is the source of the problem, while the service itself may still be running and healthy. However, users do not care which component failed. They simply observe that the requested functionality is unavailable.

This metric is essential when defining a Service Level Agreement (SLA). Availability is commonly calculated in two ways:

Time-based Availability

The first approach calculates the proportion of uptime relative to the total operating time within a given period, typically a year:

$Availability = \frac{Uptime}{Uptime + Downtime}$

For example, if a service runs for one year with approximately 3 days of downtime, its availability is:

$Availability = \frac{362}{362 + 3} \approx 99\%$

This approach assumes that requests are distributed relatively evenly over time. However, it is less sensitive to short but high-impact outages. A service may receive significantly more traffic during an outage than usual, causing the calculated availability to differ from the actual user experience.

Request-based Availability

Request-based Availability calculates availability based on the proportion of successful requests relative to the total number of requests:

$Availability = \frac{Successful\ requests}{Total\ requests}$

For example, suppose a service successfully handles 1000 out of 1010 requests:

$Availability = \frac{1000}{1010} = 99\%$

This approach can provide a more representative result, but it may also introduce bias:

• Highly active users have a disproportionately large impact on the metric, potentially overshadowing the experience of less active users.
• During outages, users often retry failed requests repeatedly, generating additional traffic that can make availability appear significantly worse.

Therefore, Request-based Availability is generally less suitable for public-facing services and more appropriate for controlled internal workloads.

Aggregate Availability

In the Microservice topic, we discussed several forms of design-time coupling that can negatively affect the development process. In production environments, runtime dependencies also emerge when services communicate over a network:

• Location Coupling: A service must know how to locate another service, such as through an IP address or domain name.
• Availability Coupling: When one service synchronously depends on another, its own availability is affected by the availability of that dependency. This is the critical form of coupling we’ll focus on here.

For example, the Subscription Service cannot complete its task without successfully communicating with the Account Service. Therefore, if the Account Service becomes unavailable, the Subscription Service is also affected.

Thus, the effective availability of a service depends on the availability of all services required to complete its operation.

$Availability = S (self) \times S1 \times S2 \times … \times Sn$

This interdependency can become a serious problem when service communication forms a complex dependency graph. Some services may become a Single Point of Failure , meaning that their failure can disrupt a large portion, or even all, of the system. For example, in this dependency graph, if C becomes unavailable, the entire request chain may stop functioning.

Availability Decoupling

We previously discussed how Messaging can decouple services in a Microservice architecture. Fortunately, Messaging can also reduce coupling at runtime.

For example, the Subscription Service may become unavailable when the Account Service is down because it cannot guarantee that requests will reach the Account Service.

By introducing Messaging , the Subscription Service can publish a message and continue its workflow without waiting for an immediate response. The Account Service can process the message later when it becomes available again. As a result, the availability of the Account Service no longer directly determines the availability of the Subscription Service.

This approach decouples the services at runtime, improving the system’s resilience and flexibility.

This approach is particularly valuable in systems containing many services. Services no longer depend directly on one another’s availability; even if some services fail temporarily, the remaining services can continue operating.

For example, in the following diagram, the effective availability of Service A is:

$(SA) = SA (self) \times SB \times SC$

With Messaging , the equation becomes:

$SA = SA (self) \times MessageBroker$

In practice, we have shifted the complex interdependencies between services to the message broker. The overall architecture becomes easier to manage because dependencies are now concentrated around the broker.

However, this introduces new trade-offs:

• The message broker can become a critical Single Point of Failure , so it must itself be highly available and fault-tolerant.
• Messaging also introduces several additional challenges, which were discussed in detail in the previous topic.

Cluster Types

Service clusters can generally be categorized into two types: Stateless and Stateful .

• Stateless services contain application logic but do not retain state between requests.
• Stateful services retain state, causing requests or connections to depend on information stored by a particular instance.

Stateless Service

A stateless service consists of instances that share identical logic and do not retain instance-specific local state.

For example, consider two instances of the Account Service. Both instances query the same database and are designed to produce identical results. The specific instance a client connects to does not matter because every instance uses the same logic and follows the same data-access pattern, ensuring a consistent response.

This consistency makes a stateless service relatively straightforward to scale: we simply increase or decrease the number of identical instances.

Stateful Service

Unlike a stateless service, a stateful service contains instances that may retain local state. As a result, different instances of the same service may behave differently depending on the state they currently hold.

Stateful services are often associated with real-time features, which require maintaining persistent client connections so the server can push messages to connected clients. A common example is a chat application that maintains client connections, typically through WebSocket, for real-time messaging.

Consider a cluster containing two instances. If Client A connects to Instance 1 while Client B connects to Instance 2, the two clients cannot communicate directly through local connections alone because each instance manages its own socket connections.

Scaling Problem

Stateful services are more difficult to scale and are generally best avoided when state can reasonably be externalized. Simply increasing the number of instances is not sufficient. Additional mechanisms are required to coordinate and share state across instances.

Centralized Cluster

One approach is to introduce a shared store that is accessible to all instances.

In the chat example, we introduce a shared component called the Presence Store, which maintains a mapping between users and the server instances they are currently connected to. Whenever a user connects to the system, the corresponding server instance creates or updates a presence record in this store.

By querying the shared store, service instances can determine where a particular user is connected and forward messages to the appropriate instance.

The simplicity of this design makes it a practical and commonly preferred solution.

However, this approach can reduce availability because the service now depends on an external shared store. Suppose we are developing a low-level service, such as a data store itself. In that case, the service may need to operate as a terminal component without relying on another centralized dependency.

In the Distributed Database topic, where a database cluster is treated as a stateful service, we’ll discuss decentralized architectures in which instances can coordinate and operate without relying on a central store.