Knowledge Base

Redundancy

Redundancy involves duplicating critical components or data so that if one fails, a backup can take over. This is the cornerstone of fault tolerance, ensuring no single point of failure.

• Hardware Redundancy: Multiple servers, power supplies, network interfaces.
• Software Redundancy: Running multiple instances of an application or service.
• Data Redundancy: Replicating databases or critical data across different locations.

Failover

Failover is the process of automatically switching to a redundant or standby system upon the failure or unusual termination of the primary system. This switch should ideally be seamless to the end-user.

• Automatic Failover: Systems that detect failures and initiate a switch without human intervention.
• Manual Failover: Requires human input to initiate the switch.

Recovery

Recovery is the process of restoring a system to a functional state after a failure. This can involve restarting services, restoring data from backups, or rebuilding components.

• Fast Recovery: Minimizing downtime by quickly restoring service.
• Data Consistency: Ensuring data is consistent and not corrupted after recovery.

Detection

The ability of a system to accurately and promptly detect component failures. Effective detection is a prerequisite for timely failover and recovery.

• Heartbeats: Periodic messages exchanged between components to indicate they are alive.
• Health Checks: Probes that check the availability and responsiveness of services.
• Monitoring and Alerting: Tools that track system metrics and notify operators of anomalies.

Replication

Maintaining multiple copies of data or services. Different replication strategies exist, each with trade-offs:

• Active-Passive: One instance handles requests, while others are on standby.
• Active-Active: All instances handle requests concurrently. Requires careful synchronization.
• Quorum-Based Replication: Ensures a majority of replicas acknowledge an operation before it's considered successful.

Example (Conceptual): In a distributed key-value store, writing a value might require acknowledgment from at least N/2 + 1 replicas (where N is the total number of replicas) for durability.

// Conceptual replication logic
function writeToReplicas(key, value, totalReplicas, quorumSize) {
    let acknowledgedCount = 0;
    const replicas = getReplicas(); // Assume this returns an array of replica clients

    replicas.forEach(replica => {
        replica.write(key, value).then(() => {
            acknowledgedCount++;
            if (acknowledgedCount >= quorumSize) {
                console.log(`Write successful, quorum reached.`);
                // Proceed with next steps or acknowledge client
            }
        }).catch(error => {
            console.error(`Replica failed: ${error}`);
            // Handle replica failure (e.g., mark as unhealthy)
        });
    });
}
                

Load Balancing

Distributing incoming network traffic across multiple servers or resources. This not only improves performance but also acts as a fault tolerance mechanism.

• Round Robin: Distributes requests sequentially.
• Least Connections: Sends requests to the server with the fewest active connections.
• IP Hash: Directs requests from the same IP address to the same server.

Load balancers typically perform health checks on backend servers, removing unhealthy ones from the pool of available resources.

Circuit Breakers

A design pattern used to stop further calls to a service that is exhibiting errors. It prevents cascading failures by stopping the client from repeatedly trying to access a failing service.

• Closed State: Normal operation; requests pass through.
• Open State: After a threshold of failures, requests are immediately rejected without attempting to call the service.
• Half-Open State: After a timeout, a limited number of requests are allowed through to test if the service has recovered.

Example (Conceptual): A consumer service calling a recommendation API. If the recommendation API returns errors for more than 5 consecutive calls, the circuit breaker trips, and subsequent calls are immediately failed for 60 seconds.

class CircuitBreaker {
    constructor(failureThreshold, resetTimeout) {
        this.failureThreshold = failureThreshold;
        this.resetTimeout = resetTimeout;
        this.state = 'CLOSED'; // CLOSED, OPEN, HALF_OPEN
        this.failureCount = 0;
        this.lastFailureTime = null;
        this.successCountAfterHalfOpen = 0;
    }

    async execute(action) {
        if (this.state === 'OPEN') {
            const currentTime = Date.now();
            if (currentTime - this.lastFailureTime > this.resetTimeout) {
                this.state = 'HALF_OPEN';
                this.successCountAfterHalfOpen = 0;
                console.log("Circuit Breaker: State changed to HALF_OPEN");
            } else {
                throw new Error("Circuit is open. Service unavailable.");
            }
        }

        if (this.state === 'HALF_OPEN') {
            try {
                const result = await action();
                this.successCountAfterHalfOpen++;
                if (this.successCountAfterHalfOpen >= 2) { // Example: Reset after 2 successes
                    this.state = 'CLOSED';
                    this.failureCount = 0;
                    console.log("Circuit Breaker: State changed to CLOSED");
                }
                return result;
            } catch (error) {
                this.state = 'OPEN';
                this.lastFailureTime = Date.now();
                console.error("Circuit Breaker: State changed to OPEN due to failure in HALF_OPEN");
                throw error;
            }
        }

        // State is CLOSED
        try {
            const result = await action();
            this.failureCount = 0; // Reset count on success
            return result;
        } catch (error) {
            this.failureCount++;
            if (this.failureCount >= this.failureThreshold) {
                this.state = 'OPEN';
                this.lastFailureTime = Date.now();
                console.log(`Circuit Breaker: State changed to OPEN after ${this.failureCount} failures.`);
            }
            throw error;
        }
    }
}
                

Idempotency

An operation is idempotent if calling it multiple times with the same parameters has the same effect as calling it once. This is crucial for retries in distributed systems.

• If a request times out, a client can safely retry it without causing unintended side effects.
• Commonly achieved using unique request IDs that are tracked by the server.

Example: A payment processing API should be idempotent. If a 'charge $10' request is sent twice, it should only charge the customer once.

Graceful Degradation

When a system experiences partial failure or overload, it should continue to operate with reduced functionality rather than failing completely. Users receive a degraded but still usable experience.

• Disabling non-essential features.
• Returning cached data instead of real-time data.
• Prioritizing critical user actions.

Example: An e-commerce site might disable personalized recommendations during high traffic periods, but still allow users to browse and purchase products.

Leader Election

In distributed systems where a single component needs to act as a leader (e.g., for coordination or task distribution), a leader election mechanism ensures that if the current leader fails, a new leader is automatically chosen from the remaining nodes.

• Often uses consensus algorithms like Raft or Paxos.
• Ensures only one leader exists at a time.

Distributed Transactions

Ensuring that a transaction that spans multiple distributed services either commits on all participating services or aborts on all participating services (atomicity).

• Two-Phase Commit (2PC): A common, but often complex and blocking, protocol.
• Sagas: A sequence of local transactions where each transaction has a compensating transaction to undo its effects. More resilient than 2PC.

Distributed transactions are notoriously difficult to implement correctly and efficiently. Often, systems are designed to avoid them where possible by using eventual consistency models.

Consensus Algorithms

Algorithms that allow a group of distributed nodes to agree on a single value or state, even in the presence of failures or network delays. Essential for tasks like leader election and maintaining replicated state.

• Paxos: A family of protocols for reaching consensus.
• Raft: Designed to be more understandable than Paxos, widely used in systems like etcd and ZooKeeper.

These algorithms provide strong guarantees about consistency but often come with performance overhead.