Redis Sentinel: Reliable Distributed Database Solution

10 min readAug 25, 2024

Redis is widely used as a low-latency, high-performance data storage solution. However, relying on a single server can be problematic for systems that require high availability. Redis Sentinel addresses this by providing high availability and automatic failover management for Redis servers.

What is Redis Sentinel?

Redis Sentinel monitors Redis servers and activates automatically in case of server failures. It ensures high availability by using Redis’s master-slave replication. If the master server fails, Sentinel promotes the most suitable slave to master, maintaining continuous system operation.

Redis Sentinel Architecture

Redis Sentinel operates with master and slave servers:

Master: The primary server where write and read operations occur. Sentinel monitors this server and, if a problem arises, promotes one of the slave servers to be the new master. Sentinel always tracks the most current master server.
Slave: A backup server that replicates data from the master. It handles read operations and there can be multiple slaves in a Sentinel setup.

Key Components of Redis Sentinel

Monitoring: Sentinel continuously monitors Redis servers and considers a server down if it becomes unreachable.
Notification: When a problem is detected, Sentinel notifies system administrators and, if needed, informs other Sentinels.
Failover: If a master server fails, Sentinel automatically promotes a slave to master and ensures all other slaves reconnect to the new master.

The quorum value, representing the number of majority votes, is used by Sentinel to decide which slave should be promoted to master. Sentinel ensures the selected server has the most recent data compared to the failed master server.

Advantages of Redis Sentinel

Automatic Failover: Sentinel automatically handles failover when a master server fails, without requiring intervention from system administrators.
High Availability: Ensures high availability by maintaining continuous operation of the Redis system.
Flexible Structure: Enhances system reliability with a distributed setup, allowing multiple Sentinel servers to work together.

When Will Be Use Redis Sentinel

In Case of Redis Server Failure: Sentinel ensures continued service by switching to a different Redis server, providing uninterrupted operation if a single server fails or encounters an interruption.
During Maintenance and Updates: Sentinel maintains service continuity by using a different server if the Redis server becomes temporarily unavailable during maintenance or updates.
High Traffic: Sentinel helps scale Redis servers, improving performance and evenly distributing the load for better speed and stability in high-traffic applications.

Redis Cluster vs. Redis Sentinel

Redis Sentinel and Redis Cluster are two distinct Redis solutions that provide high availability and scalable distributed architectures.

Redis Sentinel

Redis Sentinel is a high availability solution that automatically manages and monitors failover scenarios in Redis. It ensures Redis remains accessible even if instances encounter issues.

Key Features:

Automatic Failover: Monitors Redis instances and promotes a replica to master if the current server fails.
Monitoring: Tracks replication lag, availability, and performance, and can trigger notifications or actions in case of errors.
Single Master-Replica: Supports multiple replicas copying data from a single master. If the master fails, Sentinel automatically promotes a slave to master.
Client Redirection: Can redirect clients to the appropriate master instance.

Redis Cluster

Redis Cluster is designed for replicating and sharding distributed data, allowing data to be spread across multiple Redis nodes.

Key Features:

Horizontal Scaling: Scales Redis horizontally by distributing data across multiple nodes.
Automatic Data Sharding: Automatically directs data to the appropriate node.
High Availability: Ensures high availability by selecting a new node if the main node fails and supports data replication and fault tolerance.
Data Resharding: Allows dynamic addition and removal of nodes without data loss.
No Client-Side Redirects: Ensures requests go directly to the appropriate node without redirection.
No Single Point of Failure: The system continues operating even if one node fails.

Redis Sentinel focuses on high availability and automatic failover of the master replica, while Redis Cluster is designed for horizontal scaling and data distribution across multiple nodes. The choice between them depends on the specific requirements of your application.

Sample Project

We want to create a highly available Redis setup that can automatically failover between two data centers without interruption. This application will read data from Redis, and if the master Redis instance fails, it will switch to the other data center.

Technologies Used

Redis Sentinel: For automatic failover and high availability.
Redis Server: The primary data store.
HAProxy: To provide a single entry point to Redis, handling failover and load balancing.
Docker: To containerize and deploy the Redis and Sentinel setup.
.NET Core Application: The application that interacts with Redis.

Architecture Overview

Data Centers: Two data centers (DC1 and DC2), each hosting a Redis instance and its corresponding Sentinel.
HAProxy: Deployed in front of the Redis instances to manage connections and ensure seamless failover.
Redis Sentinel: Monitors the Redis instances and handles the failover process.

What is HaProxy?
HAProxy (High Availability Proxy) is a load balancer and task manager. It provides services such as load balancing, high availability, and proxying for many applications, offering these services for free and securely.

Project Structures

redis-sentinel-haproxy/
├── haproxy/
│   └── haproxy.cfg
├── redis/
│   ├── redis1.conf
│   ├── redis2.conf
│   ├── sentinel1.conf
│   └── sentinel2.conf
├── dotnet-app/
│   ├── Program.cs
│   ├── Dockerfile
│   └── appsettings.json
└── docker-compose.yml

1. Redis Configuration

In the redis/ directory, you have configuration files for two Redis instances and two Sentinel instances:

redis1.conf:

port 6379    # Default redis port
bind 0.0.0.0 # Redis to accept connections from any IP address on any network 
             # interface

redis2.conf

port 6379    # Default redis port
bind 0.0.0.0 # Redis to accept connections from any IP address on any network 
             # interface

Important notes: Security issues are ignored in this demo project.

sentinel1.conf

port 26379 # Redis sentinel default port
bind 0.0.0.0

sentinel monitor mymaster 172.22.0.2 6379 2
# mymaster: The name used to refer to the monitored master Redis instance.
# redis1: Hostname of ip adress of the master redis server
# 6379: port number of the master redis server
# 2: The quorum, which is the number of Sentinel instances that must agree that the master is down before a failover is initiated.
sentinel down-after-milliseconds mymaster 5000
# This line defines the amount of time (in milliseconds) that Sentinel will wait before considering the master Redis instance as "down."
sentinel failover-timeout mymaster 10000
# This line sets the timeout for the failover process.
sentinel parallel-syncs mymaster 1
# This line controls how many slave Redis instances can synchronize with the new master simultaneously during a failover.

sentinel2.conf

port 26379 # Redis sentinel default port
bind 0.0.0.0

sentinel monitor mymaster 172.22.0.2 6379 2
# mymaster: The name used to refer to the monitored master Redis instance.
# redis1: Hostname of ip adress of the master redis server
# 6379: port number of the master redis server
# 2: The quorum, which is the number of Sentinel instances that must agree that the master is down before a failover is initiated.
sentinel down-after-milliseconds mymaster 5000
# This line defines the amount of time (in milliseconds) that Sentinel will wait before considering the master Redis instance as "down."
sentinel failover-timeout mymaster 10000
# This line sets the timeout for the failover process.
sentinel parallel-syncs mymaster 1
# This line controls how many slave Redis instances can synchronize with the new master simultaneously during a failover.

Notes: Specifically, in cases where docker cannot resolve the hostname, you can find the IP addresses of the relevant redis using the following command.

docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' redis1
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' redis2

2. HAProxy Configuration

In the haproxy/ directory, you have the HAProxy configuration file:

global # global settings section
    log stdout format raw local0
    # stdout: log standart
    # format raw: log format should be raw
    # local0: log destination

defaults # all proxies default settings
    log     global # enable global logging settings
    option  tcplog # detail logging for tcp connection
    timeout connect 5000ms
    timeout client  1m
    timeout server  1m

frontend redis_front
    bind *:6379 # HAProxy to listen for incoming connections on port 6379 on all available network interfaces
    default_backend redis_back # Specifies that all incoming connections on this frontend should be forwarded to the redis_back backend.

backend redis_back
    balance roundrobin # Sets the load balancing algorithm
    server redis1 redis1:6379 check
    # check redis healthcheck and change status is up and running
    server redis2 redis2:6379 checky
    # check redis healthcheck and change status is up and running

3. .NET Core Application

In the dotnet-app/ directory, you have the .NET Core application files:

Program.cs:

using System;
using System.Threading.Tasks;
using StackExchange.Redis;

class Program
{
    static async Task Main(string[] args)
    {
        var options = new ConfigurationOptions
        {
            EndPoints = { "haproxy:6379" },
            ConnectRetry = 5,
            ConnectTimeout = 5000,
            KeepAlive = 180, 
            AbortOnConnectFail = false 
        };

        var redis = await ConnectionMultiplexer.ConnectAsync(options);

        var db = redis.GetDatabase();

        while (true)
        {
            try
            {
                await db.StringSetAsync("test", "Hello, Redis!");

                var value = await db.StringGetAsync("test");
                Console.WriteLine($"Value from Redis: {value}");
            }
            catch (RedisConnectionException ex)
            {
                Console.WriteLine($"Redis connection error: {ex.Message}");
            }

            await Task.Delay(2000);
        }
    }
}

Dockerfile

FROM mcr.microsoft.com/dotnet/runtime:8.0 AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY . .
RUN dotnet restore
RUN dotnet build -c Release -o /app/build

FROM build AS publish
RUN dotnet publish -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "dotnet-app.dll"]

4. Docker Compose Configuration

In the root directory, you have the Docker Compose file:

docker-compose.yml

version: '3'

services:
  redis1:
    image: redis
    container_name: redis1
    volumes:
      - ./redis/redis1.conf:/usr/local/etc/redis/redis.conf
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
    networks:
      - redis-net

  redis2:
    image: redis
    container_name: redis2
    volumes:
      - ./redis/redis2.conf:/usr/local/etc/redis/redis.conf
    command: ["redis-server", "/usr/local/etc/redis/redis.conf"]
    networks:
      - redis-net

  sentinel1:
    image: redis
    container_name: sentinel1
    volumes:
      - ./redis/sentinel1.conf:/usr/local/etc/redis/sentinel.conf
    command: ["redis-server", "/usr/local/etc/redis/sentinel.conf", "--sentinel"]
    networks:
      - redis-net
    depends_on:
      - redis1
      - redis2

  sentinel2:
    image: redis
    container_name: sentinel2
    volumes:
      - ./redis/sentinel2.conf:/usr/local/etc/redis/sentinel.conf
    command: ["redis-server", "/usr/local/etc/redis/sentinel.conf", "--sentinel"]
    networks:
      - redis-net
    depends_on:
      - redis1
      - redis2

  haproxy:
    image: haproxy
    container_name: haproxy
    volumes:
      - ./haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg
    ports:
      - "6379:6379"
    networks:
      - redis-net
    depends_on:
      - sentinel1
      - sentinel2

  dotnet-app:
    build: ./dotnet-app
    container_name: dotnet-app
    depends_on:
      - haproxy
    networks:
      - redis-net

networks:
  redis-net:
    driver: bridge

Running the Setup

To run this setup:

Navigate to the root directory (redis-sentinel-haproxy/).
Run the following command to build and start all the services:

docker-compose up -d

This command will start the Redis servers, Sentinel instances, HAProxy, and your .NET Core application in detached mode.

Testing the Setup

You can test the failover mechanism by stopping the primary Redis server:

docker stop redis1

In a successful scenario, you would see an output similar to the following on the console:

haproxy     | xx.xx.x.x:xx [23/Aug/2024:06:19:38.871] redis_front redis_back/redis1 1/-1/16015 0 SC 2/2/1/0/3 0/0
haproxy     | xx.xx.x.x:xx [23/Aug/2024:06:18:55.101] redis_front redis_back/redis2 1/0/60033 98 cD 2/2/1/1/0 0/0
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
redis1      | 1:C 23 Aug 2024 06:20:29.501 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis1      | 1:C 23 Aug 2024 06:20:29.501 * Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=1, just started
redis1      | 1:C 23 Aug 2024 06:20:29.501 * Configuration loaded
redis1      | 1:M 23 Aug 2024 06:20:29.501 * monotonic clock: POSIX clock_gettime
redis1      | 1:M 23 Aug 2024 06:20:29.502 * Running mode=standalone, port=6379.
redis1      | 1:M 23 Aug 2024 06:20:29.502 * Server initialized
redis1      | 1:M 23 Aug 2024 06:20:29.502 * Loading RDB produced by version 7.4.0
redis1      | 1:M 23 Aug 2024 06:20:29.502 * RDB age 51 seconds
redis1      | 1:M 23 Aug 2024 06:20:29.502 * RDB memory usage when created 1.17 Mb
redis1      | 1:M 23 Aug 2024 06:20:29.502 * Done loading RDB, keys loaded: 1, keys expired: 0.
redis1      | 1:M 23 Aug 2024 06:20:29.502 * DB loaded from disk: 0.000 seconds
redis1      | 1:M 23 Aug 2024 06:20:29.502 * Ready to accept connections tcp
dotnet-app  | Value from Redis: Hello, Redis!
haproxy     | Server redis_back/redis1 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
haproxy     | [WARNING]  (8) : Server redis_back/redis1 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
redis2      | 1:signal-handler (1724394037) Received SIGTERM scheduling shutdown...
redis2      | 1:M 23 Aug 2024 06:20:37.533 * User requested shutdown...
redis2      | 1:M 23 Aug 2024 06:20:37.533 * Saving the final RDB snapshot before exiting.
redis2      | 1:M 23 Aug 2024 06:20:37.535 * DB saved on disk
redis2      | 1:M 23 Aug 2024 06:20:37.535 # Redis is now ready to exit, bye bye...
haproxy     | xx.xx.x.x:xx[23/Aug/2024:06:19:55.133] redis_front redis_back/redis2 1/0/42402 2151 -- 2/2/0/0/0 0/0
haproxy     | xx.xx.x.x:xx [23/Aug/2024:06:19:55.137] redis_front redis_back/redis2 1/0/42399 98 -- 2/2/0/0/0 0/0
redis2 exited with code 0
redis2 exited with code 0
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!
dotnet-app  | Value from Redis: Hello, Redis!

Your application should continue to function normally, with HAProxy directing traffic to the promoted master Redis server. This setup provides a robust and highly available Redis environment with automatic failover managed by Redis Sentinel, and load balancing and failover redirection handled by HAProxy.

Redis Sentinel offers an effective solution for ensuring high availability and uninterrupted service for Redis servers. It is especially valuable for large-scale and critical applications due to its automatic failover capabilities. With proper configuration and regular monitoring, Redis Sentinel enhances the reliability and resilience of your Redis infrastructure.