使用Sentinel对Redis进行监控

2 年 ago

宇, 华

7 minutes

首先

以下是描述如何通过Sentinel来监视Redis的方法。

当Sentinel监视Redis复制状态并检测到主节点宕机时，将任意一个从节点晋升为主节点。

环境

Redis 2.8.16

组成

Redisのレプリケーション(master x 1台, slave x 2台)を3台のSentinelで監視する構成とします。

役割名称ポートmasterredis6379slaveredis026380slaveredis036381sentinelsentinel26379sentinelsentinel0226380sentinelsentinel0326381

安装Redis

$ wget http://download.redis.io/releases/redis-2.8.16.tar.gz
$ tar zxvf redis-2.8.16.tar.gz
$ cd cd redis-2.8.16
$ make 2>&1 | tee make.log
$ make test 2>&1 | tee make-test.log
$ sudo sh -c "PREFIX=/usr/local/redis make install 2>&1 | tee make-install.log"

$ sudo mkdir -p /usr/local/redis/conf
$ sudo mkdir -p /usr/local/redis/data/6379 /usr/local/redis/data/6380 /usr/local/redis/data/6381

Redis的配置

redis(master)の設定

$ sudo cp redis.conf /usr/local/redis/conf/
$ vi /usr/local/redis/conf/redis.conf
$ diff redis.conf /usr/local/redis/conf/redis.conf
37c37
< daemonize no
---
> daemonize yes
103c103
< logfile ""
---
> logfile /var/log/redis.log
187c187
< dir ./
---
> dir /usr/local/redis/data/6379

redis(master)の起動

$ sudo /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf
$ /usr/local/redis/bin/redis-cli　info replication

# Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

redis02(slave)の設定

$ sudo cp /usr/local/redis/conf/redis.conf /usr/local/redis/conf/redis-6380.conf
$ sudo vi /usr/local/redis/conf/redis-6380.conf
$ diff /usr/local/redis/conf/redis.conf /usr/local/redis/conf/redis-6380.conf
45c45
< port 6379
---
> port 6380
103c103
< logfile /var/log/redis.log
---
> logfile /var/log/redis-6380.log
187c187
< dir /usr/local/redis/data/6379
---
> dir /usr/local/redis/data/6380
205a206
> slaveof 127.0.0.1 6379

* redis03(slave)の設定

$ sudo cp /usr/local/redis/conf/redis.conf /usr/local/redis/conf/redis-6381.conf
$ sudo vi /usr/local/redis/conf/redis-6381.conf
$ diff /usr/local/redis/conf/redis.conf /usr/local/redis/conf/redis-6381.conf
45c45
< port 6379
---
> port 6381
103c103
< logfile /var/log/redis.log
---
> logfile /var/log/redis-6381.log
187c187
< dir /usr/local/redis/data/6379
---
> dir /usr/local/redis/data/6381
205a206
> slaveof 127.0.0.1 6379

redis02(slave)を起動し、レプリケーションの状態を確認します。

$ sudo /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis-6380.conf
$ /usr/local/redis/bin/redis-cli -p 6380 info replication

# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:29
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

redis03(slave)を起動し、レプリケーションの状態を確認します。

$ sudo /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis-6381.conf
$ /usr/local/redis/bin/redis-cli -p 6381 info replication

# Replication
role:slave
master_host:127.0.0.1
master_port:6379
master_link_status:up
master_last_io_seconds_ago:6
master_sync_in_progress:0
slave_repl_offset:197
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

redis(master)へ接続してレプリケーションの状態を確認します。

$ /usr/local/redis/bin/redis-cli info replication

# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=6380,state=online,offset=281,lag=1
slave1:ip=127.0.0.1,port=6381,state=online,offset=281,lag=1
master_repl_offset:281
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:280

哨兵的设置

sentinelの設定及び起動

$ sudo cp sentinel.conf /usr/local/redis/conf/
$ diff sentinel.conf /usr/local/redis/conf/sentinel.conf
181c181,182
<
---
> daemonize yes
> logfile "/var/log/redis-sentinel.log"

$ sudo /usr/local/redis/bin/redis-server /usr/local/redis/conf/sentinel.conf --sentinel

監視状態の確認

$ /usr/local/bin/redis/redis-cli -p 26379
127.0.0.1:26379> sentinel masters
1)  1) "name"
    2) "mymaster"
    3) "ip"
    4) "127.0.0.1"
    5) "port"
    6) "6379"
    7) "runid"
    8) "0d2111054628f7a654e2a43fbb6157d4f7563821"
    9) "flags"
   10) "master"
   11) "pending-commands"
   12) "0"
   13) "last-ping-sent"
   14) "0"
   15) "last-ok-ping-reply"
   16) "39"
   17) "last-ping-reply"
   18) "39"
   19) "down-after-milliseconds"
   20) "30000"
   21) "info-refresh"
   22) "1928"
   23) "role-reported"
   24) "master"
   25) "role-reported-time"
   26) "153550"
   27) "config-epoch"
   28) "0"
   29) "num-slaves"
   30) "2"
   31) "num-other-sentinels"
   32) "0"
   33) "quorum"
   34) "2"
   35) "failover-timeout"
   36) "180000"
   37) "parallel-syncs"
   38) "1"
127.0.0.1:26379> sentinel slaves mymaster
1)  1) "name"
    2) "127.0.0.1:6380"
    3) "ip"
    4) "127.0.0.1"
    5) "port"
    6) "6380"
    7) "runid"
    8) "9dbefed61658340d2da2dc041081a2ac7f7398ec"
    9) "flags"
   10) "slave"
   11) "pending-commands"
   12) "0"
   13) "last-ping-sent"
   14) "0"
   15) "last-ok-ping-reply"
   16) "934"
   17) "last-ping-reply"
   18) "934"
   19) "down-after-milliseconds"
   20) "30000"
   21) "info-refresh"
   22) "2013"
   23) "role-reported"
   24) "slave"
   25) "role-reported-time"
   26) "162731"
   27) "master-link-down-time"
   28) "0"
   29) "master-link-status"
   30) "ok"
   31) "master-host"
   32) "127.0.0.1"
   33) "master-port"
   34) "6379"
   35) "slave-priority"
   36) "100"
   37) "slave-repl-offset"
   38) "12288"
2)  1) "name"
    2) "127.0.0.1:6381"
    3) "ip"
    4) "127.0.0.1"
    5) "port"
    6) "6381"
    7) "runid"
    8) "f80c280e2b2bcb70429ecf235bfcbf341599dbc3"
    9) "flags"
   10) "slave"
   11) "pending-commands"
   12) "0"
   13) "last-ping-sent"
   14) "0"
   15) "last-ok-ping-reply"
   16) "934"
   17) "last-ping-reply"
   18) "934"
   19) "down-after-milliseconds"
   20) "30000"
   21) "info-refresh"
   22) "2013"
   23) "role-reported"
   24) "slave"
   25) "role-reported-time"
   26) "162731"
   27) "master-link-down-time"
   28) "0"
   29) "master-link-status"
   30) "ok"
   31) "master-host"
   32) "127.0.0.1"
   33) "master-port"
   34) "6379"
   35) "slave-priority"
   36) "100"
   37) "slave-repl-offset"
   38) "12288"
127.0.0.1:26379> quit

sentinel02(sentinel)の設定

$ sudo cp /usr/local/redis/conf/sentinel.conf /usr/local/redis/conf/sentinel-26380.conf
$ sudo vi /usr/local/redis/conf/sentinel-26380.conf
$ diff /usr/local/redis/conf/sentinel.conf /usr/local/redis/conf/sentinel-26380.conf
5c5
< port 26379
---
> port 26380
182c182
< logfile "/var/log/redis-sentinel.log"
---
> logfile "/var/log/redis-sentinel-26380.log"

sentinel03(sentinel)の設定

$ sudo cp /usr/local/redis/conf/sentinel.conf /usr/local/redis/conf/sentinel-26381.conf
$ sudo vi /usr/local/redis/conf/sentinel-26381.conf
$ diff /usr/local/redis/conf/sentinel.conf /usr/local/redis/conf/sentinel-26381.conf
5c5
< port 26379
---
> port 26381
182c182
< logfile "/var/log/redis-sentinel.log"
---
> logfile "/var/log/redis-sentinel-26381.log"

sentinel02(sentinel)の起動

$ sudo /usr/local/redis/bin/redis-server /usr/local/redis/conf/sentinel-26380.conf --sentinel

sentinel03(sentinel)の起動

$ sudo /usr/local/redis/bin/redis-server /usr/local/redis/conf/sentinel-26381.conf --sentinel

确认动作

redis(master)を停止します。

$ sudo lsof -i :6379
COMMAND     PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
redis-ser 32226 root    4u  IPv6 480880      0t0  TCP *:6379 (LISTEN)
redis-ser 32226 root    5u  IPv4 480882      0t0  TCP *:6379 (LISTEN)
redis-ser 32226 root    6u  IPv4 480930      0t0  TCP localhost:6379->localhost:50787 (ESTABLISHED)
redis-ser 32226 root    7u  IPv4 480988      0t0  TCP localhost:6379->localhost:50790 (ESTABLISHED)
redis-ser 32226 root    8u  IPv4 481221      0t0  TCP localhost:6379->localhost:50793 (ESTABLISHED)
redis-ser 32226 root    9u  IPv4 481222      0t0  TCP localhost:6379->localhost:50794 (ESTABLISHED)
redis-ser 32226 root   10u  IPv4 481537      0t0  TCP localhost:6379->localhost:50800 (ESTABLISHED)
redis-ser 32226 root   11u  IPv4 481538      0t0  TCP localhost:6379->localhost:50801 (ESTABLISHED)
redis-ser 32226 root   12u  IPv4 481609      0t0  TCP localhost:6379->localhost:50808 (ESTABLISHED)
redis-ser 32226 root   13u  IPv4 481610      0t0  TCP localhost:6379->localhost:50809 (ESTABLISHED)
redis-ser 32232 root    6u  IPv4 480929      0t0  TCP localhost:50787->localhost:6379 (ESTABLISHED)
redis-ser 32242 root    6u  IPv4 480987      0t0  TCP localhost:50790->localhost:6379 (ESTABLISHED)
redis-ser 32278 root    6u  IPv4 481218      0t0  TCP localhost:50793->localhost:6379 (ESTABLISHED)
redis-ser 32278 root    7u  IPv4 481220      0t0  TCP localhost:50794->localhost:6379 (ESTABLISHED)
redis-ser 32308 root    6u  IPv4 481534      0t0  TCP localhost:50800->localhost:6379 (ESTABLISHED)
redis-ser 32308 root    7u  IPv4 481536      0t0  TCP localhost:50801->localhost:6379 (ESTABLISHED)
redis-ser 32313 root    6u  IPv4 481606      0t0  TCP localhost:50808->localhost:6379 (ESTABLISHED)
redis-ser 32313 root    7u  IPv4 481608      0t0  TCP localhost:50809->localhost:6379 (ESTABLISHED)

$ sudo kill 32226

sentinel(sentinel)のログを確認

[32278] 17 Sep 05:52:41.546 # +sdown master mymaster 127.0.0.1 6379
[32278] 17 Sep 05:52:41.617 # +odown master mymaster 127.0.0.1 6379 #quorum 3/2
[32278] 17 Sep 05:52:41.649 # +new-epoch 1
[32278] 17 Sep 05:52:41.649 # +try-failover master mymaster 127.0.0.1 6379
[32278] 17 Sep 05:52:41.714 # +vote-for-leader 413cec268653074dcc0c83797f7b338517843ba0 1
[32278] 17 Sep 05:52:41.729 # 127.0.0.1:26381 voted for 55865d34479acc6872b084fb401c2e4fd220b707 1
[32278] 17 Sep 05:52:41.776 # 127.0.0.1:26380 voted for 55865d34479acc6872b084fb401c2e4fd220b707 1
[32278] 17 Sep 05:52:42.823 # +config-update-from sentinel 127.0.0.1:26381 127.0.0.1 26381 @ mymaster 127.0.0.1 6379
[32278] 17 Sep 05:52:42.824 # +switch-master mymaster 127.0.0.1 6379 127.0.0.1 6380
[32278] 17 Sep 05:52:42.824 * +slave slave 127.0.0.1:6381 127.0.0.1 6381 @ mymaster 127.0.0.1 6380
[32278] 17 Sep 05:52:42.974 * +slave slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6380
[32278] 17 Sep 05:53:13.007 # +sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6380

レプリケーションの状態を確認

$ /usr/local/redis/bin/redis-cli -p 6380 info replication

# Replication
role:master
connected_slaves:1
slave0:ip=127.0.0.1,port=6381,state=online,offset=41135,lag=0
master_repl_offset:41135
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:41134

$ /usr/local/redis/bin/redis-cli -p 6381 info replication

# Replication
role:slave
master_host:127.0.0.1
master_port:6380
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:45433
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

redis(もとのmaster)を起動する

$ sudo /usr/local/redis/bin/redis-server /usr/local/redis/conf/redis.conf

sentinel(sentienl)のログを確認します。

[32278] 17 Sep 05:58:38.187 # -sdown slave 127.0.0.1:6379 127.0.0.1 6379 @ mymaster 127.0.0.1 6380

再度、レプリケーションの状態を確認します。

$ /usr/local/redis/bin/redis-cli info replication

# Replication
role:slave
master_host:127.0.0.1
master_port:6380
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
slave_repl_offset:102919
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

$ /usr/local/redis/bin/redis-cli info replication -p 6380

# Replication
role:master
connected_slaves:2
slave0:ip=127.0.0.1,port=6381,state=online,offset=114973,lag=1
slave1:ip=127.0.0.1,port=6379,state=online,offset=114973,lag=1
master_repl_offset:114973
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:2
repl_backlog_histlen:114972

仅提供一个选项，请将以下内容翻译成中文：参考

Redis Sentinel Documentation