Radosgw 多 region 部署以及 radosgw-agent 同步

2016-01-26

介绍

接着 Radosgw 多 zone 部署以及 radosgw-agent 同步 部署两个 region 的集群。

整体的结构是：

  • cd 作为 Master Region，cd-1 作为该 Region 中的 Master Zone，cd-2 作为该 Region 中的 Secondary Zone
  • bj 作为 Secondary Region，bj-1 作为该 Region 中的 Master Zone，bj-2 作为该 Region 中的 Secondary Zone

region 之间的同步，只能是 master region 的 master zone 作为源，secondary region 的 master zone 作为目的。

部署

前提已经部署了两个单 region、多 zone 的 ceph 集群，并分别在四个节点上 部署了 rgw。

  1. 修改新部署的 bj 集群的 region file

    bj.json

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    { “name“: “bj”,
     “api_name“: “bj”,
     “is_master“: “false”,
     “endpoints“: [
     “http:\/\/bj-1:80\/“],
     “master_zone“: “bj-center1”,
     “zones“: [
     { “name“: “bj-center1”,
     “endpoints“: [
     “http:\/\/bj-1:80\/“],
     “log_meta“: “true”,
     “log_data“: “true”},
     { “name“: “bj-center2”,
     “endpoints“: [
     “http:\/\/bj-2:80\/“],
     “log_meta“: “true”,
     “log_data“: “true”}],
     “placement_targets“: [
     {
     “name“: “default-placement”,
     “tags“: []
     }
     ],

    注意该 region 作为 secondary region，"is_master": "false"

  2. 配置

    每个节点相互拷贝 region、zone 配置文件，保证每个节点的配置文件如下：

    1
    2
    3
    4
    5
    6
    7
    8
    9
    # ll /etc/ceph/
    总用量 40
    -rw-r–r– 1 root root 793 1月 26 16:33 bj-center1.json
    -rw-r–r– 1 root root 793 1月 26 16:33 bj-center2.json
    -rw-r–r– 1 root root 676 1月 26 16:33 bj.json
    -rw-r–r– 1 root root 793 1月 12 15:30 cd-center1.json
    -rw-r–r– 1 root root 793 1月 18 14:02 cd-center2.json
    -rw-r–r– 1 root root 682 1月 18 12:15 cd.json
    …

    在 secondary region 两个节点上创建 master region、master zone，并更新 regionmap
    bj-1
    1
    2
    3
    4
    5
    6
    7
    # cd /etc/ceph
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center1-bj-1 region set –infile cd.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center1-bj-1 zone set –rgw-zone=cd-center1 –infile cd-center1.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center1-bj-1 zone set –rgw-zone=cd-center2 –infile cd-center2.json
    #清空 region-map，radosgw-agent 读取的 regionmap 出自这里
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center1-bj-1 region-map set –infile bj.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center1-bj-1 regionmap update

    bj-2
    1
    2
    3
    4
    5
    6
    7
    # cd /etc/ceph
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 region set –infile cd.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 zone set –rgw-zone=cd-center1 –infile cd-center1.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 zone set –rgw-zone=cd-center2 –infile cd-center2.json
    #清空 region-map，radosgw-agent 读取的 regionmap 出自这里
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 region-map set –infile bj.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 regionmap update

    在 master region 两个节点上创建 secondary region、secondary zone，并更新 regionmap
    cd-1
    1
    2
    3
    4
    5
    6
    # cd /etc/ceph
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center1-cd-1 region set –infile bj.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center1-cd-1 zone set –rgw-zone=bj-center1 –infile bj-center1.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center1-cd-1 zone set –rgw-zone=bj-center2 –infile bj-center2.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center1-cd-1 region-map set –infile cd.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center1-cd-1 regionmap update

    cd-2
    1
    2
    3
    4
    5
    6
    # cd /etc/ceph
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center2-cd-2 region set –infile bj.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center2-cd-2 zone set –rgw-zone=bj-center1 –infile bj-center1.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center2-cd-2 zone set –rgw-zone=bj-center2 –infile bj-center2.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center2-cd-2 region-map set –infile cd.json
    # /home/ceph/src/.libs/radosgw-admin –name client.radosgw.cd-center2-cd-2 regionmap update
    3. 重启 ceph-radosgw 服务，验证

    重启各个节点的 ceph-radosgw 服务

    master region 验证 region

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    # radosgw-admin regionmap get –name client.radosgw.cd-center2-cd-2
    { “regions”: [
     { “key”: “bj”,
     “val”: { “name”: “bj”,
     “api_name”: “bj”,
     “is_master”: “false”,
     “endpoints”: [
     “http:\/\/bj-1:80\/“],
     “hostnames”: [
     “cd-1”],
     “master_zone”: “bj-center1”,
     “zones”: [
     { “name”: “bj-center1”,
     “endpoints”: [
     “http:\/\/bj-1:80\/“],
     “log_meta”: “true”,
     “log_data”: “true”},
     { “name”: “bj-center2”,
     “endpoints”: [
     “http:\/\/bj-2:80\/“],
     “log_meta”: “true”,
     “log_data”: “true”}],
     “placement_targets”: [
     { “name”: “default-placement”,
     “tags”: []}],
     “default_placement”: “default-placement”}},
     { “key”: “cd”,
     “val”: { “name”: “cd”,
     “api_name”: “cd”,
     “is_master”: “true”,
     “endpoints”: [
     “http:\/\/cd-1:80\/“],
     “hostnames”: [
     “cd-1”],
     “master_zone”: “cd-center1”,
     “zones”: [
     { “name”: “cd-center1”,
     “endpoints”: [
     “http:\/\/cd-1:80\/“],
     “log_meta”: “true”,
     “log_data”: “true”},
     { “name”: “cd-center2”,
     “endpoints”: [
     “http:\/\/cd-2:80\/“],
     “log_meta”: “true”,
     “log_data”: “true”}],
     “placement_targets”: [
     { “name”: “default-placement”,
     “tags”: []}],
     “default_placement”: “default-placement”}}],
     “master_region”: “cd”,
     “bucket_quota”: { “enabled”: false,
     “max_size_kb”: -1,
     “max_objects”: -1},
     “user_quota”: { “enabled”: false,
     “max_size_kb”: -1,
     “max_objects”: -1}}

    其他节点类似，保证 regionmap 完整即可

配置 radosgw-agent

新建文件/etc/ceph/radosgw-agent/default.conf，在节点 bj-1 上

1
2
3
4
5
6
7
8
9
10
src_access_key: NDOO5JZAAM2Z6WA9FIXG
src_secret_key: 4rWHptoevPLPlOwDITljxPuJTqoMQCMM9ICBjGgv
source: http://cd-1:80
dest_access_key: 2B28YY7D2VH67EAC8AAR
dest_secret_key: RpF0yzJehzFaxsVLi5yYpC4OgdE77hqHYTyfA9un
destination: http://bj-1:80
log_file: /var/log/radosgw/radosgw-sync-cd-bj.log
verbose: False
versioned: True
metadata_only: True

注意 region 之间的同步只能是元数据同步，所以metadata_only: True

启动服务

1
2
# mkdir /var/run/ceph/radosgw-agent/
# /etc/init.d/ceph-radosgw start

可以查看同步日志

1
# tailf /var/log/radosgw/radosgw-sync-cd-bj.log

此时，在 secondary region 任意节点可以查看 metadata

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 metadata list user
[
 “bj-center2-bj-2”,
 “cd-center1-cd-1”,
 “bj-center1-bj-1”,
 “testuser”,
 “cd-center2-cd-2”]
# /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 metadata list bucket
[
 “demo”,
 “test”,
 “aaa”]
#因为只同步了元数据，查看 bucket 内的对象则为空
# /home/ceph/src/.libs/radosgw-admin –name client.radosgw.bj-center2-bj-2 bucket list –bucket=aaa
[]
© 2019 LYang blog - Powered by Hexo - Theme Chan