标签 Ceph 下的文章

基于Harbor和CephFS搭建高可用Private Registry

我们有给客户搭建私有容器仓库的需求。开源的私有容器registry可供选择的不多,除了docker官方的distribution之外,比较知名的是VMware China出品的Harbor,我们选择了harbor。

harbor在docker distribution的基础上增加了一些安全、访问控制、管理的功能以满足企业对于镜像仓库的需求。harbor以docker-compose的规范形式组织各个组件,并通过docker-compose工具进行启停。

不过,harbor默认的安装配置是针对single node的,要想做得可靠性高一些,我们需要自己探索一些可行的方案。本文将结合harbor和CephFS搭建一个满足企业高可用性需求的private registry。

一、实验环境

这里用两台阿里云ECS作为harbor的工作节点:

node1:  10.47.217.91
node2:  10.28.61.30

两台主机运行的都是Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-58-generic x86_64),使用root用户。

docker版本与docker-compose的版本如下:

# docker version
Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:42:17 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:42:17 2016
 OS/Arch:      linux/amd64

# docker-compose -v
docker-compose version 1.12.0, build b31ff33

ceph版本如下:

# ceph -v
ceph version 10.2.7

ceph的安装和配置可参考这里

二、方案思路

首先,从部署上说,我们需要的Private Registry是独立于k8s cluster存在的,即在k8s cluster外部,其存储和管理的镜像供k8s cluster 组件以及运行于k8s cluster上的应用使用。

其次,企业对registry有高可用需求,但我们也要有折中,我们的目标并不是理想的完全高可用,那样投入成本可能有些高。一般企业环境下更注重数据安全。因此首要保证harbor的数据安全,这样即便harbor实例宕掉,保证数据依然不会丢失即可。并且生产环境下registry的使用很难称得上高频,对镜像仓库的性能要求也没那么高。这种情况下,harbor的高可用至少有两种方案:

  • 多harbor实例共享后端存储
  • 多harbor实例相互数据同步(通过配置两个harbor相互复制镜像数据)

harbor原生支持双实例的镜像数据同步。不过这里我们采用第一种方案:即多harbor实例共享后端存储,因为我们有现成的cephfs供harbor使用。理想的方案示意图如下:

img{512x368}

  • 每个安放harbor实例的node都mount cephfs;
  • 每个node上的harbor实例(包含组件:ui、db、registry等)都volume mount node上的cephfs mount路径;
  • 通过Load Balance将request流量负载到各个harbor实例上。

但这样做可行么?如果这么做,Harbor实例里的mysql container就会“抱怨”:

May 17 22:45:45 172.19.0.1 mysql[12110]: 2017-05-17 14:45:45 1 [ERROR] InnoDB: Unable to lock ./ibdata1, error: 11
May 17 22:45:45 172.19.0.1 mysql[12110]: 2017-05-17 14:45:45 1 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.

MySQL多个实例无法共享一份mysql数据文件。

那么,我们会考虑将harbor连接的mysql放到外面来,使用external database;同时考虑到session共享,我们还需要增加一个存储session信息的redis cluster,这样一来,方案示意图变更如下:

img{512x368}

图中的mysql、redis你即可以用cluster,也可以用单点,还是看你的需求和投入。如果你具备现成的mysql cluster和redis cluster,那么直接用就好了。但是如果你没有,并且你还不想投入这么多(尤其是搞mysql cluster),那么用单点就好了。考虑到数据安全,可以将单点mysql的数据存储在cephfs上,如果你已经有了现成的cephfs。

三、在一个node上安装Harbor

1、初装步骤

以一个node上的Harbor安装为例,harbor提供了详细的安装步骤文档,我们按照步骤逐步进行即可(这里我使用的是1.1.0版本,截至目前为止的最新稳定版本为1.1.1版本):

~/harbor-install# wget -c https://github.com/vmware/harbor/releases/download/v1.1.0/harbor-offline-installer-v1.1.0.tgz

~/harbor-install# tar zxvf harbor-offline-installer-v1.1.0.tgz

~/harbor-install/harbor# ls -F
common/  docker-compose.notary.yml  docker-compose.yml  harbor.cfg  harbor.v1.1.0.tar.gz  install.sh*  LICENSE  NOTICE  prepare*

~/harbor-install/harbor./install.sh

[Step 0]: checking installation environment ...

Note: docker version: 1.12.5
Note: docker-compose version: 1.12.0
[Step 1]: loading Harbor images ...
... ...
[Step 2]: preparing environment ...
Generated and saved secret to file: /data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/app.conf
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.

[Step 3]: checking existing instance of Harbor ...
[Step 4]: starting Harbor ...

Creating network "harbor_harbor" with the default driver
Creating harbor-log
Creating harbor-db
Creating registry
Creating harbor-adminserver
Creating harbor-ui
Creating nginx
Creating harbor-jobservice

ERROR: for proxy  Cannot start service proxy: driver failed programming external connectivity on endpoint nginx (fdeb3e538d5f8d714ea5c79a9f3f127f05f7ba5d519e09c4c30ef81f40b2fe77): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use

harbor实例默认的监听端口是80,但一般node上的80口都会被占用,因此我们需要修改一个端口号。注意:此时harbor仅启动成功了一些container而已,尚无法正常工作。

2、修改harbor proxy组件的listen端口

harbor的proxy组件就是一个nginx,通过nginx这个反向代理,将不同的服务请求分发到内部其他组件中去。nginx默认监听node的80端口,我们用8060端口替代80端口需要进行两处配置修改:

1、harbor.cfg

hostname = node_public_ip:8060

2、docker-compose.yml

proxy:
    image: vmware/nginx:1.11.5-patched
    container_name: nginx
    restart: always
    volumes:
      - ./common/config/nginx:/etc/nginx:z
    networks:
      - harbor
    ports:
      - 8060:80   <--- 修改端口映射
      - 443:443
      - 4443:4443

由于我们修改了harbor.cfg文件,我们需要重新prepare一下,执行下面命令:

# docker-compose down -v
Stopping harbor-jobservice ... done
Stopping nginx ... done
Stopping harbor-ui ... done
Stopping harbor-db ... done
Stopping registry ... done
Stopping harbor-adminserver ... done
Stopping harbor-log ... done
Removing harbor-jobservice ... done
Removing nginx ... done
Removing harbor-ui ... done
Removing harbor-db ... done
Removing registry ... done
Removing harbor-adminserver ... done
Removing harbor-log ... done
Removing network harbor_harbor

# ./prepare
Clearing the configuration file: ./common/config/nginx/nginx.conf
Clearing the configuration file: ./common/config/ui/env
Clearing the configuration file: ./common/config/ui/app.conf
Clearing the configuration file: ./common/config/ui/private_key.pem
Clearing the configuration file: ./common/config/adminserver/env
Clearing the configuration file: ./common/config/jobservice/env
Clearing the configuration file: ./common/config/jobservice/app.conf
Clearing the configuration file: ./common/config/db/env
Clearing the configuration file: ./common/config/registry/config.yml
Clearing the configuration file: ./common/config/registry/root.crt
loaded secret from file: /mnt/cephfs/harbor/data/secretkey
Generated configuration file: ./common/config/nginx/nginx.conf
Generated configuration file: ./common/config/adminserver/env
Generated configuration file: ./common/config/ui/env
Generated configuration file: ./common/config/registry/config.yml
Generated configuration file: ./common/config/db/env
Generated configuration file: ./common/config/jobservice/env
Generated configuration file: ./common/config/jobservice/app.conf
Generated configuration file: ./common/config/ui/app.conf
Generated certificate, key file: ./common/config/ui/private_key.pem, cert file: ./common/config/registry/root.crt
The configuration files are ready, please use docker-compose to start the service.

# docker-compose up -d

Creating network "harbor_harbor" with the default driver
Creating harbor-log
Creating harbor-adminserver
Creating registry
Creating harbor-db
Creating harbor-ui
Creating harbor-jobservice
Creating nginx

我们可以通过docker-compose ps命令查看harbor组件的状态:

# docker-compose ps
       Name                     Command               State                                 Ports
--------------------------------------------------------------------------------------------------------------------------------
harbor-adminserver   /harbor/harbor_adminserver       Up
harbor-db            docker-entrypoint.sh mysqld      Up      3306/tcp
harbor-jobservice    /harbor/harbor_jobservice        Up
harbor-log           /bin/sh -c crond && rm -f  ...   Up      127.0.0.1:1514->514/tcp
harbor-ui            /harbor/harbor_ui                Up
nginx                nginx -g daemon off;             Up      0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:8060->80/tcp
registry             /entrypoint.sh serve /etc/ ...   Up      5000/tcp

如果安全组将8060端口打开,通过访问:http://node_public_ip:8060,你将看到如下harbor的web页面:

img{512x368}

我们可以通过harbor内置的默认用户名和密码admin/Harbor12345登录harbor ui。当然,我们更重要的是通过cmdline访问harbor,push和pull image。如果这时你直接尝试docker login harbor_url,你可能会得到如下错误日志:

# docker login -u admin -p Harbor12345 node_public_ip:8060
Error response from daemon: Get https://node_public_ip:8060/v1/users/: http: server gave HTTP response to HTTPS client

这是因为docker默认采用https访问registry,因此我们需要在docker engine的配置中,添加–insecure-registry option。关于ubuntu 16.04下docker配置的问题,请参考这里

DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 --registry-mirror=https://xxxxx.mirror.aliyuncs.com --insecure-registry=node_public_ip:8060"

重启docker engine后尝试再次登录harbor:

docker login -u admin -p Harbor12345 node_public_ip:8060
Login Succeeded

一旦docker client login ok,我们就可以通过docker client对harbor中的相关repository进行操作了。

四、挂载路径修改

默认情况下,harbor将数据volume挂载到主机的/data路径下面。但由于我们采用ceph共享存储保证数据的高可用,需要修改harbor组件内容器的挂载路径,将其mount到共享存储挂载node上的路径:/mnt/cephfs/harbor/data/。对比两个路径,可以看出前缀由”/”变为了”/mnt/cephfs/harbor/”,我们需要修改docker-compose.yml和harbor.cfg两个文件。

由于docker-compose.yml文件较长,这里将原始文件改名为docker-compose.yml.orig,并将其与修改后的docker-compose.yml做对比:

# diff  docker-compose.yml.orig docker-compose.yml
8c8
<       - /var/log/harbor/:/var/log/docker/:z
---
>       - /mnt/cephfs/harbor/log/:/var/log/docker/:z
20c20
<       - /data/registry:/storage:z
---
>       - /mnt/cephfs/harbor/data/registry:/storage:z
40c40
<       - /data/database:/var/lib/mysql:z
---
>       - /mnt/cephfs/harbor/data/database:/var/lib/mysql:z
59,61c59,61
<       - /data/config/:/etc/adminserver/config/:z
<       - /data/secretkey:/etc/adminserver/key:z
<       - /data/:/data/:z
---
>       - /mnt/cephfs/harbor/data/config/:/etc/adminserver/config/:z
>       - /mnt/cephfs/harbor/data/secretkey:/etc/adminserver/key:z
>       - /mnt/cephfs/harbor/data/:/data/:z
80,81c80,81
<       - /data/secretkey:/etc/ui/key:z
<       - /data/ca_download/:/etc/ui/ca/:z
---
>       - /mnt/cephfs/harbor/data/secretkey:/etc/ui/key:z
>       - /mnt/cephfs/harbor/data/ca_download/:/etc/ui/ca/:z
100c100
<       - /data/job_logs:/var/log/jobs:z
---
>       - /mnt/cephfs/harbor/data/job_logs:/var/log/jobs:z
102c102
<       - /data/secretkey:/etc/jobservice/key:z
---
>       - /mnt/cephfs/harbor/data/secretkey:/etc/jobservice/key:z

harbor.cfg文件需要修改的地方不多:

// harbor.cfg

#The path of cert and key files for nginx, they are applied only the protocol is set to https
ssl_cert = /mnt/cephfs/harbor/data/cert/server.crt
ssl_cert_key = /mnt/cephfs/harbor/data/cert/server.key

#The path of secretkey storage
secretkey_path = /mnt/cephfs/harbor/data

配置修改完毕后,执行如下命令:

# docker-compose down -v
# prepare
# docker-compose up -d

新的harbor实例就启动起来了。注意:这一步我们用cephfs替换了本地存储,主要的存储变动针对log、database和registry三个输出数据的组件。你也许会感受到cephfs给harbor ui页面加载带来的影响,实感要比之前的加载慢一些。

五、使用外部数据库(external database)

前面提到了挂载ceph后,多个node上harbor实例中的db组件将出现竞争问题,导致只有一个node上的harbor db组件可以工作。因此,我们要使用外部数据库(或db集群)来解决这个问题。但是harbor官方针对如何配置使用外部DB很是“讳莫如深”,我们只能自己探索。

假设我们已经有了一个external database,并且建立了harbor这个user,并做了相应的授权。由于harbor习惯了独享database,在测试环境下可以考虑

GRANT ALL ON *.* TO 'harbor'@'%';

1、迁移数据

如果此时镜像库中已经有了数据,我们需要做一些迁移工作。

attach到harbor db组件的container中,将registry这张表dump到registry.dump文件中:

#docker exec -i -t  6e1e4b576315  bash

在db container中:
# mysqldump -u root -p --databases registry > registry.dump

回到node,将dump文件从container中copy出来:

#docker cp 6e1e4b576315:/root/registry.dump ./

再mysql login到external Database,将registry.dump文件导入:

# mysql -h external_db_ip -P 3306 -u harbor -p
# mysql> source ./registry.dump;

2、修改harbor配置,使得ui、jobservice组件连接external db

根据当前harbor architecture图所示:

img{512x368}

与database“有染”的组件包括ui和jobservice,如何通过配置修改来让这两个组件放弃老db,访问新的external db呢?这要从挖掘配置开始。harbor的组件配置都在common/config下:

~/harbor-install/harbor# tree -L 3 common
common
├── config
│   ├── adminserver
│   │   └── env
│   ├── db
│   │   └── env
│   ├── jobservice
│   │   ├── app.conf
│   │   └── env
│   ├── nginx
│   │   └── nginx.conf
│   ├── registry
│   │   ├── config.yml
│   │   └── root.crt
│   └── ui
│       ├── app.conf
│       ├── env
│       └── private_key.pem
└── templates
 ... ...

在修改config之前,我们先docker-compose down掉harbor。接下来,我们看到ui和jobservice下都有env文件,这里想必就是可以注入新db的相关访问信息的地方,我们来试试!

// common/config/ui/env
LOG_LEVEL=debug
CONFIG_PATH=/etc/ui/app.conf
UI_SECRET=$ui_secret
JOBSERVICE_SECRET=$jobservice_secret
GODEBUG=netdns=cgo
MYSQL_HOST=new_db_ip
MYSQL_PORT=3306
MYSQL_USR=harbor
MYSQL_PWD=harbor_password

// common/config/jobservice/env
LOG_LEVEL=debug
CONFIG_PATH=/etc/jobservice/app.conf
UI_SECRET=$ui_secret
JOBSERVICE_SECRET=$jobservice_secret
GODEBUG=netdns=cgo
MYSQL_HOST=new_db_ip
MYSQL_PORT=3306
MYSQL_USR=harbor
MYSQL_PWD=harbor_password

同时,由于不再需要harbor_db组件,因此切记:要将其从docker-compose.yml中剔除!。docker-compose up -d重新创建harbor各组件容器并启动!Harbor的日志可以在挂载的ceph路径: /mnt/cephfs/harbor/log下查找到:

/mnt/cephfs/harbor/log# tree 2017-06-09
2017-06-09
├── adminserver.log
├── anacron.log
├── CROND.log
├── jobservice.log
├── mysql.log
├── proxy.log
├── registry.log
├── run-parts.log
└── ui.log

我们以ui.log为例,我们发现harbor启动后,ui.log输出如下错误日志(jobservice.log也是相同):

Jun  9 11:00:17 172.19.0.1 ui[16039]: 2017-06-09T03:00:17Z [INFO] initializing database: type-MySQL host-mysql port-3306 user-root database-registry
Jun  9 11:00:18 172.19.0.1 ui[16039]: 2017-06-09T03:00:18Z [ERROR] [utils.go:94]: failed to connect to tcp://mysql:3306, retry after 2 seconds :dial tcp: lookup mysql: no such host

我们明明注入了新的db env,为何ui还是要访问“tcp://mysql:3306”呢?我们docker inspect一下ui的container,看看env是否包含我们添加的那些:

# docker inspect e91ab20e1dcb
... ...
            "Env": [
                "DATABASE_TYPE=mysql",
                "MYSQL_HOST=database_ip",
                "MYSQL_PORT=3306",
                "MYSQL_PWD=harbor_password",
                "MYSQL_USR=harbor",
                "MYSQL_DATABASE=registry",
            ],
.... ...

env已经注入,那么为何ui、jobservice无法连接到external database呢?要想搞清楚这点,我们只能去“啃代码”了。还好harbor代码并非很难啃。我们发现基于beego实现的ui、jobservice两个组件并未直接通过os.Getenv去获取这些env变量,而是调用了adminserver组件的服务。adminserver在初始化时,在RESET环境变量为true的情况下,读取了common/config/adminserver/env下的所有环境变量。

搞清楚原理后,我们知道了要修改的是common/config/adminserver/env,而不是common/config/ui/env和common/config/jobservice/env。我们将后两个文件还原。修改common/config/adminserver/env文件:

//common/config/adminserver/env
... ...
MYSQL_HOST=new_db_ip
MYSQL_PORT=3306
MYSQL_USR=harbor
MYSQL_PWD=harbor_password
... ...
RESET=true    <--- 改为true,非常关键

重新up harbor服务后,我们发现ui, jobservice与新database的连接成功了!打开harbor web页面,登录进去,我们看到了之前已经添加的用户、项目和镜像文件。

3、一劳永逸

如果你重新执行prepare,那么上面对config目录下的配置修改将被重新覆盖。如果要一劳永逸,那么需要修改的是common/templates下面的同位置同名配置文件。

六、安装其他节点上的harbor实例

前面,我们只搭建了一个节点,为的是验证方案的可行性。要实现高可用,我们还需要在其他节点上安装harbor实例。由于多个节点上harbor实例共同挂载ceph的同一目录,因此考虑到log的分离,在部署其他节点上的harbor时,最好对docker-compose.yml下log组件的volumes映射路径进行调整,以在多个节点间做隔离,便于日志查看,比如:

volumes:
      - /mnt/cephfs/harbor/log1/:/var/log/docker/:z

除此之外,各个节点上的harbor配置与上述配置完全一致。

七、共享session设置

到harbor的请求被负载均衡分发到多个node上的harbor实例上,这样就有了session共享的需求。Harbor对此已经给予了支持。在ui组件的代码中,我们发现ui在初始化时使用Getenv获取”_REDIS_URL”这个环境变量的值,因此我们只需要将_REDIS_URL这个环境变量配置到各个节点harbor ui组件的env文件中即可:

// common/config/adminserver/env

LOG_LEVEL=debug
CONFIG_PATH=/etc/ui/app.conf
UI_SECRET=LuAwkKUtYjF4l0mQ
JOBSERVICE_SECRET=SmsO1kVo4SrmgOIp
GODEBUG=netdns=cgo
_REDIS_URL=redis_ip:6379,100,redis_password,0

重新up harbor后,session共享生效。

不过光有一个外部redis存储共享session还不够,请求在多个harbor实例中的registry组件中进行鉴权需要harbor各个实例share相同的key和certificate。好在,我们的多harbor实例通过ceph共享存储,key和cert本就是共享的,都存放在目录:/mnt/cephfs/harbor/data/cert/的下边,因此也就不需要在各个harbor实例间同步key和cert了。

八、更换为域名访问

我们有通过域名访问docker registry的需求,那么直接通过域名访问harbor ui和registry是否可行呢?这要看harbor nginx的配置:

# docker ps |grep nginx
fa92765e8871        vmware/nginx:1.11.5-patched   "nginx -g 'daemon off"   3 hours ago
Up 3 hours          0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:8060->80/tcp               nginx

# docker exec fa92765e8871 cat /etc/nginx/nginx.conf

... ...
http {
   server {
    listen 80;
   ... ...

}

nginx在http server block并未对域名或ip进行匹配,因此直接将域名A地址设置为反向代理的地址或直接解析为Harbor暴露的公网ip地址都是可以正常访问harbor服务的,当然也包括image push和pull服务。

九、统一registry的证书和token service的私钥

这是在本篇文章发表之后发现的问题,针对该问题,我专门写了一篇文章:《解决登录Harbor Registry时鉴权失败的问题》,请移步这篇文章,完成HA Harbor的搭建。

十、参考资料


微博:@tonybai_cn
微信公众号:iamtonybai
github.com: https://github.com/bigwhite

Kubernetes集群node主机名修改导致的异常

除了在生产环境使用的Kubernetes 1.3.7集群之外,我这里还有一套1.5.1的Kubernetes测试环境,这个测试环境一来用于验证各种技术方案,二来也是为了跟踪Kubernetes的最新进展。本篇要记录的一个异常就是发生在该测试Kubernetes集群中的。

一、缘起

前两天我在Kubernetes测试环境搭建一套Ceph,为了便于ceph-deploy的安装,我通过hostnamectl命令将阿里云默认提供的复杂又冗长的主机名改为短小且更有意义的主机名:

iZ25beglnhtZ -> yypdmaster
iz2ze39jeyizepdxhwqci6z -> yypdnode

以yypdmaster为例,修改过程如下:

# hostnamectl --static set-hostname yypdmaster
# hostnamectl status
Static hostname: yypdmaster
Transient hostname: iZ25beglnhtZ
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 91aa4b8f2556de49e743dc2f53e8a5c4
           Boot ID: 5d0e642ebafa460086388da4177e488e
    Virtualization: kvm
  Operating System: Ubuntu 16.04.1 LTS
            Kernel: Linux 4.4.0-58-generic
      Architecture: x86-64

# cat /etc/hostname
yypdmaster

hostnamectl并未修改/etc/hosts,我手动在/etc/hosts中将yypdmaster对应的ip配置上:

xx.xx.xx.xx yypdmaster

重新登录后,我们看到主机名状态:Transient hostname不见了,只剩下了静态主机名:

# hostnamectl status
   Static hostname: yypdmaster
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 91aa4b8f2556de49e743dc2f53e8a5c4
           Boot ID: 5d0e642ebafa460086388da4177e488e
    Virtualization: kvm
  Operating System: Ubuntu 16.04.1 LTS
            Kernel: Linux 4.4.0-58-generic
      Architecture: x86-64

另外一台主机也是如此修改。主机名修改后,整个k8s集群工作一切正常,因此我最初以为hostname的修改对k8s cluster的运行没有影响。

二、集群”Crash”

昨天在做跨节点挂载Cephfs测试时,发现在yypdmaster上kubectl exec另外一个node上的pod不好用,提示:连接10250端口超时!而且从错误日志来看,yypdmaster上的k8s组件居然通过yypdnode的外网ip去访问yypdnode上的10250端口,也就是yypdnode上kubelet监听的端口。由于aliyun的安全组规则限制,这个端口是不允许外网访问的,因此timeout错误是合理的。但为什么之前集群都是好好的?突然间出现这个问题呢?为什么不用内网的ip地址访问呢?

我尝试重启了yypdnode上的kubelet服务。不过似乎没什么效果!正当我疑惑时,我发现集群似乎”Crash”了,下面是当时查看集群的pod情况的输出:

# kubectl get pod --all-namespaces -o wide

NAMESPACE                    NAME                                    READY     STATUS             RESTARTS   AGE       IP             NODE
default                      ceph-pod2                               1/1       Unknown            0          26m       172.30.192.4   iz2ze39jeyizepdxhwqci6z
default                      ceph-pod2-with-secret                   1/1       Unknown            0          38m       172.30.192.2   iz2ze39jeyizepdxhwqci6z
default                      ceph-pod2-with-secret-on-master         1/1       Unknown            0          34m       172.30.0.51    iz25beglnhtz
default                      nginx-kit-3630450072-2c0jk              0/2       Pending            0          12m       <none>
default                      nginx-kit-3630450072-3n50m              2/2       Unknown            20         35d       172.30.0.44    iz25beglnhtz
default                      nginx-kit-3630450072-90v4q              0/2       Pending            0          12m       <none>
default                      nginx-kit-3630450072-j8qrk              2/2       Unknown            20         72d       172.30.0.47    iz25beglnhtz
kube-system                  dummy-2088944543-9382n                  1/1       Running            0          12m       xx.xx.xx.xx   yypdmaster
kube-system                  dummy-2088944543-93f4c                  1/1       Unknown            16         130d      xx.xx.xx.xx   iz25beglnhtz
kube-system                  elasticsearch-logging-v1-dhl35          1/1       Running            0          12m       172.30.192.6   yypdnode
kube-system                  elasticsearch-logging-v1-s3sbj          1/1       Unknown            9          35d       172.30.0.45    iz25beglnhtz
kube-system                  elasticsearch-logging-v1-t8wg0          1/1       Unknown            29         68d       172.30.0.43    iz25beglnhtz
kube-system                  elasticsearch-logging-v1-zdp19          1/1       Running            0          12m       172.30.0.3     yypdmaster
kube-system                  etcd-iz25beglnhtz                       1/1       Unknown            17         130d      xx.xx.xx.xx   iz25beglnhtz
kube-system                  etcd-yypdmaster                         1/1       Running            17         17m       xx.xx.xx.xx   yypdmaster
kube-system                  fluentd-es-v1.22-ggvv4                  1/1       NodeLost           24         68d       172.30.0.46    iz25beglnhtz
kube-system                  fluentd-es-v1.22-rj871                  1/1       Running            0          17m       172.30.0.1     yypdmaster
kube-system                  fluentd-es-v1.22-xn77x                  1/1       NodeLost           0          6d        172.30.192.0   iz2ze39jeyizepdxhwqci6z
kube-system                  fluentd-es-v1.22-z82rz                  1/1       Running            0          18m       172.30.192.5   yypdnode
kube-system                  kibana-logging-3746979809-dplzv         1/1       Running            0          12m       172.30.0.4     yypdmaster
kube-system                  kibana-logging-3746979809-lq9m3         1/1       Unknown            9          35d       172.30.0.49    iz25beglnhtz
kube-system                  kube-apiserver-iz25beglnhtz             1/1       Unknown            19         104d      xx.xx.xx.xx   iz25beglnhtz
kube-system                  kube-apiserver-yypdmaster               1/1       Running            19         17m       xx.xx.xx.xx   yypdmaster
kube-system                  kube-controller-manager-iz25beglnhtz    1/1       Unknown            21         130d      xx.xx.xx.xx   iz25beglnhtz
kube-system                  kube-controller-manager-yypdmaster      1/1       Running            21         17m       xx.xx.xx.xx   yypdmaster
kube-system                  kube-discovery-1769846148-wh1z4         1/1       Unknown            12         73d       xx.xx.xx.xx   iz25beglnhtz
kube-system                  kube-discovery-1769846148-z2v87         0/1       Pending            0          12m       <none>
kube-system                  kube-dns-2924299975-206tg               4/4       Unknown            129        130d      172.30.0.48    iz25beglnhtz
kube-system                  kube-dns-2924299975-g1kks               4/4       Running            0          12m       172.30.0.5     yypdmaster
kube-system                  kube-proxy-3z29k                        1/1       Running            0          18m       yy.yy.yy.yy    yypdnode
kube-system                  kube-proxy-kfzxv                        1/1       Running            0          17m       xx.xx.xx.xx   yypdmaster
kube-system                  kube-proxy-n2xmf                        1/1       NodeLost           16         130d      xx.xx.xx.xx   iz25beglnhtz

观察这个输出,我们看到几点异常:

  • 不常见的Pod状态:Unknown、NodeLost
  • Node一列居然出现了四个Node: yypdmaster、yypdnode、 iz25beglnhtz和 iz2ze39jeyizepdxhwqci6z

等了一会儿,这种状态依然不见好转。我于是重启了master上的kubelet、重启了两个节点上的docker engine,不过启动后问题依旧!

查看Running状态的Pod情况:

# kubectl get pod --all-namespaces -o wide|grep Running
kube-system                  dummy-2088944543-9382n                  1/1       Running            0          18m       xx.xx.xx.xx   yypdmaster
kube-system                  elasticsearch-logging-v1-dhl35          1/1       Running            0          18m       172.30.192.6   yypdnode
kube-system                  elasticsearch-logging-v1-zdp19          1/1       Running            0          18m       172.30.0.3     yypdmaster
kube-system                  etcd-yypdmaster                         1/1       Running            17         23m       xx.xx.xx.xx   yypdmaster
kube-system                  fluentd-es-v1.22-rj871                  1/1       Running            0          23m       172.30.0.1     yypdmaster
kube-system                  fluentd-es-v1.22-z82rz                  1/1       Running            0          24m       172.30.192.5   yypdnode
kube-system                  kibana-logging-3746979809-dplzv         1/1       Running            0          18m       172.30.0.4     yypdmaster
kube-system                  kube-apiserver-yypdmaster               1/1       Running            19         23m       xx.xx.xx.xx   yypdmaster
kube-system                  kube-controller-manager-yypdmaster      1/1       Running            21         23m       xx.xx.xx.xx   yypdmaster
kube-system                  kube-dns-2924299975-g1kks               4/4       Running            0          18m       172.30.0.5     yypdmaster
kube-system                  kube-proxy-3z29k                        1/1       Running            0          24m       yy.yy.yy.yy    yypdnode
kube-system                  kube-proxy-kfzxv                        1/1       Running            0          23m       xx.xx.xx.xx   yypdmaster
kube-system                  kube-scheduler-yypdmaster               1/1       Running            22         23m       xx.xx.xx.xx   yypdmaster
kube-system                  kubernetes-dashboard-3109525988-cj74d   1/1       Running            0          18m       172.30.0.6     yypdmaster
mioss-namespace-s0fcvegcmw   console-sm7cg2-101699315-f3g55          1/1       Running            0          18m       172.30.0.7     yypdmaster

似乎Kubernetes集群并未真正”Crash”,但从Node列来看,正常的pod归属的node不是yypdmaster就是yypdnode, iz25beglnhtz和 iz2ze39jeyize

Kubernetes集群跨节点挂载CephFS

Kubernetes集群中运行有状态服务或应用总是不那么容易的。比如,之前我在项目中使用了CephRBD,虽然遇到过几次问题,但总体算是运行良好。但最近发现CephRBD无法满足跨节点挂载的需求,我只好另辟蹊径。由于CephFS和CephRBD师出同门,它自然成为了这次我首要考察的目标。这里将跨节点挂载CephFS的考察过程记录一下,一是备忘,二则也可以为其他有相似需求的朋友提供些资料。

一、CephRBD的问题

这里先提一嘴CephRBD的问题。最近项目中有这样的需求:让集群中的Pod共享外部分布式存储,即多个Pod共同挂载一份存储,实现存储共享,这样可大大简化系统设计和复杂性。之前CephRBD都是挂载到一个Pod中运行的,CephRBD是否支持多Pod同时挂载呢?官方文档中给出了否定的答案: 基于CephRBD的Persistent Volume仅支持两种accessmode:
ReadWriteOnce和ReadOnlyMany,不支持ReadWriteMany。这样对于有读写需求的Pod来说,一个CephRBD pv仅能被一个node挂载一次。

我们来验证一下这个“不幸的”事实。

我们首先创建一个测试用的image:foo1。这里我利用了项目里写的CephRBD API服务,也可通过ceph命令手工创建:

# curl -v  -H "Content-type: application/json" -X POST -d '{"kind": "Images","apiVersion": "v1", "metadata": {"name": "foo1", "capacity": 512} ' http://192.168.3.22:8080/api/v1/pools/rbd/images
... ...
{
  "errcode": 0,
  "errmsg": "ok"
}

# curl http://192.168.3.22:8080/api/v1/pools/rbd/images
{
  "Kind": "ImagesList",
  "APIVersion": "v1",
  "Items": [
    {
      "name": "foo1"
    }
  ]
}

利用下面文件创建pv和pvc:

//ceph-pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: foo-pv
spec:
  capacity:
    storage: 512Mi
  accessModes:
    - ReadWriteMany
  rbd:
    monitors:
      - ceph_monitor_ip:port
    pool: rbd
    image: foo1
    user: admin
    secretRef:
      name: ceph-secret
    fsType: ext4
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

//ceph-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: foo-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 512Mi

创建后:

# kubectl get pv
[NAME                CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                        REASON    AGE
foo-pv              512Mi      RWO           Recycle         Bound     default/foo-claim                      20h

# kubectl get pvc
NAME                 STATUS    VOLUME              CAPACITY   ACCESSMODES   AGE
foo-claim            Bound     foo-pv              512Mi      RWO           20h

创建挂载上述image的Pod:

// ceph-pod2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod2
spec:
  containers:
  - name: ceph-ubuntu2
    image: ubuntu:14.04
    command: ["tail", "-f", "/var/log/bootstrap.log"]
    volumeMounts:
    - name: ceph-vol2
      mountPath: /mnt/cephrbd/data
      readOnly: false
  volumes:
  - name: ceph-vol2
    persistentVolumeClaim:
      claimName: foo-claim

创建成功后,我们可以查看挂载目录的数据:

# kubectl exec ceph-pod2 ls /mnt/cephrbd/data
1.txt
lost+found

我们在同一个kubernetes node上再启动一个pod(可以把上面的ceph-pod2.yaml的pod name改为ceph-pod3),挂载同样的pv:

NAMESPACE                    NAME                                    READY     STATUS    RESTARTS   AGE       IP             NODE
default                      ceph-pod2                               1/1       Running   0          3m        172.16.57.9    xx.xx.xx.xx
default                      ceph-pod3                               1/1       Running   0          0s        172.16.57.10    xx.xx.xx.xx
# kubectl exec ceph-pod3 ls /mnt/cephrbd/data
1.txt
lost+found

我们通过ceph-pod2写一个文件,在ceph-pod3中将其读出:

# kubectl exec ceph-pod2 -- bash -c "for i in {1..10}; do sleep 1; echo 'pod2: Hello, World'>> /mnt/cephrbd/data/foo.txt ; done "
root@node1:~/k8stest/k8s-cephrbd/footest# kubectl exec ceph-pod3 cat /mnt/cephrbd/data/foo.txt
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World
pod2: Hello, World

到目前为止,在一个node上多个Pod是可以以ReadWrite模式挂载同一个CephRBD的。

我们在另外一个节点启动一个试图挂载该pv的Pod,该Pod启动后一直处于pending状态,通过kubectl describe查看其详细信息,可以看到:

Events:
  FirstSeen    LastSeen    Count    From            SubobjectPath    Type        Reason        Message
  ---------    --------    -----    ----            -------------    --------    ------        -------
.. ...
  2m        37s        2    {kubelet yy.yy.yy.yy}            Warning        FailedMount    Unable to mount volumes for pod "ceph-pod2-master_default(a45f62aa-2bc3-11e7-9baa-00163e1625a9)": timeout expired waiting for volumes to attach/mount for pod "ceph-pod2-master"/"default". list of unattached/unmounted volumes=[ceph-vol2]
  2m        37s        2    {kubelet yy.yy.yy.yy}            Warning        FailedSync    Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "ceph-pod2-master"/"default". list of unattached/unmounted volumes=[ceph-vol2]

查看kubelet.log中的错误日志:

I0428 11:39:15.737729    1241 reconciler.go:294] MountVolume operation started for volume "kubernetes.io/rbd/a45f62aa-2bc3-11e7-9baa-00163e1625a9-foo-pv" (spec.Name: "foo-pv") to pod "a45f62aa-2bc3-11e7-9baa-00163e1625a9" (UID: "a45f62aa-2bc3-11e7-9baa-00163e1625a9").
I0428 11:39:15.939183    1241 operation_executor.go:768] MountVolume.SetUp succeeded for volume "kubernetes.io/secret/923700ff-12c2-11e7-9baa-00163e1625a9-default-token-40z0x" (spec.Name: "default-token-40z0x") pod "923700ff-12c2-11e7-9baa-00163e1625a9" (UID: "923700ff-12c2-11e7-9baa-00163e1625a9").
E0428 11:39:17.039656    1241 disk_manager.go:56] failed to attach disk
E0428 11:39:17.039722    1241 rbd.go:228] rbd: failed to setup mount /var/lib/kubelet/pods/a45f62aa-2bc3-11e7-9baa-00163e1625a9/volumes/kubernetes.io~rbd/foo-pv rbd: image foo1 is locked by other nodes
E0428 11:39:17.039857    1241 nestedpendingoperations.go:254] Operation for "\"kubernetes.io/rbd/a45f62aa-2bc3-11e7-9baa-00163e1625a9-foo-pv\" (\"a45f62aa-2bc3-11e7-9baa-00163e1625a9\")" failed. No retries permitted until 2017-04-28 11:41:17.039803969 +0800 CST (durationBeforeRetry 2m0s). Error: MountVolume.SetUp failed for volume "kubernetes.io/rbd/a45f62aa-2bc3-11e7-9baa-00163e1625a9-foo-pv" (spec.Name: "foo-pv") pod "a45f62aa-2bc3-11e7-9baa-00163e1625a9" (UID: "a45f62aa-2bc3-11e7-9baa-00163e1625a9") with: rbd: image foo1 is locked by other nodes

可以看到“rbd: image foo1 is locked by other nodes”的日志。我们用试验证明了目前CephRBD仅能被k8s中的一个node挂载的事实。

二、Ceph集群安装mds以支持CephFS

这次我在两个Ubuntu 16.04的vm上新部署了一套Ceph,过程与之前第一次部署Ceph时大同小异,这里就不赘述了。要让Ceph支持CephFS,我们需要安装mds组件,有了前面的基础,通过ceph-deploy工具安装mds十分简单:

# ceph-deploy mds create yypdmaster yypdnode
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.37): /usr/bin/ceph-deploy mds create yypdmaster yypdnode
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f60fb5e71b8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f60fba4e140>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('yypdmaster', 'yypdmaster'), ('yypdnode', 'yypdnode')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts yypdmaster:yypdmaster yypdnode:yypdnode
[yypdmaster][DEBUG ] connected to host: yypdmaster
[yypdmaster][DEBUG ] detect platform information from remote host
[yypdmaster][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 16.04 xenial
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to yypdmaster
[yypdmaster][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[yypdmaster][DEBUG ] create path if it doesn't exist
[yypdmaster][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.yypdmaster osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-yypdmaster/keyring
[yypdmaster][INFO  ] Running command: systemctl enable ceph-mds@yypdmaster
[yypdmaster][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@yypdmaster.service to /lib/systemd/system/ceph-mds@.service.
[yypdmaster][INFO  ] Running command: systemctl start ceph-mds@yypdmaster
[yypdmaster][INFO  ] Running command: systemctl enable ceph.target
[yypdnode][DEBUG ] connected to host: yypdnode
[yypdnode][DEBUG ] detect platform information from remote host
[yypdnode][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 16.04 xenial
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to yypdnode
[yypdnode][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[yypdnode][DEBUG ] create path if it doesn't exist
[yypdnode][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.yypdnode osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-yypdnode/keyring
[yypdnode][INFO  ] Running command: systemctl enable ceph-mds@yypdnode
[yypdnode][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/ceph-mds@yypdnode.service to /lib/systemd/system/ceph-mds@.service.
[yypdnode][INFO  ] Running command: systemctl start ceph-mds@yypdnode
[yypdnode][INFO  ] Running command: systemctl enable ceph.target

非常顺利。安装后,可以在任意一个node上看到mds在运行:

# ps -ef|grep ceph
ceph      7967     1  0 17:23 ?        00:00:00 /usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
ceph     15674     1  0 17:32 ?        00:00:00 /usr/bin/ceph-mon -f --cluster ceph --id yypdnode --setuser ceph --setgroup ceph
ceph     18019     1  0 17:35 ?        00:00:00 /usr/bin/ceph-mds -f --cluster ceph --id yypdnode --setuser ceph --setgroup ceph

mds是存储cephfs的元信息的,我的ceph是10.2.7版本:

# ceph -v
ceph version 10.2.7 (50e863e0f4bc8f4b9e31156de690d765af245185)

虽然支持多 active mds并行运行,但官方文档建议保持一个active mds,其他mds作为standby(见下面ceph集群信息中的fsmap部分):

# ceph -s
    cluster ffac3489-d678-4caf-ada2-3dd0743158b6
    ... ...
      fsmap e6: 1/1/1 up {0=yypdnode=up:active}, 1 up:standby
     osdmap e19: 2 osds: 2 up, 2 in
            flags sortbitwise,require_jewel_osds
      pgmap v192498: 576 pgs, 5 pools, 126 MB data, 238 objects
            44365 MB used, 31881 MB / 80374 MB avail
                 576 active+clean

三、创建fs并测试挂载

我们在ceph上创建一个fs:

# ceph osd pool create cephfs_data 128
pool 'cephfs_data' created

# ceph osd pool create cephfs_metadata 128
pool 'cephfs_metadata' created

# ceph fs new test_fs cephfs_metadata cephfs_data
new fs with metadata pool 2 and data pool 1

# ceph fs ls
name: test_fs, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

不过,ceph当前正式版功能中仅支持一个fs,对多个fs的支持仅存在于实验feature中:

# ceph osd pool create cephfs1_data 128
# ceph osd pool create cephfs1_metadata 128
# ceph fs new test_fs1 cephfs1_metadata cephfs1_data
Error EINVAL: Creation of multiple filesystems is disabled.  To enable this experimental feature, use 'ceph fs flag set enable_multiple true'

在物理机上挂载cephfs可以使用mount命令、mount.ceph(apt-get install ceph-fs-common)或ceph-fuse(apt-get install ceph-fuse),我们先用mount命令挂载:

我们将上面创建的cephfs挂载到主机的/mnt下:

#mount -t ceph ceph_mon_host:6789:/ /mnt -o name=admin,secretfile=admin.secret

# cat admin.secret //ceph.client.admin.keyring中的key
AQDITghZD+c/DhAArOiWWQqyMAkMJbWmHaxjgQ==

查看cephfs信息:

# df -h
ceph_mon_host:6789:/   79G   45G   35G  57% /mnt

可以看出:cephfs将两个物理节点上的磁盘全部空间作为了自己的空间。

通过ceph-fuse挂载,还可以限制对挂载路径的访问权限,我们来创建用户foo,让其仅仅拥有对/ceph-volume1-test路径具有只读访问权限:

# ceph auth get-or-create client.foo mon 'allow *' mds 'allow r path=/ceph-volume1-test' osd 'allow *'
# ceph-fuse -n client.foo -m 10.47.217.91:6789 /mnt -r /ceph-volume1-test
ceph-fuse[10565]: starting ceph client2017-05-03 16:07:25.958903 7f1a14fbff00 -1 init, newargv = 0x557e350defc0 newargc=11
ceph-fuse[10565]: starting fuse

查看挂载路径,并尝试创建文件:

# cd /mnt
root@yypdnode:/mnt# ls
1.txt
root@yypdnode:/mnt# touch 2.txt
touch: cannot touch '2.txt': Permission denied

由于foo用户只拥有对 /ceph-volume1-test的只读权限,因此创建文件失败了!

四、Kubernetes跨节点挂载CephFS

在K8s中,至少可以通过两种方式挂载CephFS,一种是通过Pod直接挂载;另外一种则是通过pv和pvc挂载。我们分别来看。

1、Pod直接挂载CephFS

//ceph-pod2-with-secret.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod2-with-secret
spec:
  containers:
  - name: ceph-ubuntu2
    image: ubuntu:14.04
    command: ["tail", "-f", "/var/log/bootstrap.log"]
    volumeMounts:
    - name: ceph-vol2
      mountPath: /mnt/cephfs/data
      readOnly: false
  volumes:
  - name: ceph-vol2
    cephfs:
      monitors:
      - ceph_mon_host:6789
      user: admin
      secretFile: "/etc/ceph/admin.secret"
      readOnly: false

注意:保证每个节点上都存在/etc/ceph/admin.secret文件。

查看Pod挂载的内容:

# docker ps|grep pod
bc96431408c7        ubuntu:14.04                                                  "tail -f /var/log/boo"   About a minute ago   Up About a minute                                                                        k8s_ceph-ubuntu2.66c44128_ceph-pod2-with-secret_default_3d8a05f8-33c3-11e7-bcd9-6640d35a0e90_fc483b8a
bcc65ab82069        gcr.io/google_containers/pause-amd64:3.0                      "/pause"                 About a minute ago   Up About a minute                                                                        k8s_POD.d8dbe16c_ceph-pod2-with-secret_default_3d8a05f8-33c3-11e7-bcd9-6640d35a0e90_02381204

root@yypdnode:~# docker exec bc96431408c7 ls /mnt/cephfs/data
1.txt
apps
ceph-volume1-test
test1.txt

我们再在另外一个node上启动挂载同一个cephfs的Pod,看是否可以跨节点挂载:

# kubectl get pods

NAMESPACE                    NAME                                    READY     STATUS    RESTARTS   AGE       IP             NODE
default                      ceph-pod2-with-secret                   1/1       Running   0          3m        172.30.192.2   iz2ze39jeyizepdxhwqci6z
default                      ceph-pod2-with-secret-on-master         1/1       Running   0          3s        172.30.0.51    iz25beglnhtz
... ...

# kubectl exec ceph-pod2-with-secret-on-master ls /mnt/cephfs/data
1.txt
apps
ceph-volume1-test
test1.txt

可以看到不同节点可以挂载同一CephFS。我们在一个pod中操作一下挂载的cephfs:

# kubectl exec ceph-pod2-with-secret-on-master -- bash -c "for i in {1..10}; do sleep 1; echo 'pod2-with-secret-on-master: Hello, World'>> /mnt/cephfs/data/foo.txt ; done "
root@yypdmaster:~/k8stest/cephfstest/footest# kubectl exec ceph-pod2-with-secret-on-master cat /mnt/cephfs/data/foo.txt
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World

2、通过PV和PVC挂载CephFS

挂载cephfs的pv和pvc在写法方面与上面挂载rbd的类似:

//ceph-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: foo-pv
spec:
  capacity:
    storage: 512Mi
  accessModes:
    - ReadWriteMany
  cephfs:
    monitors:
      - ceph_mon_host:6789
    path: /
    user: admin
    secretRef:
      name: ceph-secret
    readOnly: false
  persistentVolumeReclaimPolicy: Recycle

//ceph-pvc.yaml

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: foo-claim
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 512Mi

使用pvc的pod:

//ceph-pod2.yaml
apiVersion: v1
kind: Pod
metadata:
  name: ceph-pod2
spec:
  containers:
  - name: ceph-ubuntu2
    image: ubuntu:14.04
    command: ["tail", "-f", "/var/log/bootstrap.log"]
    volumeMounts:
    - name: ceph-vol2
      mountPath: /mnt/cephfs/data
      readOnly: false
  volumes:
  - name: ceph-vol2
    persistentVolumeClaim:
      claimName: foo-claim

创建pv、pvc:

# kubectl create -f ceph-pv.yaml
persistentvolume "foo-pv" created
# kubectl create -f ceph-pvc.yaml
persistentvolumeclaim "foo-claim" created

# kubectl get pvc
NAME        STATUS    VOLUME    CAPACITY   ACCESSMODES   AGE
foo-claim   Bound     foo-pv    512Mi      RWX           4s
# kubectl get pv
NAME      CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM               REASON    AGE
foo-pv    512Mi      RWX           Recycle         Bound     default/foo-claim             24s

启动pod,通过exec命令查看挂载情况:

# docker ps|grep pod
a6895ec0274f        ubuntu:14.04                                                  "tail -f /var/log/boo"   About a minute ago   Up About a minute                                                                        k8s_ceph-ubuntu2.66c44128_ceph-pod2_default_4e4fc8d4-33c6-11e7-bcd9-6640d35a0e90_1b37ed76
52b6811a6584        gcr.io/google_containers/pause-amd64:3.0                      "/pause"                 About a minute ago   Up About a minute                                                                        k8s_POD.d8dbe16c_ceph-pod2_default_4e4fc8d4-33c6-11e7-bcd9-6640d35a0e90_27e5f988
55b96edbf4bf        ubuntu:14.04                                                  "tail -f /var/log/boo"   14 minutes ago       Up 14 minutes                                                                            k8s_ceph-ubuntu2.66c44128_ceph-pod2-with-secret_default_9d383b0c-33c4-11e7-bcd9-6640d35a0e90_1656e5e0
f8b699bc0459        gcr.io/google_containers/pause-amd64:3.0                      "/pause"                 14 minutes ago       Up 14 minutes                                                                            k8s_POD.d8dbe16c_ceph-pod2-with-secret_default_9d383b0c-33c4-11e7-bcd9-6640d35a0e90_effdfae7
root@yypdnode:~# docker exec a6895ec0274f ls /mnt/cephfs/data
1.txt
apps
ceph-volume1-test
foo.txt
test1.txt

# docker exec a6895ec0274f cat /mnt/cephfs/data/foo.txt
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World
pod2-with-secret-on-master: Hello, World

五、pv的状态

如果你不删除pvc,一切都安然无事:

# kubectl get pv
NAME                CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                        REASON    AGE
foo-pv              512Mi      RWX           Recycle         Bound     default/foo-claim                      1h

# kubectl get pvc
NAME                 STATUS    VOLUME              CAPACITY   ACCESSMODES   AGE
foo-claim            Bound     foo-pv              512Mi      RWX           1h

但是如果删除pvc,pv的状态将变成failed:

删除pvc:

# kubectl get pv
NAME                CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                        REASON    AGE
foo-pv              512Mi      RWX           Recycle         Failed    default/foo-claim                      2h

# kubectl describe pv/foo-pv
Name:        foo-pv
Labels:        <none>
Status:        Failed
Claim:        default/foo-claim
Reclaim Policy:    Recycle
Access Modes:    RWX
Capacity:    512Mi
Message:    No recycler plugin found for the volume!
Source:
    Type:        RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
    CephMonitors:    [xx.xx.xx.xx:6789]
    RBDImage:        foo1
    FSType:        ext4
    RBDPool:        rbd
    RadosUser:        admin
    Keyring:        /etc/ceph/keyring
    SecretRef:        &{ceph-secret}
    ReadOnly:        false
Events:
  FirstSeen    LastSeen    Count    From                SubobjectPath    Type        Reason            Message
  ---------    --------    -----    ----                -------------    --------    ------            -------
  29s        29s        1    {persistentvolume-controller }            Warning        VolumeFailedRecycle    No recycler plugin found for the volume!

我们在pv中指定的persistentVolumeReclaimPolicy是Recycle,但无论是cephrbd还是cephfs都不没有对应的recycler plugin,导致pv的status变成了failed,只能手工删除重建。




这里是Tony Bai的个人Blog,欢迎访问、订阅和留言!订阅Feed请点击上面图片

如果您觉得这里的文章对您有帮助,请扫描上方二维码进行捐赠,加油后的Tony Bai将会为您呈现更多精彩的文章,谢谢!

如果您喜欢通过微信App浏览本站内容,可以扫描下方二维码,订阅本站官方微信订阅号“iamtonybai”;点击二维码,可直达本人官方微博主页^_^:



本站Powered by Digital Ocean VPS。

选择Digital Ocean VPS主机,即可获得10美元现金充值,可免费使用两个月哟!

著名主机提供商Linode 10$优惠码:linode10,在这里注册即可免费获得。

阿里云推荐码:1WFZ0V立享9折!

View Tony Bai's profile on LinkedIn


文章

评论

  • 正在加载...

分类

标签

归档











更多