标签 docker 下的文章

Kuberize Ceph RBD API服务

在《使用Ceph RBD为Kubernetes集群提供存储卷》一文中,我们提到:借助KubernetesCeph的集成,Kubernetes可以使用Ceph RBD为集群内的Pod提供Persistent Volume。但这一过程中,RBD所使用的image的创建、删除还需要手动管理,于是我们又基于go-ceph实现了对RBD image的程序化管理,我们的最终目标是要这种对RBD image的管理服务以一个K8s service的形式发布到Kubernetes集群中去,这就是本文标题中描述的那样:Kuberize Ceph RBD API服务。

一、Dockerize Ceph RBD API服务

要想使得ceph rbd api Kuberizable,首先要Dockerize Ceph RBD API Service,即容器化。由于go-ceph是Go语言开发,我们的rbd-rest-api同样用Go语言开发。使用Go语言开发有一个众所周知的好处,那就是可以编译为静态二进制文件,可以在运行时不依赖任何外部库,生来自带“适合容器”标签。但由于go-ceph是一个go binding for librados和librbd,其通过cgo实现Go语言对C库的链接和调用。这样一来,我们如果要做static linking,那么我们就要准备齐全所有librados和librbd所依赖的第三方库的.a(archive file)。如果你仅仅是执行下面编译命令,你将得到w行级别的错误信息输出:

$ go build --ldflags '-extldflags "-static"' .

从错误的信息中,我们可以得到rbd-rest-api静态编译依赖的各种第三方库,包括boost库(apt-get install libboost-all-dev)、libssl(apt-get install libssl)以及libnss3(apt-get install libnss3-dev)。安装好这些库,再修改一下命令行,可将编译错误输出降低到百行以内:

# go build --ldflags '-extldflags "-static -L /usr/lib/x86_64-linux-gnu -lboost_system -lboost_thread -lboost_iostreams -lboost_random -lcrypto -ldl -lpthread -lm -lz  -lc -L /usr/lib/gcc/x86_64-linux-gnu/4.8/ -lstdc++"' .

不过,你将依旧得到诸多错误:

... ...
/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../../lib/librados.a(Crypto.o): In function `CryptoAESKeyHandler::init(ceph::buffer::ptr const&, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&)':
/build/ceph-10.2.3/src/auth/Crypto.cc:280: undefined reference to `PK11_GetBestSlot'
/build/ceph-10.2.3/src/auth/Crypto.cc:291: undefined reference to `PK11_ImportSymKey'
/build/ceph-10.2.3/src/auth/Crypto.cc:304: undefined reference to `PK11_ParamFromIV'
/build/ceph-10.2.3/src/auth/Crypto.cc:282: undefined reference to `PR_GetError'
/build/ceph-10.2.3/src/auth/Crypto.cc:293: undefined reference to `PR_GetError'
... ...

这些”undefined reference”指向的符号都是libnss3-dev库中的,但由于libnss3-dev的安装并没有包含libnss3.a文件,因此即便将libnss3显式放在链接参数列表中,比如:”-lnss3″也无法链接成功:

/usr/bin/ld: cannot find -lnss3

libnss库着实不是一个省油灯,经过几番折腾发现,要想使用libnss的static archive,我们只能手工编译,代码在这里可以获取到:https://github.com/nss-dev/nss,并且这里提供了nss的手工编译方法。

综上可以看出,纯静态编译rbd-rest-api是很繁琐的,于是我们这次选择默认的动态链接方式,我们只需在docker image中安装librados和librbd这两个依赖库即可,于是rbd-rest-api的Dockerfile的雏形可见:

From ubuntu:14.04
MAINTAINER Tony Bai <author@xxx.com>

# use aliyun source for ubuntu
# before building image ,make sure copy /etc/apt/sources.list here
# COPY sources.list /etc/apt/

RUN apt-get update && apt-get install -y --no-install-recommends librados-dev librbd-dev \
                   && rm -rf /var/lib/apt/lists/*

RUN mkdir -p /root/rbd-rest-api
COPY rbd-rest-api /root/rbd-rest-api
COPY conf /root/rbd-rest-api/conf
RUN chmod +x /root/rbd-rest-api/rbd-rest-api

EXPOSE 8080
WORKDIR /root/rbd-rest-api
ENTRYPOINT ["/root/rbd-rest-api/rbd-rest-api"]

我们一直在Ubuntu 14.04.x环境下进行各种测试,于是我们自然而然的选择ubuntu:14.04作为我们的base image,构建镜像:

# docker build -t "test/rbd-rest-api" .
... ...

Setting up librados-dev (0.80.11-0ubuntu1.14.04.1) ...
Setting up librbd-dev (0.80.11-0ubuntu1.14.04.1) ...
Processing triggers for libc-bin (2.19-0ubuntu6.9) ...
 ---> c987abc7a24d
Removing intermediate container 5257ac37392a
Step 5 : RUN mkdir -p /root/rbd-rest-api
 ---> Running in dcabdb990c60
 ---> ce0db2a027aa
Removing intermediate container dcabdb990c60
Step 6 : COPY rbd-rest-api /root/rbd-rest-api
 ---> 453fd4b9a27a
Removing intermediate container 8b07b5de7537
Step 7 : COPY conf /root/rbd-rest-api/conf
 ---> e956add07d60
Removing intermediate container 6eaf6e4cf334
Step 8 : RUN chmod +x /root/rbd-rest-api/rbd-rest-api
 ---> Running in cb278d1919c7
 ---> 1e7b86072011
Removing intermediate container cb278d1919c7
Step 9 : EXPOSE 8080
 ---> Running in 6a3f457eefca
 ---> e60cefb50f77
Removing intermediate container 6a3f457eefca
Step 10 : WORKDIR /root/rbd-rest-api
 ---> Running in 703baf8c5564
 ---> 6f1a5e5e145c
Removing intermediate container 703baf8c5564
Step 11 : ENTRYPOINT /root/rbd-rest-api/rbd-rest-api
 ---> Running in 16dd4e7e3995
 ---> 43f885b958c7
Removing intermediate container 16dd4e7e3995
Successfully built 43f885b958c7

# docker images
REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
test/rbd-rest-api                                      latest              43f885b958c7        57 seconds ago      298 MB

测试启动镜像,注意我们“只读”挂载了本地路径/etc/ceph:

# docker run --name rbd-rest-api --rm -p 8080:8080 -v /etc/ceph/:/etc/ceph/:ro test/rbd-rest-api
2016/11/14 14:58:17 [I] [asm_amd64.s:2086] http server Running on http://:8080

我们来测试一下这个Docker中的rbd-rest-api service:

# curl  -v   http://localhost:8080/api/v1/pools/
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /api/v1/pools/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 130
< Content-Type: application/json; charset=utf-8
* Server beegoServer:1.7.1 is not blacklisted
< Server: beegoServer:1.7.1
< Date: Mon, 14 Nov 2016 14:59:29 GMT
<
{
  "Kind": "PoolList",
  "APIVersion": "v1",
  "Items": [
    {
      "name": "rbd"
    },
    {
      "name": "rbd1"
    }
  ]
* Connection #0 to host localhost left intact
}

测试OK。

这里不得不提的是:如果你挂载的是仅仅是/etc/ceph/ceph.conf的话,那么当rbd-rest-api服务收到请求后,会返回:

Errcode=300, errmsg:
error rados: No such file or directory

这是因为容器中的rbd-rest-api没有看到ceph.client.admin.keyring,因此在登录ceph monitor时鉴权失败了。当然你也可以不映射本地目录,取而代之的是将/etc/ceph/ceph.conf和/etc/ceph/ceph.client.admin.keyring放入到镜像中,后一种方法这里就不详细描述了。librados给出的错误提示真是太差了,本来应该是一个权限的问题,居然说找不到librados。

二、Kuberize Ceph RBD API服务

容器化测试成功了,接下来就是将Ceph RBD API Kuberize化。根据上面Docker镜像的设计,承载Ceph RBD API服务 Pod的Node上,必须要安装了Ceph client,即包括ceph.conf和ceph.client.admin.keyring,于是有选择性的调度Ceph RBD API服务到安装了ceph client的kubernetes node上是这一节必须考虑的问题。

我们的思路是将rbd-rest-api的pod通过k8s调度到带有指定label的k8s node上去,我们给kubernetes集群的node打标签,安装了ceph client的集群node,打的标签为:zone=ceph。

# kubectl label nodes 10.46.181.146 zone=ceph
# kubectl label nodes 10.47.136.60 zone=ceph

# kubectl get nodes --show-labels
NAME            STATUS    AGE       LABELS
10.46.181.146   Ready     32d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.46.181.146,zone=ceph
10.47.136.60    Ready     32d       beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=10.47.136.60,zone=ceph

接下来就是在rbd-rest-api service的yaml中设定pod的调度策略了:

//rbd-rest-api.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: rbd-rest-api
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: rbd-rest-api
    spec:
      containers:
      - name: rbd-rest-api
        image: registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest
        #imagePullPolicy: IfNotPresent
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        volumeMounts:
        - mountPath: /etc/ceph
          name: ceph-default-config-volume
      volumes:
      - name: ceph-default-config-volume
        hostPath:
          path: /etc/ceph
      nodeSelector:
        zone: ceph
      imagePullSecrets:
      - name: rbd-rest-api-default-secret

---
apiVersion: v1
kind: Service
metadata:
  name: rbd-rest-api
  labels:
    app: rbd-rest-api
spec:
  ports:
  - port: 8080
  selector:
    app: rbd-rest-api

我们可以看到在Deployment的spec中有一个nodeSelector,这个设置可以让k8s scheduler在调度service时只选择具备zone=ceph label的Node。注意关于imagePullSecrets的设置,可以参考《Kubernetes从Private Registry中拉取容器镜像的方法》一文。

Kubernetes集群中的Nginx配置热更新方案

Nginx已经是互联网IT业界一个无敌的存在,作为反向代理、负载均衡、Web服务器等多种角色的扮演者,Nginx在全球各个互联网公司落地、开花和结果,Ngnix已经成为了支撑全球互联网应用的一个不可获取的组成部分。

在我们的平台中,Nginx同样被拿来作为服务接入的最前端的反向代理,并且我们的Nginx也是作为一个Service跑在我们的Kubernetes集群中的。Ngnix背后的服务众多,服务的生生死死都要在Nginx上这些服务路由的配置中有所体现,这就要求部署在Kubernetes集群中的Nginx需要有一个合理的配置热更新方案。

Nginx自身是支持配置热更新的,通过nginx -s reload命令可以实现这一点:

# sudo nginx -s reload

# sudo tail -100f /var/log/nginx/error.log
2016/11/18 08:21:03 [notice] 31516#31516: signal process started

这也是诸多nginx热更新方案的基础。

随着Docker容器以及容器集群/云的出现,Nginx也被Dockerize了,Docker中Nginx的配置热更新方案在Jason Wilder这篇文章中有体现,在该方案中,你可以直接使用Jason Wilder开源的Nginx-proxy实现容器中Nginx的配置的热更新。但这个方案并不能直接适用于Kubernetes,而且作者也并没有Plan support k8s

在Kubernetes集群中部署的Nginx,我其实也找到了一个配置热更新的方案,这是普元的一份技术资料《微服务动态路由实现:OpenResty与kubernetes》中提供的,这个方案通过OpenResty与K8s的结合实现了配置热更新。由于我对OpenResty并不熟悉,并且我个人更希望通过Kubernetes自身的一些Feature来实现这个方案,于是我开始了我自己的探索。

一、需求场景和方案原理

我们要实现的就是:当Kubernetes集群中的Service发生变化时,比如新创建一个Service或删除了一个Service,这些Service在Nginx反向代理中的路由配置需要同步更新并生效。因此,这个过程的场景大致如下:

  • 管理员通过命令或程序通过API操作K8s集群创建或删除Service;
  • 监听API Server Event的某个程序获取该Event,并从API Server读取最新Service数据,重新生成/etc/nginx/conf.d/default.conf;
  • /etc/nginx/conf.d/default.conf文件的变动触发文件变更事件,监听该事件的脚本调用“nginx -s reload”命令实现Nginx的配置热更新。

针对这一需求场景,我这里给出一个实现方案,先上图:

img{512x368}

简答说明一下:

  • Nginx作为一个Service部署在Kubernetes集群中,可以有多个Pod副本;
  • 以一个nginx pod为例,该Pod中包含三个Container,分别是init container、nginx container和config-nginx-generator container;
  • 三个Container共同挂载且共享一个Pod volume,emptyDir类型即可,无需持久化的存储卷,三个Container的挂载路径均为/etc/nginx/conf.d;
  • Pod启动时,init container首先启动并访问API Server,获取Service列表,按照一定条件过滤后(比如通过label的key和Value值),初始创建/etc/nginx/conf.d/default.conf。创建成功后,Container退出;
  • nginx container启动,加载配置,开始提供反向代理服务,并通过inotify工具监视/etc/nginx/conf.d/default.conf文件状态变化,一般变化,就执行nginx -s reload热加载最新配置。
  • config-nginx-generator container同时也启动起来,监听API Server的service变更Event,一旦有Event出现,就重新读取API Server中的Service list,并重新生成一份新的default.conf,覆盖old版本 default.conf。

二、环境

由于KubernetesDocker都在Active Develop的过程中,两个项目的变动都很快,因此,特定的Feature(比如k8s的init container)、操作和说明在某些版本是好用的,但对另外一些版本却是不灵光的。这里先把环境确定清楚,避免误导。

OS:
Ubuntu 14.04.4 LTS Kernel:3.19.0-70-generic #78~14.04.1-Ubuntu SMP Fri Sep 23 17:39:18 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Docker:
# docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 17:00:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 17:00:50 2016
 OS/Arch:      linux/amd64

Kubernetes集群:1.3.7

私有镜像仓库:阿里云镜像仓库

三、实现

1、nginx image的创建

nginx image实现了两个功能,一个自然是nginx自身了,另外一个就是监听/etc/nginx/conf.d/default.conf文件的变化,并适时调用nginx -s reload更新nginx配置。在kubernetes的源码目录kubernetes/examples下有一个例子:https-nginx,这里面已经为我们实现了一个基于auto-reload-nginx.sh的Nginx image Dockerfile,我们稍作改造就可以直接使用了:

//Dockerfile

FROM nginx
MAINTAINER Tony Bai <bigwhite.cn@aliyun.com>

COPY auto-reload-nginx.sh /home/auto-reload-nginx.sh
RUN chmod +x /home/auto-reload-nginx.sh

# install inotify
RUN apt-get update && apt-get install -y inotify-tools

基于该Dockefile构建image:

# docker build -t xxxx/nginx

# docker images
REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
xxxx/nginx                                            latest              a1503b1c2b70        42 seconds ago      191.9 MB

官方nginx image基于debian jessie版本构建,apt-get update & install时需要耐心等待一下。

打标签并推送到我们的阿里云私有镜像库

# docker tag a1503b1c2b70 registry.cn-hangzhou.aliyuncs.com/xxxx/nginx

# docker images
REPOSITORY                                             TAG                 IMAGE ID            CREATED             SIZE
xxxx/nginx                                            latest              a1503b1c2b70        12 minutes ago      191.9 MB
registry.cn-hangzhou.aliyuncs.com/xxxx/nginx          latest              a1503b1c2b70        12 minutes ago      191.9 MB

# docker push registry.cn-hangzhou.aliyuncs.com/xxxx/nginx
2、编写Pod yaml

由于init container和config-nginx-generator container在真实场景中都是要与Kubernetes的API Server交互,并生成/etc/nginx/conf.d/default.conf,这需要一个实现过程,在这里我们暂不给出两个Container的具体Dockerfile以及实现功能的实际程序,而是用两个通用docker image,并通过“手动”方式实现它们各自的功能。因此,我们在这一节中就可以给出Nginx Pod的yaml描述文件了:

//nginx-reload-on-k8s.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx-reload-on-k8s
  annotations:
    pod.beta.kubernetes.io/init-containers: '[
      {
           "name": "nginx-reload-on-k8s-init-1",
           "image": "busybox",
           "command": ["wget", "-O", "/etc/nginx/conf.d/index1.html", "http://www.baidu.com"],
           "volumeMounts": [
               {
                  "name": "conf-volume",
                  "mountPath": "/etc/nginx/conf.d"
               }
           ]
      },
      {
           "name": "nginx-reload-on-k8s-init-2",
           "image": "busybox",
           "command": ["wget", "-O", "/etc/nginx/conf.d/index2.html", "http://dict.cn"],
           "volumeMounts": [
               {
                  "name": "conf-volume",
                  "mountPath": "/etc/nginx/conf.d"
               }
           ]
      }
    ]'
spec:
  containers:
  - name: nginx-config-generator
    volumeMounts:
    - mountPath: /etc/nginx/conf.d
      name: conf-volume
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/test:latest
    imagePullPolicy: IfNotPresent
    command:
       - "tail"
       - "-f"
       - "/var/log/bootstrap.log"
  - name: nginx-origin
    volumeMounts:
    - mountPath: /etc/nginx/conf.d
      name: conf-volume
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/nginx:latest
    imagePullPolicy: IfNotPresent
    command: ["/home/auto-reload-nginx.sh"]
    ports:
    - containerPort: 80
  volumes:
  - name: conf-volume
    emptyDir: {}

Yaml中,我们创建了两个init container,分别用于从baidu.com和dict.cn抓取主页,并存储于/etc/nginx/conf.d的下面备用。nginx-config-generator我们使用image xxxx/test,这就是一个基于ubuntu且安装了诸多网络工具的镜像,用于做目标镜像调试的;nginx container用的就是上面push到私有镜像仓库的那个镜像,command则是执行/home/auto-reload-nginx.sh这个脚本,从而启动nginx和通过inotify监控/etc/nginx/conf.d/default.conf文件。

我们来创建这个Pod(注意:只有用kubectl apply命令时,init container才会被创建和执行,如果用kubectl create -f ,那么将忽略init container):

# kubectl apply -f nginx-reload-on-k8s.yaml
pod "nginx-reload-on-k8s" created

# kubectl get pod
NAME                           READY     STATUS             RESTARTS   AGE
nginx-reload-on-k8s            2/2       Running            0          41s

通过describe pod/nginx-reload-on-k8s,我们能看到一些Container创建的详细信息:

# kubectl describe pod/nginx-reload-on-k8s
Name:        nginx-reload-on-k8s
Namespace:    default
Node:        10.46.181.146/10.46.181.146
Start Time:    Thu, 17 Nov 2016 21:39:55 +0800
Labels:        <none>
Status:        Running
IP:        172.16.57.9
... ...

Events:
  FirstSeen    LastSeen    Count    From            SubobjectPath                    Type        Reason        Message
  ---------    --------    -----    ----            -------------                    --------    ------        -------
  57s        57s        1    {default-scheduler }                            Normal        Scheduled    Successfully assigned nginx-reload-on-k8s to 10.46.181.146
  39s        39s        1    {kubelet 10.46.181.146}    spec.initContainers{nginx-reload-on-k8s-init-1}    Normal        Created        Created container with docker id 0e21afb58eee
  39s        39s        1    {kubelet 10.46.181.146}    spec.initContainers{nginx-reload-on-k8s-init-1}    Normal        Started        Started container with docker id 0e21afb58eee
  56s        38s        2    {kubelet 10.46.181.146}    spec.initContainers{nginx-reload-on-k8s-init-1}    Normal        Pulling        pulling image "busybox"
  39s        26s        2    {kubelet 10.46.181.146}    spec.initContainers{nginx-reload-on-k8s-init-1}    Normal        Pulled        Successfully pulled image "busybox"
  26s        26s        1    {kubelet 10.46.181.146}    spec.initContainers{nginx-reload-on-k8s-init-2}    Normal        Created        Created container with docker id 85632ff73ea8
  26s        26s        1    {kubelet 10.46.181.146}    spec.initContainers{nginx-reload-on-k8s-init-2}    Normal        Started        Started container with docker id 85632ff73ea8
  25s        25s        1    {kubelet 10.46.181.146}    spec.containers{nginx-config-generator}        Normal        Pulled        Container image "registry.cn-hangzhou.aliyuncs.com/xxxx/test:latest" already present on machine
  25s        25s        1    {kubelet 10.46.181.146}    spec.containers{nginx-config-generator}        Normal        Created        Created container with docker id 1ce8c6d8a8af
  25s        25s        1    {kubelet 10.46.181.146}    spec.containers{nginx-config-generator}        Normal        Started        Started container with docker id 1ce8c6d8a8af
  25s        25s        1    {kubelet 10.46.181.146}    spec.containers{nginx-origin}            Normal        Pulled        Container image "registry.cn-hangzhou.aliyuncs.com/xxxx/nginx:latest" already present on machine
  25s        25s        1    {kubelet 10.46.181.146}    spec.containers{nginx-origin}            Normal        Created        Created container with docker id 0c692ec28acd
  25s        25s        1    {kubelet 10.46.181.146}    spec.containers{nginx-origin}            Normal        Started        Started container with docker id 0c692ec28acd

... ...

可以看到四个container依次被pull and create。

四、测试

现在我们就来测试一下nginx的reload。

之前的两个init container分别在/etc/nginx/conf.d下创建了index1.html和index2.html,我们就用这两个文件分别作为配置变更前和变更后的首页。

注意:这时我们还没有/etc/nginx/conf.d/default.conf文件,我们在Pod内访问localhost:80将会得到失败结果:

# curl localhost:80
curl: (7) Failed to connect to localhost port 80: Connection refused

我们进入nginx-config-generator,创建/etc/nginx/conf.d/default.conf文件,与此同时,通过docker logs -f 监控nginx-origin容器的日志:

//default.conf

server {
    listen       80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/log/host.access.log  main;

    location / {
        root   /etc/nginx/conf.d;
        index  index1.html index1.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}

我们把/etc/nginx/conf.d/index1.html作为服务站点的首页了。文件创建完毕后,我们同时就可以从nginx-origin容器的日志能看到如下内容:

At 14:07 on 17/11/16, config file update detected.
2016/11/17 14:07:25 [notice] 20#20: signal process started

我们再从Pod中访问localhost:80(注意:Pod中的多个container共享network namespace,通过localhost就可以进行互访):

root@nginx-reload-on-k8s:/etc/nginx# curl localhost:80
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下,你就知道</title></head> .... </html>

我们顺利得到index1.html的内容,这说明配置实时生效了。

我们再来“触发”一次配置变更。我们将default.conf中的:

location / {
        root   /etc/nginx/conf.d;
        index  index1.html index1.htm;
    }

改为:

location / {
        root   /etc/nginx/conf.d;
        index  index2.html index2.htm;
    }

保存!

从nginx-origin容器日志可以看到如下输出:

At 14:17 on 17/11/16, config file update detected.
2016/11/17 14:17:46 [notice] 32#32: signal process started

在Pod中再次访问站点首页:

# curl localhost:80
<!DOCTYPE HTML>
<html>
    <head>
        <meta name="renderer" content="webkit"/>
                <meta http-equiv="X-UA-Compatible" content="IE=EmulateIE7" />
                <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
        <title>海词词典_在线词典_在线翻译_海量正版权威词典官方网站</title>
... ...

可以看到配置更新成功,首页换成了dict.cn的首页。

五、测试

通过上述这些“手动”的触发和测试,可以看出这个方案是可行的。并且我们可以看出,这个方案是有一些好处的:

  • 不需要依赖外部持久化存储卷;
  • 通过k8s api server获取当前所有 service列表,通过service label来过滤,无需依赖额外的redis server或etcd服务;

剩下的就是具体init container以及config-generator的实现了。这个留给我以及大家后续去完成^_^。

Kubernetes从Private Registry中拉取容器镜像的方法

话接上文,在《使用go-ceph管理Ceph RBD映像》一文中我们提到了,我们需要自建一个ceph rbd api service用于给我的产品控制台提供RESTful API服务接口。这个服务我也是打算放在kubernetes集群中作为一个Service运行的。这两天完成了这个服务开发,并编写完Service的Dockerfile,将镜像build, tag并push到了我们在阿里云的私有镜像库。但在通过kubectl创建这个Service时,我们遇到了 ErrImagePull、ImagePullBackOff等Pod status,通过kubectl describe pod/{MyPod}命令查看,发现下面错误提示:

  23s    5s    2    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api}    Warning    Failed        Failed to pull image "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest": image pull failed for registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest, this may be because there are no credentials on this request.  details: (Error: image xxxx/rbd-rest-api:latest not found)

面前这个坑就是Kubernetes集群如何从Private Registry获取容器镜像的问题。关于这个问题,K8s官方文档有较为详细的说明,但填过坑的人都知道,那些说明还是远远不够的,实践中你会碰到很多意想不到的问题。这里就来结合实际操作说说K8s与私有容器镜像仓库是如何在一起欢乐的工作的^_^。

一、环境

由于KubernetesDocker都在Active Develop的过程中,两个项目的变动都很快,因此,特定的操作和说明在某些版本是好用的,但对另外一些版本却是不灵光的。这里先把环境确定清楚,避免误导。

OS:
Ubuntu 14.04.4 LTS Kernel:3.19.0-70-generic #78~14.04.1-Ubuntu SMP Fri Sep 23 17:39:18 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Docker:
# docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 17:00:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        Tue Oct 11 17:00:50 2016
 OS/Arch:      linux/amd64

Kubernetes集群:1.3.7

私有镜像仓库:阿里云镜像仓库

Docker镜像:非公共镜像,大家在测试中可以在自己的私有仓库建立自己的测试镜像
registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest

Kubernets在文档中描述了几种访问私有仓库的方法,这里挑选了那些可操作的,逐一测试一下。

二、方法1:利用Node上的配置访问Private Registry

在玩Docker时,很多朋友都搭建过自己的Private Registry。Docker访问那些以basic auth方式进行鉴权的Private Registry,只需在本地执行docker login,输入用户名、密码后,就可以自由向Registry Push镜像或pull 镜像到本地了:

# docker login registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api
Username: {UserName}
Password:
Login Succeeded

在这一过程结束后,Docker实际上会在~/.docker目录下创建一个config.json文件,保存后续与Registry交互过程中所要使用的鉴权串(这个鉴权串只是一个base64编码结果,安全性欠佳^_^):

# cat ~/.docker/config.json
{
    "auths": {
        "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api": {
            "auth": "xxxxyyyyzzzz"
        }
    }
}

一但Node上有了这个配置,那么K8s就可以通过docker直接访问Private Registry了,这是K8s文档中与私有镜像仓库交互的第一个方法。考虑到Pod可以被调度到集群中的任意一个Node上,需要在每个Node上执行上述login操作,或者可以简单地将~/.docker/config.json scp到各个node上的~/.docker目录下。

实际效果如何呢? 我们创建了一个Pod yaml,测试一下是否能run起来:

//rbd-rest-api-using-node-config.yaml
apiVersion: v1
kind: Pod
metadata:
  name: rbd-rest-api-using-node-config
spec:
  containers:
  - name: rbd-rest-api-using-node-config
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest
    imagePullPolicy: Always

我们来创建一下这个Pod并查看pod的创建状态:

# kubectl create -f rbd-rest-api-using-node-config.yaml
pod "rbd-rest-api-using-node-config" created
# kubectl get pods
NAME                             READY     STATUS             RESTARTS   AGE
rbd-rest-api-using-node-config   0/1       ErrImagePull       0          5s

通过describe查看Pod失败的详细信息:

# kubectl describe pod/rbd-rest-api-using-node-config
... ...

Events:
  FirstSeen    LastSeen    Count    From            SubobjectPath                    Type        Reason        Message
  ---------    --------    -----    ----            -------------                    --------    ------        -------
  1m        1m        1    {default-scheduler }                            Normal        Scheduled    Successfully assigned rbd-rest-api-using-node-config to 10.66.181.146
  1m        42s        3    {kubelet 10.66.181.146}    spec.containers{rbd-rest-api-using-node-config}    Normal        Pulling        pulling image "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest"
  1m        42s        3    {kubelet 10.66.181.146}    spec.containers{rbd-rest-api-using-node-config}    Warning        Failed        Failed to pull image "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest": image pull failed for registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest, this may be because there are no credentials on this request.  details: (Error: image xxxx/rbd-rest-api:latest not found)
  1m        42s        3    {kubelet 10.66.181.146}                            Warning        FailedSync    Error syncing pod, skipping: failed to "StartContainer" for "rbd-rest-api-using-node-config" with ErrImagePull: "image pull failed for registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest, this may be because there are no credentials on this request.  details: (Error: image xxxx/rbd-rest-api:latest not found)"
... ...

这个方法对我们的环境并不有效。并且经过多次测试,结果依旧,K8s无法从Private Registry获取我们想要的镜像文件:(。

三、方法2:通过kubectl创建docker-registry的secret

K8s提供的第二种方法是通过kubectl创建一个 docker-registry的secret,并在Pod描述文件中引用该secret以达到从Private Registry Pull Image的目的。

操作之前,我们先删除掉各个Node上的~/.docker/config.json。

执行kubectl create secret docker-registry时需要提供private registry的访问UserName和Password:

# kubectl create secret docker-registry registrykey-m2-1 --docker-server=registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api --docker-username={UserName} --docker-password={Password} --docker-email=team@domain.com
secret "registrykey-m2-1" created

# kubectl get secret
NAME                  TYPE                                  DATA      AGE
registrykey-m2-1      kubernetes.io/dockercfg               1         29s

secret: registrykey-m2-1创建成功。我们来测试一下引用这个secret对象的Pod是否能Pull Image成功并Run起来。Pod yaml文件如下:

//rbd-rest-api-registrykey-m2-1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: rbd-rest-api-registrykey-m2-1
spec:
  containers:
  - name: rbd-rest-api-registrykey-m2-1
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest
    imagePullPolicy: Always
  imagePullSecrets:
  - name: registrykey-m2-1

创建Pod,并观察Pod状态:

# kubectl create -f rbd-rest-api-registrykey-m2-1.yaml
pod "rbd-rest-api-registrykey-m2-1" created

# kubectl get pods
NAME                             READY     STATUS             RESTARTS   AGE
rbd-rest-api-registrykey-m2-1    1/1       Running            0          7s
rbd-rest-api-using-node-config   0/1       ImagePullBackOff   0          29m

通过describe pod,查看创建的event序列:

Events:
  FirstSeen    LastSeen    Count    From            SubobjectPath                    Type        Reason        Message
  ---------    --------    -----    ----            -------------                    --------    ------        -------
  1m        1m        1    {default-scheduler }                            Normal        Scheduled    Successfully assigned rbd-rest-api-registrykey-m2-1 to 10.57.136.60
  1m        1m        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-registrykey-m2-1}    Normal        Pulling        pulling image "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest"
  1m        1m        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-registrykey-m2-1}    Normal        Pulled        Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest"
  1m        1m        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-registrykey-m2-1}    Normal        Created        Created container with docker id d842565e762d
  1m        1m        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-registrykey-m2-1}    Normal        Started        Started container with docker id d842565e762d

正如我们期望的那样,引用了secret: registrykey-m2-1的Pod成功Run起来了。

如果一个pod中有来自不同私有仓库的不同镜像,我们需要怎么做呢?通过kubectl create secret docker-registry我们一次只能建立一个registrykey,如果要访问两个镜像仓库,我们就需要分别为每个仓库创建一个registrykey。我们再来创建一个registrykey,对应的仓库为:registry.cn-hangzhou.aliyuncs.com/xxxx/test:

# kubectl create secret docker-registry registrykey-m2-2 --docker-server=registry.cn-hangzhou.aliyuncs.com/xxxx/test --docker-username={UserName} --docker-password={Password} --docker-email=team@domain.com
secret "registrykey-m2-2" created

root@node1:~/pullimagetest/test# kubectl get secret
NAME                  TYPE                                  DATA      AGE
registrykey-m2-1      kubernetes.io/dockercfg               1         1h
registrykey-m2-2      kubernetes.io/dockercfg               1         6s

接下来,我们来建一个包含多个container的Pod:

//rbd-rest-api-multi-registrykeys-m2-2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: rbd-rest-api-multi-registrykeys-m2-2
spec:
  containers:
  - name: rbd-rest-api-multi-registrykeys-m2-2
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest
    imagePullPolicy: Always
  - name: test-multi-registrykeys-m2-2
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/test:latest
    imagePullPolicy: Always
    command:
       - "tail"
       - "-f"
       - "/var/log/bootstrap.log"
  imagePullSecrets:
  - name: registrykey-m2-1
  - name: registrykey-m2-2

在secret引用中,我们将两个key都引用了进来。

创建该Pod:

# kubectl create -f rbd-rest-api-multi-registrykeys-m2-2.yaml
pod "rbd-rest-api-multi-registrykeys-m2-2" created

# kubectl get pod
NAME                                   READY     STATUS             RESTARTS   AGE
rbd-rest-api-multi-registrykeys-m2-2   2/2       Running            0          5s

通过pod的event,我们看看启动的操作顺序:

Events:
  FirstSeen    LastSeen    Count    From            SubobjectPath                        Type        Reason        Message
  ---------    --------    -----    ----            -------------                        --------    ------        -------
  44s        44s        1    {default-scheduler }                                Normal        Scheduled    Successfully assigned rbd-rest-api-multi-registrykeys-m2-2 to 10.57.136.60
  43s        43s        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-multi-registrykeys-m2-2}    Normal        Pulling        pulling image "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest"
  43s        43s        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-multi-registrykeys-m2-2}    Normal        Pulled        Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest"
  42s        42s        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-multi-registrykeys-m2-2}    Normal        Created        Created container with docker id 7c09048a41f6
  42s        42s        1    {kubelet 10.57.136.60}    spec.containers{rbd-rest-api-multi-registrykeys-m2-2}    Normal        Started        Started container with docker id 7c09048a41f6
  42s        42s        1    {kubelet 10.57.136.60}    spec.containers{test-multi-registrykeys-m2-2}        Normal        Pulling        pulling image "registry.cn-hangzhou.aliyuncs.com/xxxx/test:latest"
  42s        42s        1    {kubelet 10.57.136.60}    spec.containers{test-multi-registrykeys-m2-2}        Normal        Pulled        Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/xxxx/test:latest"
  42s        42s        1    {kubelet 10.57.136.60}    spec.containers{test-multi-registrykeys-m2-2}        Normal        Created        Created container with docker id 9930834fe4a3
  42s        42s        1    {kubelet 10.57.136.60}    spec.containers{test-multi-registrykeys-m2-2}        Normal        Started        Started container with docker id 9930834fe4a3

k8s分别从两个镜像仓库尝试pull image,并且最终都成功了!

四、方法3:通过secret yaml文件创建pull image所用的secret

除了上面通过kubectl可以快捷的创建pull image所用的secret外,我们还可以使用常规的手段-yaml描述文件来创建我们需要的secret资源。

//registrykey-m3-1.yaml
apiVersion: v1
kind: Secret
metadata:
  name: registrykey-m3-1
  namespace: default
data:
    .dockerconfigjson: {base64 -w 0 ~/.docker/config.json}
type: kubernetes.io/dockerconfigjson

前面说过docker login会在~/.docker下面创建一个config.json文件保存鉴权串,这里secret yaml的.dockerconfigjson后面的数据就是那个json文件的base64编码输出(-w 0让base64输出在单行上,避免折行)。

创建registrykey-m3-1 secret:

# kubectl create -f registrykey-m3-1.yaml
secret "registrykey-m3-1" created

# kubectl get secret
NAME                  TYPE                                  DATA      AGE
myregistrykey3        kubernetes.io/dockerconfigjson        1         3h
registrykey-m2-1      kubernetes.io/dockercfg               1         1h
registrykey-m2-2      kubernetes.io/dockercfg               1         23m
registrykey-m3-1      kubernetes.io/dockerconfigjson        1         29s

对比后,我们发现通过kubectl和yaml创建的两个registrykey secret的类型略有不同,前者是kubernetes.io/dockercfg,后者是kubernetes.io/dockerconfigjson。

接下来,我们编写一个引用了registrykey-m3-1的Pod:

//rbd-rest-api-registrykey-m3-1.yaml
apiVersion: v1
kind: Pod
metadata:
  name: rbd-rest-api-registrykey-m3-1
spec:
  containers:
  - name: rbd-rest-api-registrykey-m3-1
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest
    imagePullPolicy: Always
  imagePullSecrets:
  - name: registrykey-m3-1

创建Pod:

# kubectl create -f rbd-rest-api-registrykey-m3-1.yaml
pod "rbd-rest-api-registrykey-m3-1" created
# kubectl get pods
NAME                            READY     STATUS             RESTARTS   AGE
rbd-rest-api-registrykey-m3-1   1/1       Running            0          8s

创建成功。

那么这种方法如何应对含有来自多个镜像仓库container的Pod的呢?这里的思路与方法2略有不同。我们不需要创建并引用两个或多个secret,而是创建一个可以访问多个私有镜像仓库的secret,我们需要将多个镜像仓库的访问鉴权串都放到~/.docker/config.json中:

按照方法1的介绍,我们先login registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api,得到config.json如下:

{
    "auths": {
        "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api": {
            "auth": "....省略...."
        }
    }
}

我们再login registry.cn-hangzhou.aliyuncs.com/xxxx/test,得到config.json如下:

{
    "auths": {
        "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api": {
            "auth": "....省略...."
        },
        "registry.cn-hangzhou.aliyuncs.com/xxxx/test": {
            "auth": "....省略...."
        }
    }
}

我们看到Docker自动将新login的private registry的鉴权串merge到了同一个config.json中了。现在我们基于该包含了两个库鉴权串的config.json创建一个新secret:registrykey-m3-2:

//registrykey-m3-2.yaml

apiVersion: v1
kind: Secret
metadata:
  name: registrykey-m3-2
  namespace: default
data:
  .dockerconfigjson: {base64 -w 0 ~/.docker/config.json}
type: kubernetes.io/dockerconfigjson

创建secret: registrykey-m3-2

# kubectl create -f registrykey-m3-2.yaml
secret "registrykey-m3-2" created

# kubectl get secrets
NAME                  TYPE                                  DATA      AGE
registrykey-m2-1      kubernetes.io/dockercfg               1         1h
registrykey-m2-2      kubernetes.io/dockercfg               1         42m
registrykey-m3-1      kubernetes.io/dockerconfigjson        1         19m
registrykey-m3-2      kubernetes.io/dockerconfigjson        1         6s

我们编辑一个包含两个容器,引用secret “registrykey-m3-2″ 的Pod yaml:

//rbd-rest-api-multi-registrykeys-m3-2.yaml

apiVersion: v1
kind: Pod
metadata:
  name: rbd-rest-api-multi-registrykeys-m3-2
spec:
  containers:
  - name: rbd-rest-api-multi-registrykeys-m3-2
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest
    imagePullPolicy: Always
  - name: test-multi-registrykeys-m3-2
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/test:latest
    imagePullPolicy: Always
    command:
       - "tail"
       - "-f"
       - "/var/log/bootstrap.log"
  imagePullSecrets:
  - name: registrykey-m3-2

创建该Pod:

# kubectl create -f rbd-rest-api-multi-registrykeys-m3-2.yaml
pod "rbd-rest-api-multi-registrykeys-m3-2" created

# kubectl get pod
NAME                                   READY     STATUS             RESTARTS   AGE
rbd-rest-api-multi-registrykeys-m3-2   2/2       Running            0          4s

Pod创建成功!

五、调用API创建registrykey secret

对比了方法2和方法3,方法2更简洁,方法3更强大。但在任何一个产品中,secret都不应该是手动创建的,在这种情况下,API创建registrykey secret便是必经之路。一旦选择通过API创建,我们显然将依仗着方法2中的原理,将config.json中的内容通过API请求的Body Post给K8s api server。

如何在远端构建出config.json的内容呢继而构建出secret yaml中.dockerconfigjson的值数据呢?我们发现config.json套路中,唯一不确定的就是每个private repository下的auth串,那么这个串是啥呢?你大可base64 -d一下:

# echo -n "VXNlck5hbWU6UGFzc3dvcmQ="|base64 -d
UserName:Password

没错,实质上这个auth串就是UserName:Password的base64编码值。因此,你首先要用某个仓库的UserName和Password按照’UserName:Password’格式进行base64编码,利用编码的结果值构造json内容,比如:

{
    "auths": {
        "registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api": {
            "auth": "VXNlck5hbWU6UGFzc3dvcmQ="
        }
}

然后对这段json数据再做base64编码,所得到的值就是secret yaml中的.dockerconfigjson的值数据。至此,我们来通过API创建一个secret:

$ curl -v -H "Content-type: application/json"  -X POST -d ' {
  "apiVersion": "v1",
  "kind": "Secret",
  "metadata": {
    "name": "registrykey-m4-1",
    "namespace": "default"
  },
  "data": {
    ".dockerconfigjson": "{cat ~/.docker/config.json |base64 -w 0}"
  },
  "type": "kubernetes.io/dockerconfigjson"
}' http://10.57.136.60:8080/api/v1/namespaces/default/secrets

# kubectl get secret
NAME                  TYPE                                  DATA      AGE
registrykey-m2-1      kubernetes.io/dockercfg               1         2h
registrykey-m2-2      kubernetes.io/dockercfg               1         1h
registrykey-m3-1      kubernetes.io/dockerconfigjson        1         43m
registrykey-m3-2      kubernetes.io/dockerconfigjson        1         24m
registrykey-m4-1      kubernetes.io/dockerconfigjson        1         18s

基于registrykey-m4-1,我们启动一个Pod:

//rbd-rest-api-registrykey-m4-1.yaml

apiVersion: v1
kind: Pod
metadata:
  name: rbd-rest-api-registrykey-m4-1
spec:
  containers:
  - name: rbd-rest-api-registrykey-m4-1
    image: registry.cn-hangzhou.aliyuncs.com/xxxx/rbd-rest-api:latest
    imagePullPolicy: Always
  imagePullSecrets:
  - name: registrykey-m4-1

# kubectl create -f rbd-rest-api-registrykey-m4-1.yaml
pod "rbd-rest-api-registrykey-m4-1" created

# kubectl get pod
NAME                            READY     STATUS             RESTARTS   AGE
rbd-rest-api-registrykey-m4-1   1/1       Running            0          5s

Pod创建成功!




这里是Tony Bai的个人Blog,欢迎访问、订阅和留言!订阅Feed请点击上面图片

如果您觉得这里的文章对您有帮助,请扫描上方二维码进行捐赠,加油后的Tony Bai将会为您呈现更多精彩的文章,谢谢!

如果您喜欢通过微信App浏览本站内容,可以扫描下方二维码,订阅本站官方微信订阅号“iamtonybai”;点击二维码,可直达本人官方微博主页^_^:



本站Powered by Digital Ocean VPS。

选择Digital Ocean VPS主机,即可获得10美元现金充值,可免费使用两个月哟!

著名主机提供商Linode 10$优惠码:linode10,在这里注册即可免费获得。

阿里云推荐码:1WFZ0V立享9折!

View Tony Bai's profile on LinkedIn


文章

评论

  • 正在加载...

分类

标签

归档











更多