在本篇文章中，我们继续来说Kubernetes。

经过一段时间的探索，我们先后完成了Kubernetes集群搭建，DNS、Dashboard、Heapster等插件安装，集群安全配置，搭建作为Persistent Volume的CephRBD，以及服务更新等探索和实现工作。现在Kubernetes集群层面的Logging需求逐渐浮上水面了。

随着一些小应用在我们的Kubernetes集群上的部署上线，集群的运行迈上了正轨。但问题随之而来，那就是如何查找和诊断集群自身的问题以及运行于Pod中应用的问题。日志，没错！我们也只能依赖Kubernetes组件以及Pod中应用输出的日志。不过目前我们仅能通过kubectl logs命令或Kubernetes Dashboard来查看Log。在没有cluster level logging的情况下，我们需要分别查看各个Pod的日志，操作繁琐，过程低效。我们迫切地需要为Kubernetes集群搭建一套集群级别的集中日志收集和分析设施。

对于任何基础设施或后端服务系统，日志都是极其重要的。对于受Google内部容器管理系统Borg启发而催生出的Kubernetes项目来说，自然少不了对Logging的支持。在“Logging Overview“中，官方概要介绍了Kubernetes上的几个层次的Logging方案，并给出Cluster-level logging的参考架构：

Kubernetes还给出了参考实现：
– Logging Backend：Elastic Search stack(包括：Kibana)
– Logging-agent：fluentd

ElasticSearch stack实现的cluster level logging的一个优势在于其对Kubernetes集群中的Pod没有侵入性，Pod无需做任何配合性改动。同时EFK/ELK方案在业内也是相对成熟稳定的。

在本文中，我将为我们的Kubernetes 1.3.7集群安装ElasticSearch、Fluentd和Kibana。由于1.3.7版本略有些old，EFK能否在其上面run起来，我也是心中未知。能否像《生化危机：终章》那样有一个完美的结局，我们还需要一步一步“打怪升级”慢慢看。

一、Kubernetes 1.3.7集群的 “漏网之鱼”

Kubernetes 1.3.7集群是通过kube-up.sh搭建并初始化的。按照K8s官方文档有关elasticsearch logging的介绍，在kubernetes/cluster/ubuntu/config-default.sh中，我也发现了下面几个配置项：

// kubernetes/cluster/ubuntu/config-default.sh
# Optional: Enable node logging.
ENABLE_NODE_LOGGING=false
LOGGING_DESTINATION=${LOGGING_DESTINATION:-elasticsearch}

# Optional: When set to true, Elasticsearch and Kibana will be setup as part of the cluster bring up.
ENABLE_CLUSTER_LOGGING=false
ELASTICSEARCH_LOGGING_REPLICAS=${ELASTICSEARCH_LOGGING_REPLICAS:-1}

显然，当初如果搭建集群伊始时要是知道这些配置的意义，可能那个时候就会将elastic logging集成到集群中了。现在为时已晚，集群上已经跑了很多应用，无法重新通过kube-up.sh中断集群运行并安装elastic logging了。我只能手工进行安装了！

二、镜像准备

1.3.7源码中kubernetes/cluster/addons/fluentd-elasticsearch下的manifest已经比较old了，我们直接使用kubernetes最新源码中的manifest文件：

k8s.io/kubernetes/cluster/addons/fluentd-elasticsearch$ ls *.yaml
es-controller.yaml  es-service.yaml  fluentd-es-ds.yaml  kibana-controller.yaml  kibana-service.yaml

分析这些yaml，我们需要三个镜像：

 gcr.io/google_containers/fluentd-elasticsearch:1.22
 gcr.io/google_containers/elasticsearch:v2.4.1-1
 gcr.io/google_containers/kibana:v4.6.1-1

显然镜像都在墙外。由于生产环境下的Docker引擎并没有配置加速器代理，因此我们需要手工下载一下这三个镜像。我采用的方法是通过另外一台配置了加速器的机器上的Docker引擎将三个image下载，并重新打tag，上传到我在hub.docker.com上的账号下，以elasticsearch:v2.4.1-1为例：

# docker pull  gcr.io/google_containers/elasticsearch:v2.4.1-1
# docker tag gcr.io/google_containers/elasticsearch:v2.4.1-1 bigwhite/elasticsearch:v2.4.1-1
# docker push bigwhite/elasticsearch:v2.4.1-1

下面是我们在后续安装过程中真正要使用到的镜像：

bigwhite/fluentd-elasticsearch:1.22
bigwhite/elasticsearch:v2.4.1-1
bigwhite/kibana:v4.6.1-1

三、启动fluentd

fluentd是以DaemonSet的形式跑在K8s集群上的，这样k8s可以保证每个k8s cluster node上都会启动一个fluentd(注意：将image改为上述镜像地址，如果你配置了加速器，那自然就不必了)。

# kubectl create -f fluentd-es-ds.yaml --record
daemonset "fluentd-es-v1.22" created

查看daemonset中的Pod的启动情况，我们发现：

kube-system                  fluentd-es-v1.22-as3s5                  0/1       CrashLoopBackOff   2          43s       172.16.99.6    10.47.136.60
kube-system                  fluentd-es-v1.22-qz193                  0/1       CrashLoopBackOff   2          43s       172.16.57.7    10.46.181.146

fluentd Pod启动失败，fluentd的日志可以通过/var/log/fluentd.log查看：

# tail -100f /var/log/fluentd.log

2017-03-02 02:27:01 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2017-03-02 02:27:01 +0000 [info]: starting fluentd-0.12.31
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-docker_metadata_filter' version '0.1.3'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.5.0'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-kafka' version '0.4.1'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '0.24.0'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.16'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-s3' version '0.8.0'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-td' version '0.10.29'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2017-03-02 02:27:01 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2017-03-02 02:27:01 +0000 [info]: gem 'fluentd' version '0.12.31'
2017-03-02 02:27:01 +0000 [info]: adding match pattern="fluent.**" type="null"
2017-03-02 02:27:01 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2017-03-02 02:27:02 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error="Invalid Kubernetes API v1 endpoint https://192.168.3.1:443/api: 401 Unauthorized"
2017-03-02 02:27:02 +0000 [info]: process finished code=256
2017-03-02 02:27:02 +0000 [warn]: process died within 1 second. exit.

从上述日志中的error来看：fluentd访问apiserver secure port(443)出错了：Unauthorized! 通过分析 cluster/addons/fluentd-elasticsearch/fluentd-es-image/build.sh和td-agent.conf，我们发现是fluentd image中的fluent-plugin-kubernetes_metadata_filter要去访问API Server以获取一些kubernetes的metadata信息。不过未做任何特殊配置的fluent-plugin-kubernetes_metadata_filter，我猜测它使用的是kubernetes为Pod传入的环境变量：KUBERNETES_SERVICE_HOST和KUBERNETES_SERVICE_PORT来得到API Server的访问信息的。但API Server在secure port上是开启了安全身份验证机制的，fluentd直接访问必然是失败的。

我们找到了fluent-plugin-kubernetes_metadata_filter项目在github.com上的主页，在这个页面上我们看到了fluent-plugin-kubernetes_metadata_filter支持的其他配置，包括：ca_file、client_cert、client_key等，显然这些字眼非常眼熟。我们需要修改一下fluentd image中td-agent.conf的配置，为fluent-plugin-kubernetes_metadata_filter增加一些配置项，比如：

// td-agent.conf
... ...
<filter kubernetes.**>
  type kubernetes_metadata
  ca_file /srv/kubernetes/ca.crt
  client_cert /srv/kubernetes/kubecfg.crt
  client_key /srv/kubernetes/kubecfg.key
</filter>
... ...

这里我不想重新制作image，那么怎么办呢？Kubernetes提供了ConfigMap这一强大的武器，我们可以将新版td-agent.conf制作成kubernetes的configmap资源，并挂载到fluentd pod的相应位置以替换image中默认的td-agent.conf。

需要注意两点：
* 在基于td-agent.conf创建configmap资源之前，需要将td-agent.conf中的注释行都删掉，否则生成的configmap的内容可能不正确；
* fluentd pod将创建在kube-system下，因此configmap资源也需要创建在kube-system namespace下面，否则kubectl create无法找到对应的configmap。

# kubectl create configmap td-agent-config --from-file=./td-agent.conf -n kube-system
configmap "td-agent-config" created

# kubectl get configmaps -n kube-system
NAME              DATA      AGE
td-agent-config   1         9s

# kubectl get configmaps td-agent-config -o yaml
apiVersion: v1
data:
  td-agent.conf: |
    <match fluent.**>
      type null
    </match>

    <source>
      type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      time_format %Y-%m-%dT%H:%M:%S.%NZ
      tag kubernetes.*
      format json
      read_from_head true
    </source>
... ...

fluentd-es-ds.yaml也要随之做一些改动，主要是增加两个mount: 一个是mount 上面的configmap td-agent-config，另外一个就是mount hostpath：/srv/kubernetes以获取到相关client端的数字证书：

  spec:
      containers:
      - name: fluentd-es
        image: bigwhite/fluentd-elasticsearch:1.22
        command:
          - '/bin/sh'
          - '-c'
          - '/usr/sbin/td-agent 2>&1 >> /var/log/fluentd.log'
        resources:
          limits:
            memory: 200Mi
          #requests:
            #cpu: 100m
            #memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: td-agent-config
          mountPath: /etc/td-agent
        - name: tls-files
          mountPath: /srv/kubernetes
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: td-agent-config
        configMap:
          name: td-agent-config
      - name: tls-files
        hostPath:
          path: /srv/kubernetes

接下来，我们重新创建fluentd ds，步骤不赘述。这回我们的创建成功了：

kube-system                  fluentd-es-v1.22-adsrx                  1/1       Running    0          1s        172.16.99.6    10.47.136.60
kube-system                  fluentd-es-v1.22-rpme3                  1/1       Running    0          1s        172.16.57.7    10.46.181.146

但通过查看/var/log/fluentd.log，我们依然能看到“问题”：

2017-03-02 03:57:58 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-03-02 03:57:59 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"elasticsearch-logging\", :port=>9200, :scheme=>\"http\"})!" plugin_id="object:3fd99fa857d8"
  2017-03-02 03:57:58 +0000 [warn]: suppressed same stacktrace
2017-03-02 03:58:00 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2017-03-02 03:58:03 +0000 error_class="Fluent::ElasticsearchOutput::ConnectionFailure" error="Can not reach Elasticsearch cluster ({:host=>\"elasticsearch-logging\", :port=>9200, :scheme=>\"http\"})!" plugin_id="object:3fd99fa857d8"
2017-03-02 03:58:00 +0000 [info]: process finished code=9
2017-03-02 03:58:00 +0000 [error]: fluentd main process died unexpectedly. restarting.

由于ElasticSearch logging还未创建，这是连不上elasticsearch所致。

四、启动elasticsearch

启动elasticsearch：

# kubectl create -f es-controller.yaml
replicationcontroller "elasticsearch-logging-v1" created

# kubectl create -f es-service.yaml
service "elasticsearch-logging" created

get pods：

kube-system                  elasticsearch-logging-v1-3bzt6          1/1       Running    0          7s        172.16.57.8    10.46.181.146
kube-system                  elasticsearch-logging-v1-nvbe1          1/1       Running    0          7s        172.16.99.10   10.47.136.60

elastic search logging启动成功后，上述fluentd的fail日志就没有了！

不过elastic search真的运行ok了么？我们查看一下elasticsearch相关Pod日志：

# kubectl logs -f elasticsearch-logging-v1-3bzt6 -n kube-system
F0302 03:59:41.036697       8 elasticsearch_logging_discovery.go:60] kube-system namespace doesn't exist: the server has asked for the client to provide credentials (get namespaces kube-system)
goroutine 1 [running]:
k8s.io/kubernetes/vendor/github.com/golang/glog.stacks(0x19a8100, 0xc400000000, 0xc2, 0x186)
... ...
main.main()
    elasticsearch_logging_discovery.go:60 +0xb53

[2017-03-02 03:59:42,587][INFO ][node                     ] [elasticsearch-logging-v1-3bzt6] version[2.4.1], pid[16], build[c67dc32/2016-09-27T18:57:55Z]
[2017-03-02 03:59:42,588][INFO ][node                     ] [elasticsearch-logging-v1-3bzt6] initializing ...
[2017-03-02 03:59:44,396][INFO ][plugins                  ] [elasticsearch-logging-v1-3bzt6] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
... ...
[2017-03-02 03:59:44,441][INFO ][env                      ] [elasticsearch-logging-v1-3bzt6] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-03-02 03:59:48,355][INFO ][node                     ] [elasticsearch-logging-v1-3bzt6] initialized
[2017-03-02 03:59:48,355][INFO ][node                     ] [elasticsearch-logging-v1-3bzt6] starting ...
[2017-03-02 03:59:48,507][INFO ][transport                ] [elasticsearch-logging-v1-3bzt6] publish_address {172.16.57.8:9300}, bound_addresses {[::]:9300}
[2017-03-02 03:59:48,547][INFO ][discovery                ] [elasticsearch-logging-v1-3bzt6] kubernetes-logging/7_f_M2TKRZWOw4NhBc4EqA
[2017-03-02 04:00:18,552][WARN ][discovery                ] [elasticsearch-logging-v1-3bzt6] waited for 30s and no initial state was set by the discovery
[2017-03-02 04:00:18,562][INFO ][http                     ] [elasticsearch-logging-v1-3bzt6] publish_address {172.16.57.8:9200}, bound_addresses {[::]:9200}
[2017-03-02 04:00:18,562][INFO ][node                     ] [elasticsearch-logging-v1-3bzt6] started
[2017-03-02 04:01:15,754][WARN ][discovery.zen.ping.unicast] [elasticsearch-logging-v1-3bzt6] failed to send ping to [{#zen_unicast_1#}{127.0.0.1}{127.0.0.1:9300}]
SendRequestTransportException[[][127.0.0.1:9300][internal:discovery/zen/unicast]]; nested: NodeNotConnectedException[[][127.0.0.1:9300] Node not connected];
... ...
Caused by: NodeNotConnectedException[[][127.0.0.1:9300] Node not connected]
    at org.elasticsearch.transport.netty.NettyTransport.nodeChannel(NettyTransport.java:1141)
    at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java:830)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:329)
    ... 12 more

总结了一下，日志中有两个错误：
- 无法访问到API Server，这个似乎和fluentd最初的问题一样；
- elasticsearch两个节点间互ping失败。

要想找到这两个问题的原因，还得回到源头，去分析elastic search image的组成。

通过cluster/addons/fluentd-elasticsearch/es-image/run.sh文件内容：

/elasticsearch_logging_discovery >> /elasticsearch/config/elasticsearch.yml

chown -R elasticsearch:elasticsearch /data

/bin/su -c /elasticsearch/bin/elasticsearch elasticsearch

我们了解到image中，其实包含了两个程序，一个为/elasticsearch_logging_discovery，该程序执行后生成一个配置文件： /elasticsearch/config/elasticsearch.yml。该配置文件后续被另外一个程序：/elasticsearch/bin/elasticsearch使用。

我们查看一下已经运行的docker中的elasticsearch.yml文件内容：

# docker exec 3cad31f6eb08 cat /elasticsearch/config/elasticsearch.yml
cluster.name: kubernetes-logging

node.name: ${NODE_NAME}
node.master: ${NODE_MASTER}
node.data: ${NODE_DATA}

transport.tcp.port: ${TRANSPORT_PORT}
http.port: ${HTTP_PORT}

path.data: /data

network.host: 0.0.0.0

discovery.zen.minimum_master_nodes: ${MINIMUM_MASTER_NODES}
discovery.zen.ping.multicast.enabled: false

这个结果中缺少了一项：

discovery.zen.ping.unicast.hosts: ["172.30.0.11", "172.30.192.15"]

这也是导致第二个问题的原因。综上，elasticsearch logging的错误其实都是由于/elasticsearch_logging_discovery无法访问API Server导致 /elasticsearch/config/elasticsearch.yml没有被正确生成造成的，我们就来解决这个问题。

我查看了一下/elasticsearch_logging_discovery的源码，elasticsearch_logging_discovery是一个典型通过client-go通过service account访问API Server的程序，很显然这就是我在《在Kubernetes Pod中使用Service Account访问API Server》一文中提到的那个问题：默认的service account不好用。

解决方法：在kube-system namespace下创建一个新的service account资源，并在es-controller.yaml中显式使用该新创建的service account。

创建一个新的serviceaccount在kube-system namespace下：

//serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-efk

# kubectl create -f serviceaccount.yaml -n kube-system
serviceaccount "k8s-efk" created

# kubectl get serviceaccount -n kube-system
NAME      SECRETS   AGE
default   1         139d
k8s-efk   1         17s

在es-controller.yaml中，使用service account “k8s-efk”：

//es-controller.yaml
... ...
spec:
  replicas: 2
  selector:
    k8s-app: elasticsearch-logging
    version: v1
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccount: k8s-efk
      containers:
... ...

重新创建elasticsearch logging service后，我们再来查看elasticsearch-logging pod的日志：

# kubectl logs -f elasticsearch-logging-v1-dklui -n kube-system
[2017-03-02 08:26:46,500][INFO ][node                     ] [elasticsearch-logging-v1-dklui] version[2.4.1], pid[14], build[c67dc32/2016-09-27T18:57:55Z]
[2017-03-02 08:26:46,504][INFO ][node                     ] [elasticsearch-logging-v1-dklui] initializing ...
[2017-03-02 08:26:47,984][INFO ][plugins                  ] [elasticsearch-logging-v1-dklui] modules [reindex, lang-expression, lang-groovy], plugins [], sites []
[2017-03-02 08:26:48,073][INFO ][env                      ] [elasticsearch-logging-v1-dklui] using [1] data paths, mounts [[/data (/dev/vda1)]], net usable_space [16.9gb], net total_space [39.2gb], spins? [possibly], types [ext4]
[2017-03-02 08:26:48,073][INFO ][env                      ] [elasticsearch-logging-v1-dklui] heap size [1007.3mb], compressed ordinary object pointers [true]
[2017-03-02 08:26:53,241][INFO ][node                     ] [elasticsearch-logging-v1-dklui] initialized
[2017-03-02 08:26:53,241][INFO ][node                     ] [elasticsearch-logging-v1-dklui] starting ...
[2017-03-02 08:26:53,593][INFO ][transport                ] [elasticsearch-logging-v1-dklui] publish_address {172.16.57.8:9300}, bound_addresses {[::]:9300}
[2017-03-02 08:26:53,651][INFO ][discovery                ] [elasticsearch-logging-v1-dklui] kubernetes-logging/Ky_OuYqMRkm_918aHRtuLg
[2017-03-02 08:26:56,736][INFO ][cluster.service          ] [elasticsearch-logging-v1-dklui] new_master {elasticsearch-logging-v1-dklui}{Ky_OuYqMRkm_918aHRtuLg}{172.16.57.8}{172.16.57.8:9300}{master=true}, added {{elasticsearch-logging-v1-vjxm3}{cbzgrfZATyWkHfQYHZhs7Q}{172.16.99.10}{172.16.99.10:9300}{master=true},}, reason: zen-disco-join(elected_as_master, [1] joins received)
[2017-03-02 08:26:56,955][INFO ][http                     ] [elasticsearch-logging-v1-dklui] publish_address {172.16.57.8:9200}, bound_addresses {[::]:9200}
[2017-03-02 08:26:56,956][INFO ][node                     ] [elasticsearch-logging-v1-dklui] started
[2017-03-02 08:26:57,157][INFO ][gateway                  ] [elasticsearch-logging-v1-dklui] recovered [0] indices into cluster_state
[2017-03-02 08:27:05,378][INFO ][cluster.metadata         ] [elasticsearch-logging-v1-dklui] [logstash-2017.03.02] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2017-03-02 08:27:06,360][INFO ][cluster.metadata         ] [elasticsearch-logging-v1-dklui] [logstash-2017.03.01] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[2017-03-02 08:27:07,163][INFO ][cluster.routing.allocation] [elasticsearch-logging-v1-dklui] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash-2017.03.01][3], [logstash-2017.03.01][3]] ...]).
[2017-03-02 08:27:07,354][INFO ][cluster.metadata         ] [elasticsearch-logging-v1-dklui] [logstash-2017.03.02] create_mapping [fluentd]
[2017-03-02 08:27:07,988][INFO ][cluster.metadata         ] [elasticsearch-logging-v1-dklui] [logstash-2017.03.01] create_mapping [fluentd]
[2017-03-02 08:27:09,578][INFO ][cluster.routing.allocation] [elasticsearch-logging-v1-dklui] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[logstash-2017.03.02][4]] ...]).

elasticsearch logging启动运行ok！

五、启动kibana

有了elasticsearch logging的“前车之鉴”，这次我们也把上面新创建的serviceaccount：k8s-efk显式赋值给kibana-controller.yaml:

//kibana-controller.yaml
... ...
spec:
      serviceAccount: k8s-efk
      containers:
      - name: kibana-logging
        image: bigwhite/kibana:v4.6.1-1
        resources:
          # keep request = limit to keep this container in guaranteed class
          limits:
            cpu: 100m
          #requests:
          #  cpu: 100m
        env:
          - name: "ELASTICSEARCH_URL"
            value: "http://elasticsearch-logging:9200"
          - name: "KIBANA_BASE_URL"
            value: "/api/v1/proxy/namespaces/kube-system/services/kibana-logging"
        ports:
        - containerPort: 5601
          name: ui
          protocol: TCP
... ...

启动kibana，并观察pod日志：

# kubectl create -f kibana-controller.yaml
# kubectl create -f kibana-service.yaml
# kubectl logs -f kibana-logging-3604961973-jby53 -n kube-system
ELASTICSEARCH_URL=http://elasticsearch-logging:9200
server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-logging
{"type":"log","@timestamp":"2017-03-02T08:30:15Z","tags":["info","optimize"],"pid":6,"message":"Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"}

kibana缓存着实需要一段时间，请耐心等待！可能是几分钟。之后你将会看到如下日志：

# kubectl logs -f kibana-logging-3604961973-jby53 -n kube-system
ELASTICSEARCH_URL=http://elasticsearch-logging:9200
server.basePath: /api/v1/proxy/namespaces/kube-system/services/kibana-logging
{"type":"log","@timestamp":"2017-03-02T08:30:15Z","tags":["info","optimize"],"pid":6,"message":"Optimizing and caching bundles for kibana and statusPage. This may take a few minutes"}
{"type":"log","@timestamp":"2017-03-02T08:40:04Z","tags":["info","optimize"],"pid":6,"message":"Optimization of bundles for kibana and statusPage complete in 588.60 seconds"}
{"type":"log","@timestamp":"2017-03-02T08:40:04Z","tags":["status","plugin:kibana@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:05Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":6,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:05Z","tags":["status","plugin:kbn_vislib_vis_types@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:05Z","tags":["status","plugin:markdown_vis@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:05Z","tags":["status","plugin:metric_vis@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:06Z","tags":["status","plugin:spyModes@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:06Z","tags":["status","plugin:statusPage@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:06Z","tags":["status","plugin:table_vis@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2017-03-02T08:40:06Z","tags":["listening","info"],"pid":6,"message":"Server running at http://0.0.0.0:5601"}
{"type":"log","@timestamp":"2017-03-02T08:40:11Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":6,"state":"yellow","message":"Status changed from yellow to yellow - No existing Kibana index found","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}
{"type":"log","@timestamp":"2017-03-02T08:40:14Z","tags":["status","plugin:elasticsearch@1.0.0","info"],"pid":6,"state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"No existing Kibana index found"}

接下来，通过浏览器访问下面地址就可以访问kibana的web页面了，注意：Kinaba的web页面加载也需要一段时间。

https://{API Server external IP}:{API Server secure port}/api/v1/proxy/namespaces/kube-system/services/kibana-logging/app/kibana#/settings/indices/

下面是创建一个index（相当于mysql中的一个database）页面：

取消“Index contains time-based events”，然后点击“Create”即可创建一个Index。

点击页面上的”Setting” -> “Status”，可以查看当前elasticsearch logging的整体状态，如果一切ok，你将会看到下图这样的页面：

创建Index后，可以在Discover下看到ElasticSearch logging中汇聚的日志：

六、小结

以上就是在Kubernetes 1.3.7集群上安装Fluentd和ElasticSearch stack，实现kubernetes cluster level logging的过程。在使用kubeadm安装的Kubernetes 1.5.1环境下安装这些，则基本不会遇到上述这些问题。

另外ElasticSearch logging默认挂载的volume是emptyDir，实验用可以。但要部署在生产环境，必须换成Persistent Volume，比如：CephRBD。

Kubernetes API Server是整个Kubernetes集群的核心，我们不仅有从集群外部访问API Server的需求，有时，我们还需要从Pod的内部访问API Server。

然而，在生产环境中，Kubernetes API Server都是“设防”的。在《Kubernetes集群的安全配置》一文中，我提到过：Kubernetes通过client cert、static token、basic auth等方法对客户端请求进行身份验证。对于运行于Pod中的Process而言，有些时候这些方法是适合的，但有些时候，像client cert、static token或basic auth这些信息是不便于暴露给Pod中的Process的。并且通过这些方法通过API Server验证后的请求是具有全部授权的，可以任意操作Kubernetes cluster，这显然是不能满足安全要求的。为此，Kubernetes更推荐大家使用service account这种方案的。本文就带大家详细说说如何通过service account从一个Pod中访问API Server的。

零、试验环境

本文的试验环境是Kubernetes 1.3.7 cluster，双节点，master承载负荷。cluster通过kube-up.sh搭建的，具体的搭建方法见《一篇文章带你了解Kubernetes安装》。

一、什么是service account？

什么是service account? 顾名思义，相对于user account（比如：kubectl访问APIServer时用的就是user account），service account就是Pod中的Process用于访问Kubernetes API的account，它为Pod中的Process提供了一种身份标识。相比于user account的全局性权限，service account更适合一些轻量级的task，更聚焦于授权给某些特定Pod中的Process所使用。

service account作为一种resource存在于Kubernetes cluster中，我们可以通过kubectl获取当前cluster中的service acount列表：

# kubectl get serviceaccount --all-namespaces
NAMESPACE                    NAME           SECRETS   AGE
default                      default        1         140d
kube-system                  default        1         140d

我们查看一下kube-system namespace下名为”default”的service account的详细信息：

# kubectl describe serviceaccount/default -n kube-system
Name:        default
Namespace:    kube-system
Labels:        <none>

Image pull secrets:    <none>

Mountable secrets:     default-token-hpni0

Tokens:                default-token-hpni0

我们看到service account并不复杂，只是关联了一个secret资源作为token，该token也叫service-account-token，该token才是真正在API Server验证(authentication)环节起作用的：

# kubectl get secret  -n kube-system
NAME                  TYPE                                  DATA      AGE
default-token-hpni0   kubernetes.io/service-account-token   3         140d

# kubectl get secret default-token-hpni0 -o yaml -n kube-system
apiVersion: v1
data:
  ca.crt: {base64 encoding of ca.crt data}
  namespace: a3ViZS1zeXN0ZW0=
  token: {base64 encoding of bearer token}

kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: default
    kubernetes.io/service-account.uid: 90ded7ff-9120-11e6-a0a6-00163e1625a9
  creationTimestamp: 2016-10-13T08:39:33Z
  name: default-token-hpni0
  namespace: kube-system
  resourceVersion: "2864"
  selfLink: /api/v1/namespaces/kube-system/secrets/default-token-hpni0
  uid: 90e71909-9120-11e6-a0a6-00163e1625a9
type: kubernetes.io/service-account-token

我们看到这个类型为service-account-token的secret资源包含的数据有三部分：ca.crt、namespace和token。

• ca.crt
  这个是API Server的CA公钥证书，用于Pod中的Process对API Server的服务端数字证书进行校验时使用的；

• namespace
  这个就是Secret所在namespace的值的base64编码：# echo -n “kube-system”|base64 => “a3ViZS1zeXN0ZW0=”

• token

这是一段用API Server私钥签发(sign)的bearer tokens的base64编码，在API Server authenticating环节，它将派上用场。

二、API Server的service account authentication(身份验证)

前面说过，service account为Pod中的Process提供了一种身份标识，在Kubernetes的身份校验(authenticating)环节，以某个service account提供身份的Pod的用户名为：

system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT)

以上面那个kube-system namespace下的“default” service account为例，使用它的Pod的username全称为：

system:serviceaccount:kube-system:default

有了username，那么credentials呢？就是上面提到的service-account-token中的token。在《Kubernetes集群的安全配置》一文中我们谈到过，API Server的authenticating环节支持多种身份校验方式：client cert、bearer token、static password auth等，这些方式中有一种方式通过authenticating（Kubernetes API Server会逐个方式尝试），那么身份校验就会通过。一旦API Server发现client发起的request使用的是service account token的方式，API Server就会自动采用signed bearer token方式进行身份校验。而request就会使用携带的service account token参与验证。该token是API Server在创建service account时用API server启动参数：–service-account-key-file的值签署(sign)生成的。如果–service-account-key-file未传入任何值，那么将默认使用–tls-private-key-file的值，即API Server的私钥（server.key）。

通过authenticating后，API Server将根据Pod username所在的group：system:serviceaccounts和system:serviceaccounts:(NAMESPACE)的权限对其进行authority 和admission control两个环节的处理。在这两个环节中，cluster管理员可以对service account的权限进行细化设置。

三、默认的service account

Kubernetes会为每个cluster中的namespace自动创建一个默认的service account资源，并命名为”default”：

# kubectl get serviceaccount --all-namespaces
NAMESPACE                    NAME           SECRETS   AGE
default                      default        1         140d
kube-system                  default        1         140d

如果Pod中没有显式指定spec.serviceAccount字段值，那么Kubernetes会将该namespace下的”default” service account自动mount到在这个namespace中创建的Pod里。我们以namespace “default”为例，我们查看一下其中的一个Pod的信息：

# kubectl describe pod/index-api-2822468404-4oofr
Name:        index-api-2822468404-4oofr
Namespace:    default
... ...

Containers:
  index-api:
   ... ...
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-40z0x (ro)
    Environment Variables:    <none>
... ...
Volumes:
... ...
  default-token-40z0x:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-40z0x

QoS Class:    BestEffort
Tolerations:    <none>
No events.

可以看到，kubernetes将default namespace中的service account “default”的service account token挂载(mount)到了Pod中容器的/var/run/secrets/kubernetes.io/serviceaccount路径下。

深入容器内部，查看mount的serviceaccount路径下的结构：

# docker exec 3d11ee06e0f8 ls  /var/run/secrets/kubernetes.io/serviceaccount
ca.crt
namespace
token

这三个文件与上面提到的service account的token中的数据是一一对应的。

四、default service account doesn’t work

上面提到过，每个Pod都会被自动挂载一个其所在namespace的default service account，该service account用于该Pod中的Process访问API Server时使用。Pod中的Process该怎么用这个service account呢？Kubernetes官方提供了一个client-go项目可以为你演示如何使用service account访问API Server。这里我们就基于client-go项目中的examples/in-cluster/main.go来测试一下是否能成功访问API Server。

先下载client-go源码：

# go get k8s.io/client-go

# ls -F
CHANGELOG.md  dynamic/   Godeps/     INSTALL.md   LICENSE   OWNERS  plugin/    rest/     third_party/  transport/  vendor/
discovery/    examples/  informers/  kubernetes/  listers/  pkg/    README.md  testing/  tools/        util/

我们改造一下examples/in-cluster/main.go，考虑到panic会导致不便于观察Pod日志，我们将panic改为输出到“标准输出”，并且不return，让Pod周期性的输出相关日志，即便fail：

// k8s.io/client-go/examples/in-cluster/main.go
... ...
func main() {
    // creates the in-cluster config
    config, err := rest.InClusterConfig()
    if err != nil {
        fmt.Println(err)
    }
    // creates the clientset
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        fmt.Println(err)
    }
    for {
        pods, err := clientset.CoreV1().Pods("").List(metav1.ListOptions{})
        if err != nil {
            fmt.Println(err)
        } else {
            fmt.Printf("There are %d pods in the cluster\n", len(pods.Items))
        }
        time.Sleep(10 * time.Second)
    }
}

基于该main.go的go build默认输出，创建一个简单的Dockerfile：

From ubuntu:14.04
MAINTAINER Tony Bai <bigwhite.cn@gmail.com>

COPY main /root/main
RUN chmod +x /root/main
WORKDIR /root
ENTRYPOINT ["/root/main"]

构建一个测试用docker image：

# docker build -t k8s/example1:latest .
... ...

# docker images|grep k8s
k8s/example1                                                  latest              ceb3efdb2f91        14 hours ago        264.4 MB

创建一份deployment manifest：

//main.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-example1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: k8s-example1
    spec:
      containers:
      - name: k8s-example1
        image: k8s/example1:latest
        imagePullPolicy: IfNotPresent

我们来创建该deployment（kubectl create -f main.yaml -n kube-system），观察Pod中的main程序能否成功访问到API Server：

# kubectl logs k8s-example1-1569038391-jfxhx
the server has asked for the client to provide credentials (get pods)
the server has asked for the client to provide credentials (get pods)

API Server log(/var/log/upstart/kube-apiserver.log):

E0302 15:45:40.944496   12902 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
E0302 15:45:50.946598   12902 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error
E0302 15:46:00.948398   12902 handlers.go:54] Unable to authenticate the request due to an error: crypto/rsa: verification error

出错了！kube-system namespace下的”default” service account似乎不好用啊！（注意：这是在kubernetes 1.3.7环境）。

五、创建一个新的自用的service account

在kubernetes github issues中，有好多issue是关于”default” service account不好用的问题，给出的解决方法似乎都是创建一个新的service account。

service account的创建非常简单，我们创建一个serviceaccount.yaml：

//serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: k8s-example1

创建该service account：

# kubectl create -f serviceaccount.yaml
serviceaccount "k8s-example1" created

# kubectl get serviceaccount
NAME           SECRETS   AGE
default        1         139d
k8s-example1   1         12s

修改main.yaml，让Pod显示使用这个新的service account：

//main.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: k8s-example1
spec:
  replicas: 1
  template:
    metadata:
      labels:
        run: k8s-example1
    spec:
      serviceAccount: k8s-example1
      containers:
      - name: k8s-example1
        image: k8s/example1:latest
        imagePullPolicy: IfNotPresent

好了，我们重新创建该deployment，查看Pod日志：

# kubectl logs k8s-example1-456041623-rqj87
There are 14 pods in the cluster
There are 14 pods in the cluster
... ...

我们看到main程序使用新的service account成功通过了API Server的身份验证环节，并获得了cluster的相关信息。

六、尾声

在我的另外一个使用kubeadm安装的k8s 1.5.1环境中，我重复做了上面这个简单测试，不同的是这次我直接使用了default service account。在k8s 1.5.1下，pod的执行结果是ok的，也就是说通过default serviceaccount，我们的client-go in-cluster example程序可以顺利通过API Server的身份验证，获取到相关的Pods元信息。

七、参考资料

• Kubernetes authentication
• Service Accounts
• Accessing the cluster
• Service Accounts Admin

微博：@tonybai_cn
微信公众号：iamtonybai
github.com: https://github.com/bigwhite