标签 docker 下的文章

使用kubectl访问Kubernetes集群时的身份验证和授权

kubectl是日常访问和管理Kubernetes集群最为常用的工具。

当我们使用kubeadm成功引导启动(init)一个Kubernetes集群的控制平面后,kubeadm会在init的输出结果中给予我们下面这样的“指示”:

... ...
Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config
... ...

kubeadm init在结尾处输出的这些信息是在告知我们如何配置kubeconfig文件。按照上述命令配置后,master节点上的kubectl就可以直接使用$HOME/.kube/config的信息访问k8s cluster了。并且,通过这种配置方式,kubectl也拥有了整个集群的管理员(root)权限

很多K8s初学者在这里都会有疑问:当kubectl使用这种kubeconfig方式访问集群时,Kubernetes的kube-apiserver是如何对来自kubectl的访问进行身份验证(authentication)和授权(authorization)的呢?为什么来自kubectl的请求拥有最高的管理员权限呢?在本文中,我们就来分析说明一下这个过程。

一. Kubernetes API的访问控制原理回顾

《Kubernetes的安全设置》一文中我曾介绍过Kubernetes集群的访问权限控制由kube-apiserver负责,kube-apiserver的访问权限控制由身份验证(authentication)、授权(authorization)和准入控制(admission control)三步骤组成,这三步骤是按序进行的:

img{512x368}

要想搞明白kubectl访问Kubernetes集群时的身份验证和授权,就是要弄清kube-apiserver在进行身份验证和授权两个环节都做了什么:

  • Authentication:即身份验证,这个环节它面对的输入是整个http request,它负责对来自client的请求进行身份校验,支持的方法包括:client证书验证(https双向验证)、basic auth、普通token以及jwt token(用于serviceaccount)。APIServer启动时,可以指定一种Authentication方法,也可以指定多种方法。如果指定了多种方法,那么APIServer将会逐个使用这些方法对客户端请求进行验证,只要请求数据通过其中一种方法的验证,APIServer就会认为Authentication成功;在较新版本kubeadm引导启动的k8s集群的apiserver初始配置中,默认支持client证书验证和serviceaccount两种身份验证方式。在这个环节,apiserver会通过client证书或http header中的字段(比如serviceaccount的jwt token)来识别出请求的“用户身份”,包括”user”、”group”等,这些信息将在后面的authorization环节用到。

  • Authorization:授权。这个环节面对的输入是http request context中的各种属性,包括:user、group、request path(比如:/api/v1、/healthz、/version等)、request verb(比如:get、list、create等)。APIServer会将这些属性值与事先配置好的访问策略(access policy)相比较。APIServer支持多种authorization mode,包括NodeRBAC、Webhook等。APIServer启动时,可以指定一种authorization mode,也可以指定多种authorization mode,如果是后者,只要Request通过了其中一种mode的授权,那么该环节的最终结果就是授权成功。在较新版本kubeadm引导启动的k8s集群的apiserver初始配置中,authorization-mode的默认配置是”Node,RBAC”。Node授权器主要用于各个node上的kubelet访问apiserver时使用的,其他一般均由RBAC授权器来授权。

RBAC,Role-Based Access Control即Role-Based Access Control,它使用”rbac.authorization.k8s.io”实现授权决策,允许管理员通过Kubernetes API动态配置策略。在RBAC API中,一个角色(Role)包含了一组权限规则。Role有两种:Role和ClusterRole。一个Role对象只能用于授予对某一单一命名空间(namespace)中资源的访问权限。ClusterRole对象可以授予与Role对象相同的权限,但由于它们属于集群范围对象, 也可以使用它们授予对以下几种资源的访问权限:

  • 集群范围资源(例如节点,即node)
  • 非资源类型endpoint(例如”/healthz”)
  • 跨所有命名空间的命名空间范围资源(例如所有命名空间下的pod资源)

rolebinding,角色绑定则是定义了将一个角色的各种权限授予一个或者一组用户。 角色绑定包含了一组相关主体(即subject, 包括用户——User、用户组——Group、或者服务账户——Service Account)以及对被授予角色的引用。 在命名空间中可以通过RoleBinding对象进行用户授权,而集群范围的用户授权则可以通过ClusterRoleBinding对象完成。

好了,有了上面这些知识基础,要搞清楚kubectl访问集群的身份验证和授权过程,我们只需要逐一解决下面的一些问题即可:

1、authencation中识别出了哪些http request context中的信息?
2、authorization中RBAC authorizer找到的对应的rolebinding或clusterrolebinding是什么?
3、对应的role或clusterrole的权限规则?

二. 在身份验证(authentication)识别出Group

我们先从kubectl使用的kubeconfig入手。kubectl使用的kubeconfig文件实质上就是kubeadm init过程中生成的/etc/kubernetes/admin.conf,我们查看一下该kubeconfig文件的内容:

环境k8s 1.10.3:

# kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://172.16.66.101:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

关于kubeconfig文件的解释,可以在这里自行脑补。在这些输出信息中,我们着重提取到两个信息:

user name: kubernetes-admin
client-certificate-date: XXXX

前面提到过apiserver的authentication支持通过tls client certificate、basic auth、token等方式对客户端发起的请求进行身份校验,从kubeconfig信息来看,kubectl显然在请求中使用了tls client certificate的方式,即客户端的证书。另外我们知道Kubernetes是没有user这种资源的,通过k8s API也无法创建user。那么kubectl的身份信息就应该“隐藏”在client-certificate的数据中,我们来查看一下。

首先我们将 /etc/kubernetes/admin.conf中client-certificate-data的数据内容保存到一个临时文件admin-client-certificate.txt中:

// admin-client-certificate.txt

LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM4akNDQWRxZ0F3SUJBZ0lJZjJkVlJqbThFTFF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB4T0RBMU1UUXdPREUzTVROYUZ3MHhPVEExTVRRd09ERTNNVGRhTURReApGekFWQmdOVkJBb1REbk41YzNSbGJUcHRZWE4wWlhKek1Sa3dGd1lEVlFRREV4QnJkV0psY201bGRHVnpMV0ZrCmJXbHVNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXhCbjNqZHc4MGIxR2ZiNnMKdzJOcnFwTG90TVQ0bnlBZjJIaHFNclhqbk8rd25hSzFBSVRPdy8yMm1EajByd0l1SndkUUlqNS9CYUY2M3BQRQoxcFUwdmhJUFZLNG42Skk0ZG1Nem8vbFIzalpwR2VaVzF6ZFhhQ292dzljN2NsYmlIby9tRkc0eHF5dFZMZlg0Ci9TOG1GcDJBOVFjaWVKR0lvNVMwQlIzRlpsVTFQTTdEUmJMRFZWcTFQZHlOWTJHZnNiR3JIbEdnWHZXQUtDZC8KSDc5Z0FxVm9UWGpTSVdDVll1WWNvTHZkdlZYUVNJaVlscFhGUDFqQlFMdmNVN3ZycXRiMTJSbXJ4bnBrVzRwbApkR0VPWDJzTG1mWVo1VGlGcGtSd3oyR3hzbVd5UmJ0Nk91SVNKRkk2UlowcitSbjR5TURLUHJZbEVuZ0RWYzVLClBaNXptd0lEQVFBQm95Y3dKVEFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUgKQXdJd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFWk5UdlR6Mk9nekNVZHZNRmJyaFBzcCttRDJ2UGpNUkN4aQozQmtBMTB2SUNPU2ZkeW1NbjhhdzBJYktZejJnUWJYcVVmcXpRbVFmYTNpZitRWUJrQis3N3pmc3Y5YW00RVAvCmU2VGc1MnRxVjJQN3MyZUY3dE5BZTIwR3lWNnlGbFExUVVXNS9NNE0rSk1sVitCVWJsOXlFeVFsRU51Y0tmK3UKVFB5S0tUVXR6dlVZcjVFM0VKa3Q4NEVRSU52dzJuUjJqTnZlWjFYV09saVVyS2ZqSEh0ZnZPL241NlVTdUk0dwp1MkxUbElDUmNqNGcrWldsSWplTUZrR3lQYkp5SkFRNjVQMnNHclptMWtsR0dIM216d081Q1AxeVpXdm9VampQCmp6U2pNQ0lhSy9mUjhlUkFKNnExdFQ2YkcyNkwrbmprS0NRRFdLcGpBV09hcHVST2Niaz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=

然后针对该文件数据做base64解码,得到client certificate文件:

cat admin-client-certificate.txt | base64 -d > admin-client.crt

# cat admin-client.crt

-----BEGIN CERTIFICATE-----
MIIC8jCCAdqgAwIBAgIIf2dVRjm8ELQwDQYJKoZIhvcNAQELBQAwFTETMBEGA1UE
AxMKa3ViZXJuZXRlczAeFw0xODA1MTQwODE3MTNaFw0xOTA1MTQwODE3MTdaMDQx
FzAVBgNVBAoTDnN5c3RlbTptYXN0ZXJzMRkwFwYDVQQDExBrdWJlcm5ldGVzLWFk
bWluMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAxBn3jdw80b1Gfb6s
w2NrqpLotMT4nyAf2HhqMrXjnO+wnaK1AITOw/22mDj0rwIuJwdQIj5/BaF63pPE
1pU0vhIPVK4n6JI4dmMzo/lR3jZpGeZW1zdXaCovw9c7clbiHo/mFG4xqytVLfX4
/S8mFp2A9QcieJGIo5S0BR3FZlU1PM7DRbLDVVq1PdyNY2GfsbGrHlGgXvWAKCd/
H79gAqVoTXjSIWCVYuYcoLvdvVXQSIiYlpXFP1jBQLvcU7vrqtb12RmrxnpkW4pl
dGEOX2sLmfYZ5TiFpkRwz2GxsmWyRbt6OuISJFI6RZ0r+Rn4yMDKPrYlEngDVc5K
PZ5zmwIDAQABoycwJTAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAwwCgYIKwYBBQUH
AwIwDQYJKoZIhvcNAQELBQADggEBAEZNTvTz2OgzCUdvMFbrhPsp+mD2vPjMRCxi
3BkA10vICOSfdymMn8aw0IbKYz2gQbXqUfqzQmQfa3if+QYBkB+77zfsv9am4EP/
e6Tg52tqV2P7s2eF7tNAe20GyV6yFlQ1QUW5/M4M+JMlV+BUbl9yEyQlENucKf+u
TPyKKTUtzvUYr5E3EJkt84EQINvw2nR2jNveZ1XWOliUrKfjHHtfvO/n56USuI4w
u2LTlICRcj4g+ZWlIjeMFkGyPbJyJAQ65P2sGrZm1klGGH3mzwO5CP1yZWvoUjjP
jzSjMCIaK/fR8eRAJ6q1tT6bG26L+njkKCQDWKpjAWOapuROcbk=
-----END CERTIFICATE-----

查看证书内容:

# openssl x509 -in ./admin-client.crt -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 9180400125522743476 (0x7f67554639bc10b4)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: May 14 08:17:13 2018 GMT
            Not After : May 14 08:17:17 2019 GMT
        Subject: O=system:masters, CN=kubernetes-admin
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

   ... ...

从证书输出的信息中,我们看到了下面这行:

Subject: O=system:masters, CN=kubernetes-admin

k8s apiserver对kubectl的请求进行client certificate验证(通过ca证书–client-ca-file=/etc/kubernetes/pki/ca.crt对其进行校验),验证通过后kube-apiserver会得到:group = system:masters的http上下文信息,并传给后续的authorizers。

三. 在授权(authorization)时根据Group确定所绑定的角色(Role)

kubeadm在init初始引导集群启动过程中,创建了许多default的role、clusterrole、rolebinding和clusterrolebinding,在k8s有关RBAC的官方文档中,我们看到下面一些default clusterrole列表:

img{512x368}

其中第一个cluster-admin这个cluster role binding绑定了system:masters group,这和authentication环节传递过来的身份信息不谋而合。沿着system:masters group对应的cluster-admin clusterrolebinding“追查”下去,真相就会浮出水面。

我们查看一下这一binding:

# kubectl get clusterrolebinding/cluster-admin -n kube-system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2018-06-07T06:14:55Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
  resourceVersion: "103"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/cluster-admin
  uid: 18c89690-6a1a-11e8-a0e8-00163e0cd764
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:masters

我们看到在kube-system名字空间中,一个名为cluster-admin的clusterrolebinding将cluster-admin cluster role与system:masters Group绑定到了一起,赋予了所有归属于system:masters Group中用户cluster-admin角色所拥有的权限。

我们再来查看一下cluster-admin这个role的具体权限信息:

# kubectl get clusterrole/cluster-admin -n kube-system -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    rbac.authorization.kubernetes.io/autoupdate: "true"
  creationTimestamp: 2018-06-07T06:14:55Z
  labels:
    kubernetes.io/bootstrapping: rbac-defaults
  name: cluster-admin
  resourceVersion: "52"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/cluster-admin
  uid: 18abe535-6a1a-11e8-a0e8-00163e0cd764
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'

从rules列表中来看,cluster-admin这个角色对所有resources、verbs、apiGroups均有无限制的操作权限,即整个集群的root权限。于是kubectl的请求就可以操控和管理整个集群了。

四. 小结

至此,我们应该明确了为什么采用了admin.conf kubeconfig的kubectrl拥有root权限了。下面是一幅示意图,简要总结了对kubectl访问请求的身份验证和授权过程:

img{512x368}

大家可以结合这幅图,重温一下上面的文字描述,加深一下理解。


51短信平台:企业级短信平台定制开发专家 https://51smspush.com/
smspush : 可部署在企业内部的定制化短信平台,三网覆盖,不惧大并发接入,可定制扩展; 短信内容你来定,不再受约束, 接口丰富,支持长短信,签名可选。

著名云主机服务厂商DigitalOcean发布最新的主机计划,入门级Droplet配置升级为:1 core CPU、1G内存、25G高速SSD,价格5$/月。有使用DigitalOcean需求的朋友,可以打开这个链接地址:https://m.do.co/c/bff6eed92687 开启你的DO主机之路。

我的联系方式:

微博:https://weibo.com/bigwhite20xx
微信公众号:iamtonybai
博客:tonybai.com
github: https://github.com/bigwhite

微信赞赏:
img{512x368}

商务合作方式:撰稿、出书、培训、在线课程、合伙创业、咨询、广告合作。

在Kubernetes 1.10.3上以Hard模式搭建EFK日志分析平台

在一年多之前,我曾写过一篇文章《使用Fluentd和ElasticSearch Stack实现Kubernetes的集群Logging》,文中讲解了如何在Kubernetes上利用EFK(elastic, fluentd, kibana)搭建一套可用的集中日志分析平台。当时的k8s使用的是1.3.7版本,创建EFK使用的是kubernetes项目cluster/addons/fluentd-elasticsearch下面的全套yaml文件,yaml中Elastic Search的volume用的还是emptyDir,并未真正持久化。

经过一年多的发展,Kubernetes发生了“翻天覆地”的变化,EFK技术栈也有了很大的进展。虽然那篇文章中的方案、步骤以及问题的解决思路仍有参考价值,但毕竟“年代”不同了,有些东西需要“与时俱进”。恰好近期在协助同事搭建一个移动互联网医院的演示环境时,我又一次搭建了一套“较新”版本的EFK,这里记录一下搭建过程、遇到的坑以及问题的解决过程,算是对之前“陈旧知识”的一个更新吧。

一. 环境和部署方案

这次部署我使用了较新的Kubernetes stable版本:1.10.3,这是一个单master node和三个worker node组成的演示环境,集群由kubeadm创建并引导启动。经过这些年的发展和演进,kubeadm引导启动的集群已经十分稳定了,并且搭建过程也是十分顺利(集群使用的是weave network插件)。

在EFK部署方案上,我没有再选择直接使用kubernetes项目cluster/addons/fluentd-elasticsearch下面的全套yaml文件,而是打算逐个组件单独安装的hard模式。

下面是一个部署示意图:

img{512x368}

虽然Kubernetes在持久化存储方面有诸多机制和插件可用,但总体来说,目前的k8s在storage这块依旧是短板,用起来体验较差,希望Container Storage Interface, CSI的引入和未来发展能降低开发人员的心智负担。因此,这次我将Elastic Search放在了k8s集群外单独单点部署,并直接使用local file system进行数据存取;fluentd没有变化,依旧是以DaemonSet控制的Pod的形式运行在每个k8s node上; kibana部署在集群内部,并通过ingress将服务暴露到集群外面。

二. 部署Elastic Search

按照部署方案,我们将Elastic Search部署在k8s集群外面,但我们依旧使用容器化部署方式。Elastic Search的官方镜像仓库已经由docker hub迁移到elasticsearch自己维护的仓库了。

我们下载当前ElasticSearch的最新版6.2.4:

docker pull docker.elastic.co/elasticsearch/elasticsearch:6.2.4

# docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
docker.elastic.co/elasticsearch/elasticsearch   6.2.4               7cb69da7148d        8 weeks ago         515 MB

在本地创建elasticsearch的数据存储目录:~/es_data,修改该目录的owner和group均为1000:

# mkdir ~/es_data
# chmod g+rwx es_data
# chgrp 1000 es_data
# chown 1000 -R es_data

# ls -l /root/es_data/
total 8
drwxrwxr-x 2 1000 1000 4096 Jun  8 09:50 ./
drwx------ 8 root root 4096 Jun  8 09:50 ../

注意:务必对es_data按上述命令执行修改,否则在启动elasticsearch容器可能会出现如下错误:

[WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
_*org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment*_
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
... ...
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
... ...

启动elasticsearch容器:

# docker run -d --restart=unless-stopped -p 9200:9200 -p 9300:9300 -v /root/es_data:/usr/share/elasticsearch/data --ulimit nofile=65536:65536 -e "bootstrap.memory_lock=true" --ulimit memlock=-1:-1 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:6.2.4

如果看到下面日志,说明elasticsearch容器启动成功了!

[INFO ][o.e.c.m.MetaDataCreateIndexService] [sGZc7Wa] [.monitoring-es-6-2018.06.08] creating index, cause [auto(bulk api)], templates [.monitoring-es], shards [1]/[0], mappings [doc]
[INFO ][o.e.c.r.a.AllocationService] [sGZc7Wa] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.monitoring-es-6-2018.06.08][0]] ...]).

检查es健康状态:

# curl http://127.0.0.1:9200/_cat/health
1528424599 02:23:19 docker-cluster green 1 1 1 1 0 0 0 0 - 100.0%

es工作一切健康!

三. 部署Fluentd

相比较而言,fluentd的部署相对简单,因为fluentd官网文档有明确的安装说明。由于k8s默认授权机制采用了RBAC,因此我们使用fluentd-daemonset-elasticsearch.yaml来创建fluentd daemonset。

不过在创建前,我们需要打开fluentd-daemonset-elasticsearch.yaml修改一下它连接的elasticsearch的地址信息:

      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:elasticsearch
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "172.16.66.104" // 172.16.66.104就是我们的elasticsearch运行的节点的ip

接下来创建fluentd:

# kubectl apply -f fluentd-daemonset-elasticsearch-rbac.yaml
serviceaccount "fluentd" created
clusterrole.rbac.authorization.k8s.io "fluentd" created
clusterrolebinding.rbac.authorization.k8s.io "fluentd" created
daemonset.extensions "fluentd" created

查看某一个fluentd pod的启动日志如下:

# kubectl logs -f pods/fluentd-4rptt -n kube-system
[info]: reading config file path="/fluentd/etc/fluent.conf"
[info]: starting fluentd-0.12.33
[info]: gem 'fluent-plugin-elasticsearch' version '1.16.0'
[info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '1.0.2'
[info]: gem 'fluent-plugin-record-reformer' version '0.9.1'
[info]: gem 'fluent-plugin-secure-forward' version '0.4.5'
[info]: gem 'fluentd' version '0.12.33'
[info]: adding match pattern="fluent.**" type="null"
[info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
[info]: adding match pattern="**" type="elasticsearch"
[info]: adding source type="tail"
... ...
[info]: following tail of /var/log/containers/weave-net-9kds5_kube-system_weave-13ef6f321b2bc64dc920878c7d361440c0157b91f6025f23c631edb5feb3473a.log
[info]: following tail of /var/log/containers/fluentd-4rptt_kube-system_fluentd-bdc80586d5cafc10729fb277ce01cf28d595059eabf96b66324f32b3b6873e28.log
[info]: Connection opened to Elasticsearch cluster => {:host=>"172.16.66.104", :port=>9200, :scheme=>"http", :user=>"elastic", :password=>"obfuscated"}
... ...

没有报错!似乎fluentd启动ok了。

再来通过elasticsearch日志验证一下:

[INFO ][o.e.c.m.MetaDataCreateIndexService] [sGZc7Wa] [logstash-2018.06.07] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[INFO ][o.e.c.m.MetaDataCreateIndexService] [sGZc7Wa] [logstash-2018.06.08] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.07/XetLly2ZQFKKd0JVvxl5fA] create_mapping [fluentd]
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.07/XetLly2ZQFKKd0JVvxl5fA] update_mapping [fluentd]
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.07/XetLly2ZQFKKd0JVvxl5fA] update_mapping [fluentd]
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.08/j5soBzyVSNOvBQg-E3NkCA] create_mapping [fluentd]
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.08/j5soBzyVSNOvBQg-E3NkCA] update_mapping [fluentd]
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.08/j5soBzyVSNOvBQg-E3NkCA] update_mapping [fluentd]
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.07/XetLly2ZQFKKd0JVvxl5fA] update_mapping [fluentd]
[INFO ][o.e.c.m.MetaDataMappingService] [sGZc7Wa] [logstash-2018.06.08/j5soBzyVSNOvBQg-E3NkCA] update_mapping [fluentd]

fluentd已经成功连接上es了!

四. 部署Kibana

我们将kibana部署到Kubernetes集群内,我们使用kubernetes项目中的cluster/addons/fluentd-elasticsearch下的kibana yaml文件来创建kibana部署和服务:

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/kibana-deployment.yaml

https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/fluentd-elasticsearch/kibana-service.yaml

创建前,我们需要修改一下kibana-deployment.yaml:

... ...
        image: docker.elastic.co/kibana/kibana:6.2.4  // 这里,我们使用最新的版本:6.2.4

          - name: ELASTICSEARCH_URL
            value: http://172.16.66.104:9200  //这里,我们用上面的elasticsearch的服务地址填入到value的值中
.... ...

创建kibana:

# kubectl apply -f kibana-service.yaml
service "kibana-logging" created
# kubectl apply -f kibana-deployment.yaml
deployment.apps "kibana-logging" created

查看启动的kibana pod,看到如下错误日志:

{"type":"log","@timestamp":"2018-06-08T07:09:08Z","tags":["fatal"],"pid":1,"message":"\"xpack.monitoring.ui.container.elasticsearch.enabled\" setting was not applied. Check for spelling errors and ensure that expected plugins are installed and enabled."}
FATAL "xpack.monitoring.ui.container.elasticsearch.enabled" setting was not applied. Check for spelling errors and ensure that expected plugins are installed and enabled.

似乎与xpack有关。我们删除kibana-deployment.yaml中的两个环境变量:XPACK_MONITORING_ENABLED和XPACK_SECURITY_ENABLED,再重新apply。查看kibana pod日志:

# kubectl logs -f kibana-logging-648dbdf986-bc24x -n kube-system
{"type":"log","@timestamp":"2018-06-08T07:16:27Z","tags":["status","plugin:kibana@6.2.4","info"],"pid":1,"state":"green","message":"Status changed from uninitialized to green - Ready","prevState":"uninitialized","prevMsg":"uninitialized"}
{"type":"log","@timestamp":"2018-06-08T07:16:27Z","tags":["status","plugin:elasticsearch@6.2.4","info"],"pid":1,"state":"yellow","message":"Status changed from uninitialized to yellow - Waiting for Elasticsearch","prevState":"uninitialized","prevMsg":"uninitialized"}
... ...
{"type":"log","@timestamp":"2018-06-08T07:16:30Z","tags":["info","monitoring-ui","kibana-monitoring"],"pid":1,"message":"Starting all Kibana monitoring collectors"}
{"type":"log","@timestamp":"2018-06-08T07:16:30Z","tags":["license","info","xpack"],"pid":1,"message":"Imported license information from Elasticsearch for the [monitoring] cluster: mode: basic | status: active | expiry date: 2018-07-08T02:06:08+00:00"}

可以看到kibana启动成功!

使用kubectl proxy启动代理,在浏览器中建立sock5 proxy,然后在浏览器访问:http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy, 你应该可以看到下面的kibana首页:

img{512x368}

创建index pattern后,等待一会,查看边栏中的”Discover”,如果你看到类似下面截图中的日志内容输出,说明kibana可以正常从elasticsearch获取数据了:

img{512x368}

五. 为kibana添加ingress

使用kubectl proxy查看kibana虽然简单,但略显麻烦,将kibana服务暴露到集群外更为方便。下面我们就给kibana添加带basic auth的ingress。

1. 部署ingress controller及默认后端(如果cluster已经部署过,则忽略此步骤)

我们选择k8s官方的ingress-nginx作为ingress controller,并部署默认后端default-backend,我们把ingress-nginx controller和default-backend统统部署在kube-system命令空间下。

下载https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml

mandatory.yaml中的namespace的值都改为kube-system

docker pull anjia0532/defaultbackend:1.4
docker tag anjia0532/defaultbackend:1.4 gcr.io/google_containers/defaultbackend:1.4
docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0

# kubectl apply -f mandatory.yaml
deployment.extensions "default-http-backend" created
service "default-http-backend" created
configmap "nginx-configuration" created
configmap "tcp-services" created
configmap "udp-services" created
serviceaccount "nginx-ingress-serviceaccount" created
clusterrole.rbac.authorization.k8s.io "nginx-ingress-clusterrole" created
role.rbac.authorization.k8s.io "nginx-ingress-role" created
rolebinding.rbac.authorization.k8s.io "nginx-ingress-role-nisa-binding" created
clusterrolebinding.rbac.authorization.k8s.io "nginx-ingress-clusterrole-nisa-binding" created
deployment.extensions "nginx-ingress-controller" created

此时nginx-ingress controller已经安装完毕,nginx-ingress controller本质上就是一个nginx,目前它还没有暴露服务端口,我们通过nodeport方式暴露nginx-ingress service到集群外面:

下载https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml

修改service-nodeport.yaml:

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: kube-system
spec:
  type: NodePort
  ports:
  - name: http
    port: 80
    targetPort: 80
    nodePort: 30080
    protocol: TCP
  - name: https
    port: 443
    targetPort: 443
    nodePort: 30443
    protocol: TCP
  selector:
    app: ingress-nginx

# kubectl apply -f service-nodeport.yaml
service "ingress-nginx" created
# lsof -i tcp:30080
COMMAND     PID USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
kube-prox 24565 root    9u  IPv6 10447591      0t0  TCP *:30080 (LISTEN)

我们验证一下nginx-ingress controller工作是否正常:

在任意一个集群node上:

# curl localhost:30080
default backend - 404

2. 为kibana添加ingress

ingress是一种抽象。对于nginx ingress controller来说,创建一个ingress相当于在nginx.conf中添加一个server入口,并nginx -s reload生效。

我们创建kibana的ingress yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
  name: kibana-logging-ingress
  namespace: kube-system
spec:
  rules:
  - host: kibana.tonybai.com
    http:
      paths:
      - backend:
          serviceName: kibana-logging
          servicePort: 5601

由于ingress中的host只能是域名,这里用 kibana.tonybai.com,然后在/etc/hosts中增加该域名的ip地址映射。

创建kibana-logging-ingress:

# kubectl apply -f kibana-logging-ingress.yaml
ingress.extensions "kibana-logging-ingress" created

此时,我们打开浏览器,访问http://kibana.tonybai.com:30080,我们得到了如下结果:

{"statusCode":404,"error":"Not Found","message":"Not Found"}

我们再次用curl试一下:

# curl -L kibana.tonybai.com:30080
<script>var hashRoute = '/api/v1/namespaces/kube-system/services/kibana-logging/proxy/appl;
var defaultRoute = '/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;

这显然不是我们预想的结果。我们查看一下kibana pod对应的日志,并对比了一下使用kubectl proxy访问kibana的日志:

通过ingress访问的错误日志:

{"type":"response","@timestamp":"2018-06-11T10:20:55Z","tags":[],"pid":1,"method":"get","statusCode":404,"req":{"url":"/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana","method":"get","headers":{"host":"kibana.tonybai.com:30080","connection":"close","x-request-id":"b066d69c31ce3c9e89efa6264966561c","x-real-ip":"192.168.16.1","x-forwarded-for":"192.168.16.1","x-forwarded-host":"kibana.tonybai.com:30080","x-forwarded-port":"80","x-forwarded-proto":"http","x-original-uri":"/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana","x-scheme":"http","cache-control":"max-age=0","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8","accept-language":"zh-CN,zh;q=0.9,en;q=0.8,zh-TW;q=0.7"},"remoteAddress":"192.168.20.5","userAgent":"192.168.20.5"},"res":{"statusCode":404,"responseTime":4,"contentLength":9},"message":"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana 404 4ms - 9.0B"}

通过kubectl proxy访问的正确日志:

{"type":"response","@timestamp":"2018-06-11T10:20:43Z","tags":[],"pid":1,"method":"get","statusCode":304,"req":{"url":"/ui/fonts/open_sans/open_sans_v13_latin_regular.woff2","method":"get","headers":{"host":"localhost:8001","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36","accept":"*/*","accept-encoding":"gzip, deflate, br","accept-language":"zh-CN,zh;q=0.9,en;q=0.8,zh-TW;q=0.7","if-modified-since":"Thu, 12 Apr 2018 20:57:06 GMT","if-none-match":"\"afc44700053c9a28f9ab26f6aec4862ac1d0795d\"","origin":"http://localhost:8001","referer":"http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana","x-forwarded-for":"127.0.0.1, 172.16.66.101","x-forwarded-uri":"/api/v1/namespaces/kube-system/services/kibana-logging/proxy/ui/fonts/open_sans/open_sans_v13_latin_regular.woff2"},"remoteAddress":"192.168.16.1","userAgent":"192.168.16.1","referer":"http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana"},"res":{"statusCode":304,"responseTime":3,"contentLength":9},"message":"GET /ui/fonts/open_sans/open_sans_v13_latin_regular.woff2 304 3ms - 9.0B"}

我们看到通过ingress访问,似乎将/api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana这个url path也传递给后面的kibana了,而kibana却无法处理。

我们回头看一下kibana-deployment.yaml,那里面有一个env var:

          - name: SERVER_BASEPATH
            value: /api/v1/namespaces/kube-system/services/kibana-logging/proxy

问题似乎就出在这里。我们去掉这个env var,并重新apply kibana-deployment.yaml。然后再用浏览器访问:http://kibana.tonybai.com:30080/app/kibana,kibana的页面就会出现在眼前了。

但是这样更新后,通过kubectl proxy方式似乎就无法正常访问kibana了,这里也只能二选一了,我们选择ingress访问。

3. 添加basic auth for kibana-logging ingress

虽然kibana ingress生效了,但目前kibana ingress目前在“裸奔”,我们还是要适当加上一些auth的,我们选择basic auth,从原理上讲这是加到nginx上的basic auth,kibana自身并没有做basic auth:

我们借助htpasswd工具生成用户名和密码,并基于此创建secret对象:

# htpasswd -c auth tonybai
New password:
Re-type new password:
Adding password for user tonybai

# cat auth
tonybai:$apr1$pQuJZfll$KPfa1rXJUTBBKktxtbVsI0

#kubectl create secret generic basic-auth --from-file=auth -n kube-system
secret "basic-auth" created

# kubectl get secret basic-auth -o yaml -n kube-system
apiVersion: v1
data:
  auth: dG9ueWJhaTokYXByMSRwUXVKWmZsbCRLUGZhMXJYSlVUQkJLa3R4dGJWc0kwCg==
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"auth":"dG9ueWJhaTokYXByMSRwUXVKWmZsbCRLUGZhMXJYSlVUQkJLa3R4dGJWc0kwCg=="},"kind":"Secret","metadata":{"annotations":{},"name":"basic-auth","namespace":"kube-system"},"type":"Opaque"}
  creationTimestamp: 2018-06-11T23:05:42Z
  name: basic-auth
  namespace: kube-system
  resourceVersion: "579134"
  selfLink: /api/v1/namespaces/kube-system/secrets/basic-auth
  uid: f6ec373e-6dcb-11e8-a0e8-00163e0cd764
type: Opaque

在kibana-logging-ingress.yaml中增加有关auth的annotations:

// kibana-logging-ingress.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "Authentication Required - tonybai"
  name: kibana-logging-ingress
  namespace: kube-system
spec:
  rules:
  - host: kibana.tonybai.com
    http:
      paths:
      - backend:
          serviceName: kibana-logging
          servicePort: 5601

apply kibana-logging-ingress.yaml后,我们再次访问:kibana.tonybai.com:30080

img{512x368}

至此,一个演示环境下的EFK日志平台就搭建完毕了。相信有了这种hard way的安装搭建经验,我们可以灵活应对针对其中某个组件的变种部署了(比如将elasticsearch放到k8s中部署)。


51短信平台:企业级短信平台定制开发专家 https://51smspush.com/
smspush : 可部署在企业内部的定制化短信平台,三网覆盖,不惧大并发接入,可定制扩展; 短信内容你来定,不再受约束, 接口丰富,支持长短信,签名可选。

著名云主机服务厂商DigitalOcean发布最新的主机计划,入门级Droplet配置升级为:1 core CPU、1G内存、25G高速SSD,价格5$/月。有使用DigitalOcean需求的朋友,可以打开这个链接地址:https://m.do.co/c/bff6eed92687 开启你的DO主机之路。

我的联系方式:

微博:https://weibo.com/bigwhite20xx
微信公众号:iamtonybai
博客:tonybai.com
github: https://github.com/bigwhite

微信赞赏:
img{512x368}

商务合作方式:撰稿、出书、培训、在线课程、合伙创业、咨询、广告合作。

欢迎使用邮件订阅我的博客

输入邮箱订阅本站,只要有新文章发布,就会第一时间发送邮件通知你哦!

这里是 Tony Bai的个人Blog,欢迎访问、订阅和留言! 订阅Feed请点击上面图片

如果您觉得这里的文章对您有帮助,请扫描上方二维码进行捐赠 ,加油后的Tony Bai将会为您呈现更多精彩的文章,谢谢!

如果您希望通过微信捐赠,请用微信客户端扫描下方赞赏码:

如果您希望通过比特币或以太币捐赠,可以扫描下方二维码:

比特币:

以太币:

如果您喜欢通过微信浏览本站内容,可以扫描下方二维码,订阅本站官方微信订阅号“iamtonybai”;点击二维码,可直达本人官方微博主页^_^:
本站Powered by Digital Ocean VPS。
选择Digital Ocean VPS主机,即可获得10美元现金充值,可 免费使用两个月哟! 著名主机提供商Linode 10$优惠码:linode10,在 这里注册即可免费获 得。阿里云推荐码: 1WFZ0V立享9折!


View Tony Bai's profile on LinkedIn

文章

评论

  • 正在加载...

分类

标签

归档



View My Stats