解决Kubernetes 1.6.4 Dashboard无法访问的问题
前一段时间将之前采用kubeadm安装的Kubernetes 1.5.1环境升级到了1.6.4版本，升级过程较为顺利。由于该k8s cluster是一个测试环境，当时并没有过于关注，就忙别的事情了。最近项目组打算在这个环境下做一些事情，而当我们重新“捡起”这个环境时，发现Kubernetes Dashboard无法访问了。
Kubernetes的dashboard可以有很多种访问方式，比如：可以通过暴露nodeport的方式(无身份验证，不安全)、可以通过访问apiserver的api服务的方式等。我们的Dashboard通过APIServer进行访问：
https://apiserver_ip:secure_port/ui
正常情况下通过浏览器访问：https://apiserver_ip:secure_port/ui，浏览器会弹出身份验证对话框，待输入正确的用户名和密码后，便可成功进入Dashboard了。但当前，我们得到的结果却是：
User "system:anonymous" cannot proxy services in the namespace "kube-system".
而访问apiserver(https://apiserver_ip:secure_port/)得到的结果如下：
User "system:anonymous" cannot get at the cluster scope.
一、问题原因分析
k8s 1.6.x版本与1.5.x版本的一个很大不同在于1.6.x版本启用了RBAC的Authorization mode(授权模型)，这点在K8s master init的日志中可以得到证实：
# kubeadm init --apiserver-advertise-address xx.xx.xx
... ...
[init] Using Kubernetes version: v1.6.4
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key
.... ...
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns
Your Kubernetes master has initialized successfully!
... ...
在《Kubernetes集群的安全配置》一文中我们提到过Kubernetes API server的访问方法：
Authentication(身份验证) -> Authorization（授权）-> Admission Control(入口条件控制)
只不过在Kubernetes 1.5.x及以前的版本中，Authorization的环节都采用了默认的配置，即”AlwaysAllow”，对访问APIServer并不产生什么影响：
# kube-apiserver -h
... ...
--authorization-mode="AlwaysAllow": Ordered list of plug-ins to do authorization on secure port. Comma-delimited list of: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC
... ...
但K8s 1.6.x版本中，–authorization-mode的值发生了变化：
# cat /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
containers:
- command:
- kube-apiserver
- --allow-privileged=true
... ...
- --basic-auth-file=/etc/kubernetes/basic_auth_file
- --authorization-mode=RBAC
... ...
注：这里我们依旧通过basic auth方式进行apiserver的Authentication，而不是用客户端数字证书校验等其他方式。
显然问题的原因就在于这里RBAC授权方式的使用，让我们无法正常访问Dashboard了。
二、Kubernetes RBAC Authorization简介
RBAC Authorization的基本概念是Role和RoleBinding。Role是一些permission的集合；而RoleBinding则是将Role授权给某些User、某些Group或某些ServiceAccount。K8s官方博客《RBAC Support in Kubernetes》一文的中的配图对此做了很生动的诠释：
从上图中我们可以看到：
Role: pod-reader 拥有Pod的get和list permissions；
RoleBinding: pod-reader 将Role: pod-reader授权给右边的User、Group和ServiceAccount。
和Role和RoleBinding对应的是，K8s还有ClusterRole和ClusterRoleBinding的概念，它们不同之处在于：ClusterRole和ClusterRoleBinding是针对整个Cluster范围内有效的，无论用户或资源所在的namespace是什么；而Role和RoleBinding的作用范围是局限在某个k8s namespace中的。
Kubernetes 1.6.4安装时内建了许多Role/ClusterRole和RoleBinds/ClusterRoleBindings：
# kubectl get role -n kube-system
NAME AGE
extension-apiserver-authentication-reader 50d
system:controller:bootstrap-signer 50d
system:controller:token-cleaner 50d
# kubectl get rolebinding -n kube-system
NAME AGE
system:controller:bootstrap-signer 50d
system:controller:token-cleaner 50d
# kubectl get clusterrole
NAME AGE
admin 50d
cluster-admin 50d
edit 50d
system:auth-delegator 50d
system:basic-user 50d
system:controller:attachdetach-controller 50d
... ...
system:discovery 50d
system:heapster 50d
system:kube-aggregator 50d
system:kube-controller-manager 50d
system:kube-dns 50d
system:kube-scheduler 50d
system:node 50d
system:node-bootstrapper 50d
system:node-problem-detector 50d
system:node-proxier 50d
system:persistent-volume-provisioner 50d
view 50d
weave-net 50d
# kubectl get clusterrolebinding
NAME AGE
cluster-admin 50d
kubeadm:kubelet-bootstrap 50d
kubeadm:node-proxier 50d
kubernetes-dashboard 50d
system:basic-user 50d
system:controller:attachdetach-controller 50d
... ...
system:controller:statefulset-controller 50d
system:controller:ttl-controller 50d
system:discovery 50d
system:kube-controller-manager 50d
system:kube-dns 50d
system:kube-scheduler 50d
system:node 50d
system:node-proxier 50d
weave-net 50d
三、Dashboard的role和rolebinding
Kubernetes 1.6.x启用RBAC后，诸多周边插件也都推出了适合K8s 1.6.x的manifest描述文件，比如：weave-net等。Dashboard的manifest文件中也增加了关于rolebinding的描述，我当初用的是1.6.1版本，文件内容摘录如下：
// kubernetes-dashboard.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
... ...
我们看到在kubernetes-dashboard.yaml中，描述文件新建了一个ClusterRoleBinding：kubernetes-dashboard。该binding将ClusterRole: cluster-admin授权给了一个ServiceAccount: kubernetes-dashboard。我们看看ClusterRole: cluster-admin都包含了哪些permission:
# kubectl get clusterrole/cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
annotations:
rbac.authorization.kubernetes.io/autoupdate: "true"
creationTimestamp: 2017-05-30T14:06:39Z
labels:
kubernetes.io/bootstrapping: rbac-defaults
name: cluster-admin
resourceVersion: "11"
selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolescluster-admin
uid: 331c79dc-4541-11e7-bc9a-12584ec3a8c9
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
- nonResourceURLs:
- '*'
verbs:
- '*'
可以看到，在rules设定中，cluster-admin似乎拥有了“无限”权限。不过注意：这里仅仅授权给了一个service account，并没有授权给user或group。并且这里的kubernetes-dashboard是dashboard访问apiserver时使用的(下图右侧流程)，并不是user访问APIServer时使用的。
我们需要给登录dashboard或者说apiserver的user(图左侧)进行授权。
四、为user: admin进行授权
我们的kube-apiserver的启动参数中包含：
- --basic-auth-file=/etc/kubernetes/basic_auth_file
也就是说我们访问apiserver使用的是basic auth的身份验证方式，而user恰为admin。而从本文开头的错误现象来看，admin这个user并未得到足够的授权。这里我们要做的就是给admin选择一个合适的clusterrole。但kubectl并不支持查看user的信息，初始的clusterrolebinding又那么多，一一查看十分麻烦。我们知道cluster-admin这个clusterrole是全权限的，我们就来将admin这个user与clusterrole: cluster-admin bind到一起：
# kubectl create clusterrolebinding login-on-dashboard-with-cluster-admin --clusterrole=cluster-admin --user=admin
clusterrolebinding "login-on-dashboard-with-cluster-admin" created
# kubectl get clusterrolebinding/login-on-dashboard-with-cluster-admin -o yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
creationTimestamp: 2017-07-20T08:57:07Z
name: login-on-dashboard-with-cluster-admin
resourceVersion: "5363564"
selfLink: /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindingslogin-on-dashboard-with-cluster-admin
uid: 686a3f36-6d29-11e7-8f69-00163e1001d7
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: admin
binding后，我们再来访问一下dashboard UI，不出意外的话，熟悉的dashboard界面就会出现在你的眼前。
注：Kubernetes API Server新增了–anonymous-auth选项，允许匿名请求访问secure port。没有被其他authentication方法拒绝的请求即Anonymous requests， 这样的匿名请求的username为”system:anonymous”, 归属的组为”system:unauthenticated”。并且该选线是默认的。这样一来，当采用chrome浏览器访问dashboard UI时很可能无法弹出用户名、密码输入对话框，导致后续authorization失败。为了保证用户名、密码输入对话框的弹出，需要将–anonymous-auth设置为false：
// /etc/kubernetes/manifests/kube-apiserver.yaml
- --anonymous-auth=false
用curl测试结果如下：
$curl -u admin:YOUR_PASSWORD -k https://apiserver_ip:secure_port/
{
"paths": [
"/api",
"/api/v1",
"/apis",
"/apis/apps",
"/apis/apps/v1beta1",
"/apis/authentication.k8s.io",
"/apis/authentication.k8s.io/v1",
"/apis/authentication.k8s.io/v1beta1",
"/apis/authorization.k8s.io",
"/apis/authorization.k8s.io/v1",
"/apis/authorization.k8s.io/v1beta1",
"/apis/autoscaling",
"/apis/autoscaling/v1",
"/apis/autoscaling/v2alpha1",
"/apis/batch",
"/apis/batch/v1",
"/apis/batch/v2alpha1",
"/apis/certificates.k8s.io",
"/apis/certificates.k8s.io/v1beta1",
"/apis/extensions",
"/apis/extensions/v1beta1",
"/apis/policy",
"/apis/policy/v1beta1",
"/apis/rbac.authorization.k8s.io",
"/apis/rbac.authorization.k8s.io/v1alpha1",
"/apis/rbac.authorization.k8s.io/v1beta1",
"/apis/settings.k8s.io",
"/apis/settings.k8s.io/v1alpha1",
"/apis/storage.k8s.io",
"/apis/storage.k8s.io/v1",
"/apis/storage.k8s.io/v1beta1",
"/healthz",
"/healthz/ping",
"/healthz/poststarthook/bootstrap-controller",
"/healthz/poststarthook/ca-registration",
"/healthz/poststarthook/extensions/third-party-resources",
"/healthz/poststarthook/rbac/bootstrap-roles",
"/logs",
"/metrics",
"/swaggerapi/",
"/ui/",
"/version"
]
}
微博：@tonybai_cn
微信公众号：iamtonybai
github.com: https://github.com/bigwhite
评论