一步步打造基于Kubeadm的高可用Kubernetes集群-第二部分

续接上文。

五、第三步：启动emei、wudang上的apiserver

跨三个node的etcd cluster已经建成并完成了数据同步，下面进行ha cluster改造的重要一步：启动wudang、emei上的apiserver

1、启动emei、wudang上的apiserver

以shaolin node上的/etc/kubernetes/manifests/kube-apiserver.yaml为副本，制作emei、wudang上的kube-apiserver.yaml：

唯一需要变动的就是- --advertise-address这个option的值：

wudang:

- --advertise-address=10.24.138.208

emei:

- --advertise-address=10.27.52.72

在各自node上将kube-apiserver.yaml放入/etc/kubernetes/manifests中，各自node上的kubelet将会启动kube-apiserver并且各个apiserver默认连接本节点的etcd:

root@emei:~# pods
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
... ...
kube-system   kube-apiserver-emei               1/1       Running   0          1d        10.27.52.72     emei
kube-system   kube-apiserver-shaolin            1/1       Running   0          1d        10.27.53.32     shaolin
kube-system   kube-apiserver-wudang             1/1       Running   0          2d        10.24.138.208   wudang

2、将emei、wudang上的kubelet改为连接自己所在节点的apiserver

所有apiserver都启动了。wudang、emei上的kubelet也应该连接自己节点的apiserver了！修改各自的/etc/kubernetes/kubelet.conf，修改server配置项：

wudang:

server: https://10.24.138.208:6443

emei:

server: https://10.27.52.72:6443

各自重启kubelet:

以wudang为例：

root@wudang:~# systemctl daemon-reload
root@wudang:~# systemctl restart kubelet

不过，问题出现了！查看重启的kubelet日志：

root@wudang:~# journalctl -u kubelet -f
-- Logs begin at Mon 2017-05-08 15:12:01 CST. --
May 11 14:33:27 wudang kubelet[8794]: I0511 14:33:27.919223    8794 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
May 11 14:33:27 wudang kubelet[8794]: I0511 14:33:27.921166    8794 kubelet_node_status.go:77] Attempting to register node wudang
May 11 14:33:27 wudang kubelet[8794]: E0511 14:33:27.926865    8794 kubelet_node_status.go:101] Unable to register node "wudang" with API server: Post https://10.24.138.208:6443/api/v1/nodes: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208
May 11 14:33:28 wudang kubelet[8794]: E0511 14:33:28.283258    8794 event.go:208] Unable to write event: 'Post https://10.24.138.208:6443/api/v1/namespaces/default/events: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208' (may retry after sleeping)
May 11 14:33:28 wudang kubelet[8794]: E0511 14:33:28.499209    8794 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.24.138.208:6443/api/v1/nodes?fieldSelector=metadata.name%3Dwudang&resourceVersion=0: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208
May 11 14:33:28 wudang kubelet[8794]: E0511 14:33:28.504593    8794 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.24.138.208:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dwudang&resourceVersion=0: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208

从错误日志判断来看，似乎是wudang上的kubelet在与同一节点上的kube-apiserver通信过程中，发现这个apiserver返回的tls证书是属于10.27.53.32的，即shaolin node上的apiserver的，而不是wudang node上的apiserver的，于是报了错！问题的原因很明了，因为Wudang上的kube-apiserver用的apiserver.crt的确是从shaolin node上copy过来的。也就是说要解决这个问题，我们需要为wudang、emei两个node上的apiserver各自生成自己的数字证书。

我们先来查看一下shaolin上的apiserver.crt内容是什么样子的：

root@shaolin:/etc/kubernetes/pki# openssl x509 -noout -text -in apiserver.crt

Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes

Subject: CN=kube-apiserver

X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:shaolin, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.27.53.32

我们看到证书使用到了x509v3的扩展功能：subject alternative name，并且指定了多个value。我们为wudang、emei生成的apiserver.crt也应该如此。如何做呢？好在我们有整个集群的ca.key和ca.crt，可以用来签署证书请求。以wudang node为例，我们来为wudang node上的apiserver生成apiserver-wudang.key和apiserver-wudang.crt：

//生成2048位的密钥对
root@wudang:~# openssl genrsa -out apiserver-wudang.key 2048

//生成证书签署请求文件
root@wudang:~# openssl req -new -key apiserver-wudang.key -subj "/CN=kube-apiserver," -out apiserver-wudang.csr

// 编辑apiserver-wudang.ext文件，内容如下：
subjectAltName = DNS:wudang,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP:10.96.0.1, IP:10.24.138.208

// 使用ca.key和ca.crt签署上述请求
root@wudang:~# openssl x509 -req -in apiserver-wudang.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out apiserver-wudang.key.crt -days 365 -extfile apiserver-wudang.ext
Signature ok
subject=/CN=10.24.138.208
Getting CA Private Key

//查看新生成的证书：
root@wudang:~# openssl x509 -noout -text -in apiserver-wudang.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 16019625340257831745 (0xde51245f10ea0b41)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: May 12 08:40:40 2017 GMT
            Not After : May 12 08:40:40 2018 GMT
        Subject: CN=kube-apiserver,
        Subject Public Key Info:
            ... ...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:wudang, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.24.138.208

将apiserver-wudang.key和apiserver-wudang.crt放入/etc/kubernetes/pki目录下，修改kube-apiserver.yaml文件：

// /etc/kubernetes/pki
- --tls-cert-file=/etc/kubernetes/pki/apiserver-wudang.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver-wudang.key

kube-apiserver重启后，再来查看kubelet日志，你会发现kubelet运行一切ok了。emei节点也要进行同样的操作。

至此，整个集群的状态示意图如下：

六、第四步：启动emei、wudang上的kube-controller-manager和kube-scheduler

这一步我们只需要将shaolin node上的/etc/kubernetes/manifests中的kube-controller-manager.yaml和kube-scheduler.yaml拷贝到wudang、emei两个node的相应目录下即可：

root@emei:~/kubernetes-conf-shaolin/manifests# pods
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
... ...
kube-system   kube-controller-manager-emei      1/1       Running   0          8s        10.27.52.72     emei
kube-system   kube-controller-manager-shaolin   1/1       Running   3          1d        10.27.53.32     shaolin
kube-system   kube-controller-manager-wudang    1/1       Running   0          1m        10.24.138.208   wudang
... ...
kube-system   kube-scheduler-emei               1/1       Running   0          15s       10.27.52.72     emei
kube-system   kube-scheduler-shaolin            1/1       Running   3          1d        10.27.53.32     shaolin
kube-system   kube-scheduler-wudang             1/1       Running   0          3m        10.24.138.208   wudang
... ...

查看一下各个node下kcm和scheduler的日志：

root@wudang:~/demo# kubectl logs -f kube-controller-manager-emei -n kube-system
I0511 07:34:53.804831       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-controller-manager-wudang -n kube-system
I0511 07:33:20.725669       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-scheduler-emei -n kube-system
I0511 07:34:45.711032       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-scheduler-wudang -n kube-system
I0511 07:31:35.077090       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-scheduler-shaolin -n kube-system

I0512 08:55:30.838806       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-v1dst", UID:"c075c6c7-36f0-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"166279", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-v1dst to emei
I0512 08:55:30.843104       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-drnzv", UID:"c075da9f-36f0-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"166278", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-drnzv to wudang
I0512 09:13:21.121864       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-ld1dr", UID:"3e73d350-36f3-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"168070", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-ld1dr to wudang
I0512 09:13:21.124295       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-cmmkh", UID:"3e73c8b2-36f3-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"168071", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-cmmkh to emei

可以看出，当前shaolin node上的kcm和scheduler是leader。

至此，整个集群的状态示意图如下：

六、第五步：将wudang、emei设置为master node

我们试着在wudang节点上创建一个pod:

// run-my-nginx.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.10.1
        ports:
        - containerPort: 80

发现pod居然被调度到了wudang、emei节点上了！

NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
default       my-nginx-2267614806-drnzv         1/1       Running   0          5s        172.32.192.1    wudang
default       my-nginx-2267614806-v1dst         1/1       Running   0          5s        172.32.64.0     emei

emei、wudang并没有执行taint，为何能承载workload? 查看当前cluster的node状态：

root@wudang:~# kubectl get node --show-labels
NAME      STATUS    AGE       VERSION   LABELS
emei      Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=emei
shaolin   Ready     2d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=shaolin,node-role.kubernetes.io/master=
wudang    Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=wudang

从label看到，status列并没有明确输出谁是master，这和1.5.1版本以前似乎不同。emei、wudang与shaolin唯一的不同就是shaolin有一个key: node-role.kubernetes.io/master。难道这个label是指示谁是master的？我们给wudang打上这个label：

root@wudang:~/demo# kubectl label node wudang node-role.kubernetes.io/master=
node "wudang" labeled
root@wudang:~/demo# kubectl get node --show-labels
NAME      STATUS    AGE       VERSION   LABELS
emei      Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=emei
shaolin   Ready     2d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=shaolin,node-role.kubernetes.io/master=
wudang    Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=wudang,node-role.kubernetes.io/master=

再创建nginx pod，我们发现pod依旧分配在wudang、emei两个node上:

NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
default       my-nginx-2267614806-cmmkh         1/1       Running   0          5s        172.32.64.0     emei
default       my-nginx-2267614806-ld1dr         1/1       Running   0          5s        172.32.192.1    wudang

我们进一步查看并对比相关信息：

查看clustre-info：

wuddang node:
root@wudang:~/demo# kubectl cluster-info
Kubernetes master is running at https://10.24.138.208:6443 //wudang node:
KubeDNS is running at https://10.24.138.208:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns

shaolin node:

root@shaolin:~/k8s-install/demo# kubectl cluster-info
Kubernetes master is running at https://10.27.53.32:6443
KubeDNS is running at https://10.27.53.32:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns

查看详细node信息：

root@wudang:~# kubectl describe node/shaolin

Name:            shaolin
Role:
Labels:            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=shaolin
            node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:            node-role.kubernetes.io/master:NoSchedule

root@wudang:~# kubectl describe node/wudang

Name:            wudang
Role:
Labels:            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=wudang
            node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:            <none>

我们看到，在Taints属性里，shaolin node的值为 node-role.kubernetes.io/master:NoSchedule，而wudang node的为空。初步猜测这就是wudang被分配pod的原因了。

我们设置wudang node的Taints属性：

root@wudang:~# kubectl taint nodes wudang node-role.kubernetes.io/master=:NoSchedule
node "wudang" tainted

root@wudang:~# kubectl describe node/wudang|more
Name:            wudang
Role:
Labels:            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=wudang
            node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:            node-role.kubernetes.io/master:NoSchedule

再创建nginx deployment:

root@wudang:~/demo# pods
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default my-nginx-2267614806-hmz5d 1/1 Running 0 14s 172.32.64.0 emei
default my-nginx-2267614806-kkt79 1/1 Running 0 14s 172.32.64.1 emei

发现pod全部分配到emei上了！

接下来按同样操作对emei的taints属性进行设置，这里就不赘述了。

到目前为止，整个k8s cluster的状态如下示意图：

七、第六步：Load Balance

Kubernetes HA cluster的建立得益于kube-apiserver的无状态，按照最终目标，在三个kube-apiserver的前面是要假设一个负载均衡器的。考虑到apiserver对外通过https暴露服务，在七层做lb需要将证书配置在lb上，这改动较大；这里我们用四层lb。在这里，我们仅是搭建一个简易的demo性质的基于nginx的四层lb，在生产环境，如果你有硬件lb或者你所在的cloud provider提供类似lb服务，可以直接使用。

演示方便起见，我直接在emei上安装一个nginx（注意一定要安装支持–with-stream支持的nginx，可以通过-V查看）:

root@emei:~# nginx -V
nginx version: nginx/1.10.3 (Ubuntu)
built with OpenSSL 1.0.2g  1 Mar 2016
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --with-stream --with-stream_ssl_module --with-mail --with-mail_ssl_module --with-threads

我这里直接修改nginx的默认配置文件：/etc/nginx/nginx.conf，添加如下配置：

// /etc/nginx/nginx.conf
... ...
stream {
    upstream apiserver {
        server 10.27.53.32:6443 weight=5 max_fails=3 fail_timeout=30s;
        server 10.24.138.208:6443 weight=5 max_fails=3 fail_timeout=30s;
        server 10.27.52.72:6443 weight=5 max_fails=3 fail_timeout=30s;
    }

    server {
        listen 8443;
        proxy_connect_timeout 1s;
        proxy_timeout 3s;
        proxy_pass apiserver;
    }
}
... ...

nginx -s reload后，配置生效！

我们用wudang上的kubectl来访问一下lb，我们先来做一下配置

root@wudang:~# cp /etc/kubernetes/admin.conf ./
root@wudang:~# mv admin.conf admin-lb.conf
root@wudang:~# vi admin-lb.conf

修改admin-lb.conf中的：
server: https://10.27.52.72:8443

export KUBECONFIG=~/admin-lb.conf

执行下面命令：

root@wudang:~# kubectl get pods -n kube-system
Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.27.52.72
root@wudang:~# kubectl get pods -n kube-system
Unable to connect to the server: x509: certificate is valid for 10.24.138.208, not 10.27.52.72

可以看到上述两个请求被lb分别转到了shaolin和wudang两个node的apiserver上，客户端在校验server端发送的证书时认为server端”有诈“，于是报了错！怎么解决呢？在上面我们为每个apiserver生成apiserver.crt时，我们在subject alternative name值中填写了多个域名，我们用域名来作为client端访问的目的地址，再来看看：

修改~/admin-lb.conf中的：

server: https://kubernetes.default.svc:8443

在wudang node的/etc/hosts中添加：

10.27.52.72 kubernetes.default.svc

再访问集群：

root@wudang:~# kubectl get pods -n kube-system
NAME                              READY     STATUS    RESTARTS   AGE
etcd-emei                         1/1       Running   0          1d
etcd-shaolin                      1/1       Running   0          1d
etcd-wudang                       1/1       Running   0          4d
kube-apiserver-emei               1/1       Running   0          1d
... ...

这里只是一个demo，在您自己的环境里如何将lb与apiserver配合在一起，方法有很多种，需要根据实际情况具体确定。

到目前为止，整个k8s cluster的状态如下示意图：

八、第七步：kube-proxy配置修改

kube-proxy是一个由一个daemonset创建的：

root@wudang:~# kubectl get ds -n kube-system
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
kube-proxy   3         3         3         3            3           <none>          5d

并且kube-proxy的配置是由一个configmap提供的，并未在外部留有修改的口，比如类似kube-scheduler.yaml或.conf那样：

root@shaolin:~# kubectl get configmap -n kube-system
NAME                                 DATA      AGE
kube-proxy                           1         5d

root@shaolin:~# kubectl get configmap/kube-proxy -n kube-system -o yaml
apiVersion: v1
data:
  kubeconfig.conf: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://10.27.53.32:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
  creationTimestamp: 2017-05-10T01:48:28Z
  labels:
    app: kube-proxy
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "81"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kube-proxy
  uid: c34f7d5f-3522-11e7-8f77-00163e000c7f

在这个默认的configmap中，kube-proxy连接的cluster的server地址硬编码为 https://10.27.53.32:6443，即shaolin node上apiserver的公共接口地址。这样一旦shaolin node宕掉了，其他node上的kube-proxy将无法连接到apiserver进行正常操作。而kube-proxy pod自身又是使用的是host network，因此我们需要将server地址配置为lb的地址，这样保证各node上kube-proxy的高可用。

我们根据上述输出的configmap的内容进行修改，并更新kube-proxy-configmap的内容：

root@shaolin:~# kubectl get configmap/kube-proxy -n kube-system -o yaml > kube-proxy-configmap.yaml

修改kube-proxy-configmap.yaml中的server为：

server: https://kubernetes.default.svc:6443

保存并更新configmap: kube-proxy：

root@shaolin:~# kubectl apply -f kube-proxy-configmap.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap "kube-proxy" configured

root@shaolin:~# kubectl get configmap/kube-proxy -n kube-system -o yaml
apiVersion: v1
data:
  kubeconfig.conf: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://kubernetes.default.svc:6443
      name: default
... ...

重启kube-proxy(kubectl delete pods/kube-proxy-xxx -n kube-system)后，查看kube-proxy的日志：

root@shaolin:~# kubectl logs -f kube-proxy-h5sg8 -n kube-system
I0515 13:57:03.526032       1 server.go:225] Using iptables Proxier.
W0515 13:57:03.621532       1 proxier.go:298] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0515 13:57:03.621578       1 server.go:249] Tearing down userspace rules.
I0515 13:57:03.738015       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0515 13:57:03.741824       1 conntrack.go:66] Setting conntrack hashsize to 32768
I0515 13:57:03.742555       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0515 13:57:03.742731       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600

九、小结

到这里，我们在第一部分中的最终思路方案已经实现了。不过这两篇文章对kubernetes ha cluster的打造还仅限于探索阶段，可能还有一些深层次的问题没有暴露出来，因此不建议在生产环境中采用。kubeadm在后续的版本中必然加入对k8s ha cluster的支持，那个时候，搭建一套可用于生产环境的HA cluster将不再这么麻烦了！

© 2017, bigwhite. 版权所有.

No related posts.