分类 技术志 下的文章

初窥dep

Go语言程序组织和构建的基本单元是Package,但Go语言官方却没有提供一款“像样的”Package Management Tool(包管理工具)。随着Go语言在全球范围内应用的愈加广泛,缺少官方包管理工具这一问题变得日益突出。

2016年GopherCon大会后,在Go官方的组织下,一个旨在改善Go包管理的commitee成立了,共同应对Go在package management上遇到的各种问题。经过各种脑洞和讨论后,该commitee在若干月后发布了“Package Management Proposal”,并启动了最有可能被接纳为官方包管理工具的项目dep的设计和开发。2017年年初,dep项目正式对外开放。截至目前,dep发布了v0.1.0版本,并处于alpha测试阶段。

可以说,dep的进展还是蛮快的。按照dep官方说法,dep目前的manifest和lock文件格式已经stable,并保证向后兼容。同时,dep实现了“自举”,即dep使用自己作为自己的包管理工具。由于dep的“特殊身份”,虽然dep离成熟尚远,但dep的进展也吸引了诸多gopher的目光,很多组织已经开始将package management tool迁移为dep,为dep进行早期测试。

这里,我也打算“尝尝鲜”,在本篇文章中和大家一起窥探和试用一下dep。

一、Go包管理的演进历史

1、go get

在管窥dep之前,我们先来简单看看Go语言包管理的演进历史。首当其冲的就是go get。

Go语言新手在初次接触Go语言时会感觉到Go语言的package获取真的是很方便:只需一行go get xxx,github.com上的大量go package就可以随你取用。 但随着对Go语言使用的深入,人们会发现go get给我们带来方便的同时,也带来了不少的麻烦。go get本质上是githg等这些vcs工具的高级wrapper。对于使用git的go package来说,go get的实质就是将package git clone到本地的特定目录下($GOPATH/src),同时go get可以自动解析包的依赖,并自动下载相关依赖包。

go get机制的设计很大程度上源于Google公司内部的单一root的代码仓库的开发模式,并且似乎google内部各个project/repository的master分支上的代码都是被认为stable的,因此go get仅仅支持获取master branch上的latest代码,没有指定version、branch或revision的能力。而在Google公司以外的世界里,这样的做法会给gopher带来不便:依赖的第三方包总是在变。一旦第三方包提交了无法正常build或接口不兼容的代码,依赖方立即就会受到影响。

而gopher们又恰恰希望自己项目所依赖的第三方包能受到自己的控制,而不是随意变化。这样,godepgbglide等一批第三方包管理工具出现了。

以应用最为广泛的godep为例。为了能让第三方依赖包“稳定下来”,实现项目的reproduceble build,godep将项目当前依赖包的版本信息记录在Godeps/Godeps.json中,并将依赖包的相关版本存放在Godeps/_workspace中。在编译时(godep go build)godep通过临时修改GOPATH环境变量的方法让go编译器使用缓存在Godeps/_workspace下的项目依赖的特定版本的第三方包,这样保证了项目不再受制于依赖的第三方包的master branch上的latest代码的变动了。

不过,godep的“版本管理”本质上是通过缓存第三方库的某个revision的快照实现的,这种方式依然让人感觉难于管理。同时,通过对GOPATH的“偷梁换柱”的方式实现使用Godeps/_workspace中的第三方库的快照进行编译也无法兼容Go原生编译器,必须使用godep go xxx来进行。

为此,Go进一步引入vendor机制减少gopher在包管理问题上的心智负担

2、vendor机制

Go team也一直在关注Go语言包依赖的问题,尤其是在Go 1.5实现自举的情况下,官方同样在1.5版本中推出了vendor机制。vendor机制是Russ CoxGo 1.5发布前期以一个experiment feature身份紧急加入到go中的(go 1.6脱离experiment身份)。vendor标准化了项目依赖的第三方库的存放位置(不再需要Godeps/_workspace了),同时也无需对GOPATH环境变量进行“偷梁换柱”了,go compiler原生优先感知和使用vendor下缓存的第三方包。

不过即便有了vendor的支持,vendor内第三方依赖包的代码的管理依旧是不规范的,要么是手动的,要么是借助godep这样的第三方包管理工具。目前自举后的Go代码本身也引入了vendor,不过go项目自身对vendor中代码的管理方式也是手动更新,Go自身并未使用任何第三方的包管理工具。

题外话:作为一门语言的标准库,应该是使用这门语言的开发者所使用的所有lib依赖的根依赖。但在go中,go标准库居然还要依赖golang.org/x/目录下的包,既然能被std lib依赖,那么说明其已经成熟,那为何不把x内的stable的库挪到std lib中呢?这点着实让人有些不解。

~/.bin/go18/src/vendor/golang_org/x]$ls
crypto/    net/    text/

从Go官方角度出发,官方go包依赖的解决方案的下一步就应该是解决对vendor下的第三方包如何进行管理的问题:依赖包的分析、记录和获取等,进而实现项目的reproducible build。dep就是用来做这事儿的。

二、dep简介

go package management commitee的牵头人物是微服务框架go-kit作者Peter Bourgon,但当前主导dep开发的是sam boyer,sam也是dep底层包依赖分析引擎-gps的作者。

和其他一些第三方Go包管理工具有所不同,dep在进行active dev前是经过commitee深思熟虑的,包括:featuresuser story等都在事前做了初步设计。如果你拜读这些文档,你可能会觉得解决包依赖问题,还是蛮复杂的。不过,对于这些工具的使用者来说,我们面对的是一些十分简化的交互接口。

1、安装dep

dep是标准的go cli程序,执行一条命令即完成安装:

# go get -u github.com/golang/dep/cmd/dep

# dep help
dep is a tool for managing dependencies for Go projects

Usage: dep <command>

Commands:

  init    Initialize a new project with manifest and lock files
  status  Report the status of the project's dependencies
  ensure  Ensure a dependency is safely vendored in the project
  prune   Prune the vendor tree of unused packages

Examples:
  dep init                          set up a new project
  dep ensure                        install the project's dependencies
  dep ensure -update                update the locked versions of all dependencies
  dep ensure github.com/pkg/errors  add a dependency to the project

Use "dep help [command]" for more information about a command.

在我的测试环境中,go的版本为1.8;dep的版本为commit d31c621c3381b9bebc7c10b1ac7849a96c21f2c3。

注意:由于dep还在active dev过程中且处于alpha测试阶段,因此本文中执行的dep命令、命令行为以及输出结果在后续dep版本中很可能会有变动,甚至是很大变动。

2、dep一般工作流

安装好dep后,我们就来看看使用dep的一般工作流。我们首先准备一个demo程序:

//depdemo/main.go
package main

import (
    "net/http"

    "go.uber.org/zap"

    "github.com/beego/mux"
)

func main() {
    logger, _ := zap.NewProduction()
    defer logger.Sync()
    sugar := logger.Sugar()

    mx := mux.New()
    mx.Handler("GET", "/", http.FileServer(http.Dir(".")))
    sugar.Fatal(http.ListenAndServe("127.0.0.1:8080", mx))
}

a) dep init

如果一个项目要使用dep进行包管理,那么首先需要在这个项目的根下执行dep init。在这里,我们对depdemo进行dep改造。

在depdemo目录下,执行dep init:

# dep init -v
Searching GOPATH for projects...
  Using master as constraint for direct dep github.com/beego/mux
  Locking in master (626af65) for direct dep github.com/beego/mux
Following dependencies were not found in GOPATH. Dep will use the most recent versions of these projects.
  go.uber.org/zap
Root project is "github.com/bigwhite/experiments/depdemo"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
(0)   ✓ select (root)
(1)    ? attempt github.com/beego/mux with 1 pkgs; at least 1 versions to try
(1)        try github.com/beego/mux@master
(1)    ✓ select github.com/beego/mux@master w/1 pkgs
(2)    ? attempt go.uber.org/zap with 1 pkgs; 12 versions to try
(2)        try go.uber.org/zap@v1.4.0
(2)    ✓ select go.uber.org/zap@v1.4.0 w/7 pkgs
(3)    ? attempt go.uber.org/atomic with 1 pkgs; 6 versions to try
(3)        try go.uber.org/atomic@v1.2.0
(3)    ✓ select go.uber.org/atomic@v1.2.0 w/1 pkgs
  ✓ found solution with 9 packages from 3 projects

Solver wall times by segment:
     b-source-exists: 1.090607387s
  b-deduce-proj-root: 288.126482ms
         b-list-pkgs: 131.059753ms
              b-gmal: 114.716587ms
         select-atom:    337.787µs
             satisfy:    298.743µs
         select-root:    292.889µs
            new-atom:    257.256µs
     b-list-versions:     42.408µs
               other:     22.307µs

  TOTAL: 1.625761599s

当前阶段,dep init命令的执行效率的确不高,因此需要你耐心的等待一会儿。如果你的project依赖的外部包很多,那么等待的时间可能会很长。并且由于dep会下载依赖包,对于国内的朋友来说,一旦下载qiang外的包,那么dep可能会“阻塞”在那里!

dep init大致会做这么几件事:

  • 利用gps分析当前代码包中的包依赖关系;
  • 将分析出的项目包的直接依赖(即main.go显式import的第三方包,direct dependency)约束(constraint)写入项目根目录下的Gopkg.toml文件中;
  • 将项目依赖的所有第三方包(包括直接依赖和传递依赖transitive dependency)在满足Gopkg.toml中约束范围内的最新version/branch/revision信息写入Gopkg.lock文件中;
  • 创建root vendor目录,并且以Gopkg.lock为输入,将其中的包(精确checkout 到revision)下载到项目root vendor下面。

执行完dep init后,dep会在当前目录下生成若干文件:

├── Gopkg.lock
├── Gopkg.toml
├── main.go
└── vendor/

我们逐一来看一下:

Gopkg.toml:

[[constraint]]
  branch = "master"
  name = "github.com/beego/mux"

[[constraint]]
  name = "go.uber.org/zap"
  version = "1.4.0"

Gopkg.toml记录了depdemo/main.go的两个direct dependency:mux和zap。通过gps的分析(可以参见上面init执行时输出的详细分析过程日志),dep确定的依赖版本约束为:mux的master分支、zap的1.4.0 version。

生成的Gopkg.lock中则记录了depdemo/main.go在上述约束下的所有依赖的可用的最新版本:

Gopkg.lock:

[[projects]]
  branch = "master"
  name = "github.com/beego/mux"
  packages = ["."]
  revision = "626af652714cc0092f492644e298e5f3ac7db31a"

[[projects]]
  name = "go.uber.org/atomic"
  packages = ["."]
  revision = "4e336646b2ef9fc6e47be8e21594178f98e5ebcf"
  version = "v1.2.0"

[[projects]]
  name = "go.uber.org/zap"
  packages = [".","buffer","internal/bufferpool","internal/color","internal/exit","internal/multierror","zapcore"]
  revision = "fab453050a7a08c35f31fc5fff6f2dbd962285ab"
  version = "v1.4.0"

[solve-meta]
  analyzer-name = "dep"
  analyzer-version = 1
  inputs-digest = "77d32776fdc88e1025460023bef70534c5457bdc89b817c9bab2b2cf7cccb22f"
  solver-name = "gps-cdcl"
  solver-version = 1

vendor目录下,则是lock文件中各个依赖包的本地clone:

# tree -L 2 vendor
vendor
├── github.com
│   └── beego
└── go.uber.org
    ├── atomic
    └── zap

至此,dep init完毕,相关依赖包也已经被vendor,你可以使用go build/install进行程序构建了。

b)、提交Gopkg.toml和Gopkg.lock

如果你对dep自动分析出来的各种约束和依赖的版本没有异议,那么这里就可以将Gopkg.toml和Gopkg.lock作为项目源码的一部分提交到代码库中了。这样其他人在下载了你的代码后,可以通过dep直接下载lock文件中的第三方包版本,并存在vendor里。这样就使得无论在何处,项目构建的依赖库理论上都是一致的,实现reproduceable build。

是否需要提交vendor下的依赖包代码到代码仓库?这取决于你。提交vendor的好处是即便没有dep,也可以实现真正的reproduceable build。但vendor的提交会让你的代码库变得异常庞大,且更新vendor时,大量的diff会影响到你对代码的review。下面的内容我们以不提交vendor为前提。

c)、dep ensure

现在我们的depdemo已经加入了Gopkg.toml和Gopkg.lock。这时,如果你将depdemo clone到你的本地,你还无法进行reproduceable build,因为这时vendor还不存在。这时我们需要执行下面命令来根据Gopkg.toml和Gopkg.lock中的数据构建vendor目录和同步里面的包:

# dep ensure

# ls -F
Gopkg.lock  Gopkg.toml  main.go  vendor/

ensure成功后,你就可以进行reproduceable build了。

我们可以通过dep status查看当前的依赖情况(包括direct and transitive dependency):

# dep status
PROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USED
github.com/beego/mux  branch master  branch master  626af65   626af65  1
go.uber.org/atomic    *              v1.2.0         4e33664   4e33664  1
go.uber.org/zap       ^1.4.0         v1.4.0         fab4530   fab4530  7

d) 指定约束

dep init生成的Gopkg.toml中的约束是否是我们预期的呢?这个还真不一定。比如:我们将对zap的约束手工改为1.3.0:

//Gopkg.toml
... ...

[[constraint]]
  name = "go.uber.org/zap"
  version = "<=1.3.0"

执行dep ensure后,查看status:

# dep status
PROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USED
github.com/beego/mux  branch master  branch master  626af65   626af65  1
go.uber.org/atomic    *              v1.2.0         4e33664   4e33664  1
go.uber.org/zap       <=1.3.0         v1.4.0         fab4530   fab4530  7

不过,此时Gopkg.lock中的zap version依旧是v1.4.0,并没有修改。要想更新lock和vendor下的数据,我们需要给ensure加上一个-update参数:

# dep ensure -update

# git diff Gopkg.lock
diff --git a/depdemo/Gopkg.lock b/depdemo/Gopkg.lock
index fce53dc..7fe3640 100644
--- a/depdemo/Gopkg.lock
+++ b/depdemo/Gopkg.lock
@@ -16,12 +16,12 @@
 [[projects]]
   name = "go.uber.org/zap"
   packages = [".","buffer","internal/bufferpool","internal/color","internal/exit","internal/multierror","zapcore"]
-  revision = "fab453050a7a08c35f31fc5fff6f2dbd962285ab"
-  version = "v1.4.0"
+  revision = "6a4e056f2cc954cfec3581729e758909604b3f76"
+  version = "v1.3.0"

 [solve-meta]
   analyzer-name = "dep"
   analyzer-version = 1
-  inputs-digest = "77d32776fdc88e1025460023bef70534c5457bdc89b817c9bab2b2cf7cccb22f"
+  inputs-digest = "b09c1497771f6fe7cdfcf61ab1a026ccc909f4801c08f2c25f186f93f14526b0"
   solver-name = "gps-cdcl"
   solver-version = 1

-update让dep ensure尝试去保证并同步Gopkg.lock和vendor目录下的数据,将Gopkg.lock下的zap的version改为Gopkg.toml下约束的最大值,即v1.3.0,同时更新vendor下的zap代码。

e) 指定依赖

我们也可以直接更新dependency,这将影响Gopkg.lock和vendor下的数据,但Gopkg.toml不会被修改:

# dep ensure 'go.uber.org/zap@<1.4.0'

# git diff
diff --git a/depdemo/Gopkg.lock b/depdemo/Gopkg.lock
index fce53dc..3b17b9b 100644
--- a/depdemo/Gopkg.lock
+++ b/depdemo/Gopkg.lock
@@ -16,12 +16,12 @@
 [[projects]]
   name = "go.uber.org/zap"
   packages = [".","buffer","internal/bufferpool","internal/color","internal/exit","internal/multierror","zapcore"]

-  revision = "fab453050a7a08c35f31fc5fff6f2dbd962285ab"
-  version = "v1.4.0"
+  revision = "6a4e056f2cc954cfec3581729e758909604b3f76"
+  version = "v1.3.0"

 [solve-meta]
   analyzer-name = "dep"
   analyzer-version = 1
-  inputs-digest = "77d32776fdc88e1025460023bef70534c5457bdc89b817c9bab2b2cf7cccb22f"
+  inputs-digest = "3307cd7d5942d333c4263fddda66549ac802743402fe350c0403eb3657b33b0b"
   solver-name = "gps-cdcl"
   solver-version = 1

这种情况下会出现Gopkg.lock中的version不满足Gopkg.toml中约束的情况。这里也让我比较困惑!

三、dep探索

上面的dep使用基本工作流完全可以满足日常包管理的需求了。但对于喜欢求甚解的我来说,必要要探索一下dep背后的行为和原理。

1、dep init的两种不同结果

我们回到depdemo的初始状态,即起点:尚未生成dep metadata file的时刻。我们在两种情况下,分别执行dep init:

  • $GOPATH/src下没有go.uber.org/zap
# dep init -v
Searching GOPATH for projects...
  Using master as constraint for direct dep github.com/beego/mux
  Locking in master (626af65) for direct dep github.com/beego/mux
Following dependencies were not found in GOPATH. Dep will use the most recent versions of these projects.
  go.uber.org/zap
Root project is "github.com/bigwhite/experiments/depdemo"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
... ...

# dep status
PROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USED
github.com/beego/mux  branch master  branch master  626af65   626af65  1
go.uber.org/atomic    *              v1.2.0         4e33664   4e33664  1
go.uber.org/zap       ^1.4.0         v1.4.0         fab4530   fab4530  7

  • $GOPATH/src下存在go.uber.org/zap
# dep init -v
Searching GOPATH for projects...
  Using master as constraint for direct dep github.com/beego/mux
  Locking in master (626af65) for direct dep github.com/beego/mux
  Using master as constraint for direct dep go.uber.org/zap
  Locking in master (b33459c) for direct dep go.uber.org/zap
  Locking in master (908889c) for transitive dep go.uber.org/atomic
Root project is "github.com/bigwhite/experiments/depdemo"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
... ...

# dep status
PROJECT               CONSTRAINT     VERSION        REVISION  LATEST   PKGS USED
github.com/beego/mux  branch master  branch master  626af65   626af65  1
go.uber.org/atomic    *              branch master  908889c   4e33664  1
go.uber.org/zap       branch master  branch master  b33459c   b33459c  7

不知道大家发现两种情况下生成的结果的异同与否。我们只看两个dep status输出中的zap一行:

go.uber.org/zap       ^1.4.0         v1.4.0         fab4530   fab4530  7

vs.

go.uber.org/zap       branch master  branch master  b33459c   b33459c  7

dep自动分析后得到截然不同的两个结果。

第一种情况,我们称之为dep init的network mode,即dep发现本地GOPATH下面没有zap,于是dep init通过network到upstream上查找zap,并“Dep will use the most recent versions of these projects”,即v1.4.0版本。

第二种情况,我们称之为dep init的GOPATH mode, 即dep发现本地GOPATH下面存在zap,于是dep init认定“Using master as constraint for direct dep go.uber.org/zap”,即master branch。

至于为何GOPATH mode下,dep init会选择master,我个人猜测是因为dep觉得既然你本地有zap,那很大可能zap master的稳定性是被你所接受了的。在“dep: updated command spec”中,似乎dep init打算通过增加一个-gopath的flag来区分两种工作模式,并将network mode作为默认工作mode。但目前我所使用的dep版本还没有实现这个功能,其默认工作方式依旧是先GOPATH mode,如果没有找到依赖包的存在,则针对该包实施network mode。

从这里也可以看得出来,对于dep init 输出的约束,你最好还是检视一下,看是否能接受,否则就通过上面提到的“指定约束”来更正dep的输出。

2、dep对项目的依赖包的cache

在进行上面的试验中,我们发现:在本地GOPATH/src下面没有zap的情况下,dep似乎是直接将zap get到本地vendor目录的,而不是先get到GOPATH/src下,在copy到vendor中。事实是什么样的呢?dep的确没有操作GOPATH/src目录,因为那是共享的。dep在$GOPATH/pkg/dep/sources下留了一块“自留地”,用于cache所有从network上下载的依赖包:

# ls -F $GOPATH/pkg/dep/sources/
https---github.com-beego-mux/  https---github.com-uber--go-atomic/  https---github.com-uber--go-zap/

# ls -aF /root/go/pkg/dep/sources/https---github.com-uber--go-zap
./             buffer/            config_test.go   field.go       .gitignore      http_handler.go       LICENSE.txt           options.go          sugar.go       writer.go
../            CHANGELOG.md       CONTRIBUTING.md  field_test.go  glide.lock      http_handler_test.go  logger_bench_test.go  README.md           sugar_test.go  writer_test.go
array.go       check_license.sh*  doc.go           flag.go        glide.yaml      internal/             logger.go             .readme.tmpl        time.go        zapcore/
array_test.go  common_test.go     encoder.go       flag_test.go   global.go       level.go              logger_test.go        stacktrace.go       time_test.go   zapgrpc/
benchmarks/    config.go          encoder_test.go  .git/          global_test.go  level_test.go         Makefile              stacktrace_test.go  .travis.yml    zaptest/

dep对于依赖包的所以git请求均在这个缓存目录下进行。

3、 vendor flatten平坦化

go在1.5加入vendor机制时,是考虑到“钻石形依赖”中存在同一个依赖包的不同版本的。我们来看看dep是否支持这一点。我们设计了一个试验:

img{512x368}

我们建立一个这样的“钻石形”试验环境,foo依赖ab两个包,而a、b两个包分别依赖f的不同版本(通过在a、b中的Gopkg.toml声明这种约束,见图中标注)。

下面是foo项目下面的main.go:

// foo/main.go

package main

import "bitbucket.org/bigwhite/b"
import "bitbucket.org/bigwhite/a"

func main() {
    a.CallA()
    b.CallB()
}

未引入dep前,我们来运行一下该代码:

$go run main.go
call A: master branch
   --> call F:
    call F: v1.1.0
   --> call F end
call B: master branch
   --> call F:
    call F: v2.0.1
   --> call F end

可以看到同样是f包的输出,由于a、b分别依赖f的不同版本,因此输出不同。

我们对foo进行一个dep 分析,看看dep给了我们什么结果:

$dep init -v
Searching GOPATH for projects...
  Using master as constraint for direct dep bitbucket.org/bigwhite/a
  Locking in master (9122a5d) for direct dep bitbucket.org/bigwhite/a
  Using master as constraint for direct dep bitbucket.org/bigwhite/b
  Locking in master (2415845) for direct dep bitbucket.org/bigwhite/b
  Locking in master (971460c) for transitive dep bitbucket.org/bigwhite/f
Root project is "Foo"
 1 transitively valid internal packages
 2 external packages imported from 2 projects
 ... ...

No versions of bitbucket.org/bigwhite/b met constraints:
    master: Could not introduce bitbucket.org/bigwhite/b@master, as it has a dependency on bitbucket.org/bigwhite/f with constraint ^2.0.0, which has no overlap with existing constraint ^1.1.0 from bitbucket.org/bigwhite/a@master
    v2.0.0: Could not introduce bitbucket.org/bigwhite/b@v2.0.0, as it is not allowed by constraint master from project Foo.
    v1.0.0: Could not introduce bitbucket.org/bigwhite/b@v1.0.0, as it is not allowed by constraint master from project Foo.
    master: Could not introduce bitbucket.org/bigwhite/b@master, as it has a dependency on bitbucket.org/bigwhite/f with constraint ^2.0.0, which has no overlap with existing constraint ^1.1.0 from bitbucket.org/bigwhite/a@master

dep init运行失败。由于a依赖的f@^1.1.0和b依赖的f@^2.0.0两个约束之间没有交集,无法调和,dep无法solve这个依赖,于是init failed!

但失败背后还有一层原因,那就是dep的设计要求flatten vendor,即使用dep的项目只能有一个root vendor,所以直接依赖或传递依赖的包中包含vendor的,vendor目录也都会被strip掉。这样一旦依赖包中存在带有冲突的约束,那么dep init必将失败。

四、小结

dep一个重要feature就是支持semver 2.0规范,不过semver的规则好多,不是这里能说清楚的,大家可以到semver官方站细读规则,或者在npm semver calculator这个站点直观感受semver规则带来的变化。

dep试验告一段落。从目前来看,dep已经进入可用阶段,建议有条件的童鞋能积极的使用dep,并为dep进行前期测试,发现问题提issue,为dep的快速完善出出力。

depdemo的代码在这里;a, b,f包的代码在这里这里这里

五、参考资料


微博:@tonybai_cn
微信公众号:iamtonybai
github.com: https://github.com/bigwhite

一步步打造基于Kubeadm的高可用Kubernetes集群-第二部分

续接上文

五、第三步:启动emei、wudang上的apiserver

跨三个node的etcd cluster已经建成并完成了数据同步,下面进行ha cluster改造的重要一步:启动wudang、emei上的apiserver

1、启动emei、wudang上的apiserver

以shaolin node上的/etc/kubernetes/manifests/kube-apiserver.yaml为副本,制作emei、wudang上的kube-apiserver.yaml:

唯一需要变动的就是- --advertise-address这个option的值:

wudang:

- --advertise-address=10.24.138.208

emei:

- --advertise-address=10.27.52.72

在各自node上将kube-apiserver.yaml放入/etc/kubernetes/manifests中,各自node上的kubelet将会启动kube-apiserver并且各个apiserver默认连接本节点的etcd:

root@emei:~# pods
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
... ...
kube-system   kube-apiserver-emei               1/1       Running   0          1d        10.27.52.72     emei
kube-system   kube-apiserver-shaolin            1/1       Running   0          1d        10.27.53.32     shaolin
kube-system   kube-apiserver-wudang             1/1       Running   0          2d        10.24.138.208   wudang

2、将emei、wudang上的kubelet改为连接自己所在节点的apiserver

所有apiserver都启动了。wudang、emei上的kubelet也应该连接自己节点的apiserver了!修改各自的/etc/kubernetes/kubelet.conf,修改server配置项:

wudang:

server: https://10.24.138.208:6443

emei:

server: https://10.27.52.72:6443

各自重启kubelet:

以wudang为例:

root@wudang:~# systemctl daemon-reload
root@wudang:~# systemctl restart kubelet

不过,问题出现了!查看重启的kubelet日志:

root@wudang:~# journalctl -u kubelet -f
-- Logs begin at Mon 2017-05-08 15:12:01 CST. --
May 11 14:33:27 wudang kubelet[8794]: I0511 14:33:27.919223    8794 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
May 11 14:33:27 wudang kubelet[8794]: I0511 14:33:27.921166    8794 kubelet_node_status.go:77] Attempting to register node wudang
May 11 14:33:27 wudang kubelet[8794]: E0511 14:33:27.926865    8794 kubelet_node_status.go:101] Unable to register node "wudang" with API server: Post https://10.24.138.208:6443/api/v1/nodes: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208
May 11 14:33:28 wudang kubelet[8794]: E0511 14:33:28.283258    8794 event.go:208] Unable to write event: 'Post https://10.24.138.208:6443/api/v1/namespaces/default/events: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208' (may retry after sleeping)
May 11 14:33:28 wudang kubelet[8794]: E0511 14:33:28.499209    8794 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/kubelet.go:390: Failed to list *v1.Node: Get https://10.24.138.208:6443/api/v1/nodes?fieldSelector=metadata.name%3Dwudang&resourceVersion=0: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208
May 11 14:33:28 wudang kubelet[8794]: E0511 14:33:28.504593    8794 reflector.go:190] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://10.24.138.208:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dwudang&resourceVersion=0: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.24.138.208

从错误日志判断来看,似乎是wudang上的kubelet在与同一节点上的kube-apiserver通信过程中,发现这个apiserver返回的tls证书是属于10.27.53.32的,即shaolin node上的apiserver的,而不是wudang node上的apiserver的,于是报了错!问题的原因很明了,因为Wudang上的kube-apiserver用的apiserver.crt的确是从shaolin node上copy过来的。也就是说要解决这个问题,我们需要为wudang、emei两个node上的apiserver各自生成自己的数字证书。

我们先来查看一下shaolin上的apiserver.crt内容是什么样子的:

root@shaolin:/etc/kubernetes/pki# openssl x509 -noout -text -in apiserver.crt

Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes

Subject: CN=kube-apiserver

X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Alternative Name:
                DNS:shaolin, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.27.53.32

我们看到证书使用到了x509v3的扩展功能:subject alternative name,并且指定了多个value。我们为wudang、emei生成的apiserver.crt也应该如此。如何做呢?好在我们有整个集群的ca.key和ca.crt,可以用来签署证书请求。以wudang node为例,我们来为wudang node上的apiserver生成apiserver-wudang.key和apiserver-wudang.crt:

//生成2048位的密钥对
root@wudang:~# openssl genrsa -out apiserver-wudang.key 2048

//生成证书签署请求文件
root@wudang:~# openssl req -new -key apiserver-wudang.key -subj "/CN=kube-apiserver," -out apiserver-wudang.csr

// 编辑apiserver-wudang.ext文件,内容如下:
subjectAltName = DNS:wudang,DNS:kubernetes,DNS:kubernetes.default,DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP:10.96.0.1, IP:10.24.138.208

// 使用ca.key和ca.crt签署上述请求
root@wudang:~# openssl x509 -req -in apiserver-wudang.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out apiserver-wudang.key.crt -days 365 -extfile apiserver-wudang.ext
Signature ok
subject=/CN=10.24.138.208
Getting CA Private Key

//查看新生成的证书:
root@wudang:~# openssl x509 -noout -text -in apiserver-wudang.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 16019625340257831745 (0xde51245f10ea0b41)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=kubernetes
        Validity
            Not Before: May 12 08:40:40 2017 GMT
            Not After : May 12 08:40:40 2018 GMT
        Subject: CN=kube-apiserver,
        Subject Public Key Info:
            ... ...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:wudang, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.96.0.1, IP Address:10.24.138.208

将apiserver-wudang.key和apiserver-wudang.crt放入/etc/kubernetes/pki目录下,修改kube-apiserver.yaml文件:

// /etc/kubernetes/pki
- --tls-cert-file=/etc/kubernetes/pki/apiserver-wudang.crt
- --tls-private-key-file=/etc/kubernetes/pki/apiserver-wudang.key

kube-apiserver重启后,再来查看kubelet日志,你会发现kubelet运行一切ok了。emei节点也要进行同样的操作。

至此,整个集群的状态示意图如下:

img{512x368}

六、第四步:启动emei、wudang上的kube-controller-manager和kube-scheduler

这一步我们只需要将shaolin node上的/etc/kubernetes/manifests中的kube-controller-manager.yaml和kube-scheduler.yaml拷贝到wudang、emei两个node的相应目录下即可:

root@emei:~/kubernetes-conf-shaolin/manifests# pods
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
... ...
kube-system   kube-controller-manager-emei      1/1       Running   0          8s        10.27.52.72     emei
kube-system   kube-controller-manager-shaolin   1/1       Running   3          1d        10.27.53.32     shaolin
kube-system   kube-controller-manager-wudang    1/1       Running   0          1m        10.24.138.208   wudang
... ...
kube-system   kube-scheduler-emei               1/1       Running   0          15s       10.27.52.72     emei
kube-system   kube-scheduler-shaolin            1/1       Running   3          1d        10.27.53.32     shaolin
kube-system   kube-scheduler-wudang             1/1       Running   0          3m        10.24.138.208   wudang
... ...

查看一下各个node下kcm和scheduler的日志:

root@wudang:~/demo# kubectl logs -f kube-controller-manager-emei -n kube-system
I0511 07:34:53.804831       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-controller-manager-wudang -n kube-system
I0511 07:33:20.725669       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-scheduler-emei -n kube-system
I0511 07:34:45.711032       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-scheduler-wudang -n kube-system
I0511 07:31:35.077090       1 leaderelection.go:179] attempting to acquire leader lease...

root@wudang:~/demo# kubectl logs -f kube-scheduler-shaolin -n kube-system

I0512 08:55:30.838806       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-v1dst", UID:"c075c6c7-36f0-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"166279", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-v1dst to emei
I0512 08:55:30.843104       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-drnzv", UID:"c075da9f-36f0-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"166278", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-drnzv to wudang
I0512 09:13:21.121864       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-ld1dr", UID:"3e73d350-36f3-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"168070", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-ld1dr to wudang
I0512 09:13:21.124295       1 event.go:217] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"my-nginx-2267614806-cmmkh", UID:"3e73c8b2-36f3-11e7-9c66-00163e000c7f", APIVersion:"v1", ResourceVersion:"168071", FieldPath:""}): type: 'Normal' reason: 'Scheduled' Successfully assigned my-nginx-2267614806-cmmkh to emei

可以看出,当前shaolin node上的kcm和scheduler是leader。

至此,整个集群的状态示意图如下:

img{512x368}

六、第五步:将wudang、emei设置为master node

我们试着在wudang节点上创建一个pod:

// run-my-nginx.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: my-nginx
spec:
  replicas: 2
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - name: my-nginx
        image: nginx:1.10.1
        ports:
        - containerPort: 80

发现pod居然被调度到了wudang、emei节点上了!

NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
default       my-nginx-2267614806-drnzv         1/1       Running   0          5s        172.32.192.1    wudang
default       my-nginx-2267614806-v1dst         1/1       Running   0          5s        172.32.64.0     emei

emei、wudang并没有执行taint,为何能承载workload? 查看当前cluster的node状态:

root@wudang:~# kubectl get node --show-labels
NAME      STATUS    AGE       VERSION   LABELS
emei      Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=emei
shaolin   Ready     2d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=shaolin,node-role.kubernetes.io/master=
wudang    Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=wudang

从label看到,status列并没有明确输出谁是master,这和1.5.1版本以前似乎不同。emei、wudang与shaolin唯一的不同就是shaolin有一个key: node-role.kubernetes.io/master。难道这个label是指示谁是master的?我们给wudang打上这个label:

root@wudang:~/demo# kubectl label node wudang node-role.kubernetes.io/master=
node "wudang" labeled
root@wudang:~/demo# kubectl get node --show-labels
NAME      STATUS    AGE       VERSION   LABELS
emei      Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=emei
shaolin   Ready     2d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=shaolin,node-role.kubernetes.io/master=
wudang    Ready     1d        v1.6.2    beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=wudang,node-role.kubernetes.io/master=

再创建nginx pod,我们发现pod依旧分配在wudang、emei两个node上:

NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
default       my-nginx-2267614806-cmmkh         1/1       Running   0          5s        172.32.64.0     emei
default       my-nginx-2267614806-ld1dr         1/1       Running   0          5s        172.32.192.1    wudang

我们进一步查看并对比相关信息:

查看clustre-info:

wuddang node:
root@wudang:~/demo# kubectl cluster-info
Kubernetes master is running at https://10.24.138.208:6443 //wudang node:
KubeDNS is running at https://10.24.138.208:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns

shaolin node:

root@shaolin:~/k8s-install/demo# kubectl cluster-info
Kubernetes master is running at https://10.27.53.32:6443
KubeDNS is running at https://10.27.53.32:6443/api/v1/proxy/namespaces/kube-system/services/kube-dns

查看详细node信息:

root@wudang:~# kubectl describe node/shaolin

Name:            shaolin
Role:
Labels:            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=shaolin
            node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:            node-role.kubernetes.io/master:NoSchedule

root@wudang:~# kubectl describe node/wudang

Name:            wudang
Role:
Labels:            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=wudang
            node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:            <none>

我们看到,在Taints属性里,shaolin node的值为 node-role.kubernetes.io/master:NoSchedule,而wudang node的为空。初步猜测这就是wudang被分配pod的原因了。

我们设置wudang node的Taints属性:

root@wudang:~# kubectl taint nodes wudang node-role.kubernetes.io/master=:NoSchedule
node "wudang" tainted

root@wudang:~# kubectl describe node/wudang|more
Name:            wudang
Role:
Labels:            beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=wudang
            node-role.kubernetes.io/master=
Annotations:        node.alpha.kubernetes.io/ttl=0
            volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:            node-role.kubernetes.io/master:NoSchedule

再创建nginx deployment:

root@wudang:~/demo# pods
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
default my-nginx-2267614806-hmz5d 1/1 Running 0 14s 172.32.64.0 emei
default my-nginx-2267614806-kkt79 1/1 Running 0 14s 172.32.64.1 emei

发现pod全部分配到emei上了!

接下来按同样操作对emei的taints属性进行设置,这里就不赘述了。

到目前为止,整个k8s cluster的状态如下示意图:
img{512x368}

七、第六步:Load Balance

Kubernetes HA cluster的建立得益于kube-apiserver的无状态,按照最终目标,在三个kube-apiserver的前面是要假设一个负载均衡器的。考虑到apiserver对外通过https暴露服务,在七层做lb需要将证书配置在lb上,这改动较大;这里我们用四层lb。在这里,我们仅是搭建一个简易的demo性质的基于nginx的四层lb,在生产环境,如果你有硬件lb或者你所在的cloud provider提供类似lb服务,可以直接使用。

演示方便起见,我直接在emei上安装一个nginx(注意一定要安装支持–with-stream支持的nginx,可以通过-V查看):

root@emei:~# nginx -V
nginx version: nginx/1.10.3 (Ubuntu)
built with OpenSSL 1.0.2g  1 Mar 2016
TLS SNI support enabled
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --with-stream --with-stream_ssl_module --with-mail --with-mail_ssl_module --with-threads

我这里直接修改nginx的默认配置文件:/etc/nginx/nginx.conf,添加如下配置:

// /etc/nginx/nginx.conf
... ...
stream {
    upstream apiserver {
        server 10.27.53.32:6443 weight=5 max_fails=3 fail_timeout=30s;
        server 10.24.138.208:6443 weight=5 max_fails=3 fail_timeout=30s;
        server 10.27.52.72:6443 weight=5 max_fails=3 fail_timeout=30s;
    }

    server {
        listen 8443;
        proxy_connect_timeout 1s;
        proxy_timeout 3s;
        proxy_pass apiserver;
    }
}
... ...

nginx -s reload后,配置生效!

我们用wudang上的kubectl来访问一下lb,我们先来做一下配置

root@wudang:~# cp /etc/kubernetes/admin.conf ./
root@wudang:~# mv admin.conf admin-lb.conf
root@wudang:~# vi admin-lb.conf

修改admin-lb.conf中的:
server: https://10.27.52.72:8443

export KUBECONFIG=~/admin-lb.conf

执行下面命令:

root@wudang:~# kubectl get pods -n kube-system
Unable to connect to the server: x509: certificate is valid for 10.96.0.1, 10.27.53.32, not 10.27.52.72
root@wudang:~# kubectl get pods -n kube-system
Unable to connect to the server: x509: certificate is valid for 10.24.138.208, not 10.27.52.72

可以看到上述两个请求被lb分别转到了shaolin和wudang两个node的apiserver上,客户端在校验server端发送的证书时认为server端”有诈“,于是报了错!怎么解决呢?在上面我们为每个apiserver生成apiserver.crt时,我们在subject alternative name值中填写了多个域名,我们用域名来作为client端访问的目的地址,再来看看:

修改~/admin-lb.conf中的:

server: https://kubernetes.default.svc:8443

在wudang node的/etc/hosts中添加:

10.27.52.72 kubernetes.default.svc

再访问集群:

root@wudang:~# kubectl get pods -n kube-system
NAME                              READY     STATUS    RESTARTS   AGE
etcd-emei                         1/1       Running   0          1d
etcd-shaolin                      1/1       Running   0          1d
etcd-wudang                       1/1       Running   0          4d
kube-apiserver-emei               1/1       Running   0          1d
... ...

这里只是一个demo,在您自己的环境里如何将lb与apiserver配合在一起,方法有很多种,需要根据实际情况具体确定。

到目前为止,整个k8s cluster的状态如下示意图:
img{512x368}

八、第七步:kube-proxy配置修改

kube-proxy是一个由一个daemonset创建的:

root@wudang:~# kubectl get ds -n kube-system
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE-SELECTOR   AGE
kube-proxy   3         3         3         3            3           <none>          5d

并且kube-proxy的配置是由一个configmap提供的,并未在外部留有修改的口,比如类似kube-scheduler.yaml或.conf那样:

root@shaolin:~# kubectl get configmap -n kube-system
NAME                                 DATA      AGE
kube-proxy                           1         5d

root@shaolin:~# kubectl get configmap/kube-proxy -n kube-system -o yaml
apiVersion: v1
data:
  kubeconfig.conf: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://10.27.53.32:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
  creationTimestamp: 2017-05-10T01:48:28Z
  labels:
    app: kube-proxy
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "81"
  selfLink: /api/v1/namespaces/kube-system/configmaps/kube-proxy
  uid: c34f7d5f-3522-11e7-8f77-00163e000c7f

在这个默认的configmap中,kube-proxy连接的cluster的server地址硬编码为 https://10.27.53.32:6443,即shaolin node上apiserver的公共接口地址。这样一旦shaolin node宕掉了,其他node上的kube-proxy将无法连接到apiserver进行正常操作。而kube-proxy pod自身又是使用的是host network,因此我们需要将server地址配置为lb的地址,这样保证各node上kube-proxy的高可用。

我们根据上述输出的configmap的内容进行修改,并更新kube-proxy-configmap的内容:

root@shaolin:~# kubectl get configmap/kube-proxy -n kube-system -o yaml > kube-proxy-configmap.yaml

修改kube-proxy-configmap.yaml中的server为:

server: https://kubernetes.default.svc:6443

保存并更新configmap: kube-proxy:

root@shaolin:~# kubectl apply -f kube-proxy-configmap.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap "kube-proxy" configured

root@shaolin:~# kubectl get configmap/kube-proxy -n kube-system -o yaml
apiVersion: v1
data:
  kubeconfig.conf: |
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://kubernetes.default.svc:6443
      name: default
... ...

重启kube-proxy(kubectl delete pods/kube-proxy-xxx -n kube-system)后,查看kube-proxy的日志:

root@shaolin:~# kubectl logs -f kube-proxy-h5sg8 -n kube-system
I0515 13:57:03.526032       1 server.go:225] Using iptables Proxier.
W0515 13:57:03.621532       1 proxier.go:298] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0515 13:57:03.621578       1 server.go:249] Tearing down userspace rules.
I0515 13:57:03.738015       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0515 13:57:03.741824       1 conntrack.go:66] Setting conntrack hashsize to 32768
I0515 13:57:03.742555       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0515 13:57:03.742731       1 conntrack.go:81] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600

九、小结

到这里,我们在第一部分中的最终思路方案已经实现了。不过这两篇文章对kubernetes ha cluster的打造还仅限于探索阶段,可能还有一些深层次的问题没有暴露出来,因此不建议在生产环境中采用。kubeadm在后续的版本中必然加入对k8s ha cluster的支持,那个时候,搭建一套可用于生产环境的HA cluster将不再这么麻烦了!

一步步打造基于Kubeadm的高可用Kubernetes集群-第一部分

Kubernetes集群的核心是其master node,但目前默认情况下master node只有一个,一旦master node出现问题,Kubernetes集群将陷入“瘫痪”,对集群的管理、Pod的调度等均将无法实施,即便此时某些用户的Pod依旧可以正常运行。这显然不能符合我们对于运行于生产环境下的Kubernetes集群的要求,我们需要一个高可用的Kubernetes集群。

不过,目前Kubernetes官方针对构建高可用(high-availability)的集群的支持还是非常有限的,只是针对少数cloud-provider提供了粗糙的部署方法,比如:使用kube-up.sh脚本在GCE上使用kops在AWS上等等。

高可用Kubernetes集群是Kubernetes演进的必然方向,官方在“Building High-Availability Clusters”一文中给出了当前搭建HA cluster的粗略思路。Kubeadm也将HA列入了后续版本的里程碑计划,并且已经出了一版使用kubeadm部署高可用cluster的方法提议草案

在kubeadm没有真正支持自动bootstrap的HA Kubernetes cluster之前,如果要搭建一个HA k8s cluster,我们应该如何做呢?本文将探索性地一步一步的给出打造一个HA K8s cluster的思路和具体步骤。不过需要注意的是:这里搭建的HA k8s cluser仅在实验室中测试ok,还并未在生产环境中run过,因此在某些未知的细节方面可能存在思路上的纰漏

一、测试环境

高可用Kubernetes集群主要就是master node的高可用,因此,我们申请了三台美国西部区域的阿里云ECS作为三个master节点。通过hostnamectl将这三个节点的static hostname分别改为shaolin、wudang和emei:

shaolin: 10.27.53.32
wudang: 10.24.138.208
emei: 10.27.52.72

三台主机运行的都是Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-63-generic x86_64),使用root用户。

Docker版本如下:

root@shaolin:~# docker version
Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

Ubuntu上Docker CE版本的安装步骤可以参看这里,由于我的服务器在美西,因此不存在”墙”的问题。对于主机在国内的朋友,你需要根据安装过程中是否输出错误日志自行决定是否需要配置一个加速器。另外,这里用的docker版本有些新,Kubernetes官网上提及最多的、兼容最好的还是docker 1.12.x版本,你也可以直接安装这个版本。

二、Master节点高可用的思路

通过对single-master node的探索,我们知道master节点上运行着如下几个Kubernetes组件:

  • kube-apiserver:集群核心,集群API接口、集群各个组件通信的中枢;集群安全控制;
  • etcd:集群的数据中心;
  • kube-scheduler:集群Pod的调度中心;
  • kube-controller-manager:集群状态管理器,当集群状态与期望不同时,kcm会努力让集群恢复期望状态,比如:当一个pod死掉,kcm会努力新建一个pod来恢复对应replicas set期望的状态;
  • kubelet: kubernetes node agent,负责与node上的docker engine打交道;
  • kubeproxy: 每个node上一个,负责service vip到endpoint pod的流量转发,当前主要通过设置iptables规则实现。

Kubernetes集群的高可用就是master节点的高可用,master节点的高可用归根结底就是上述这些运行于master node上的组件的高可用。因此,我们的思路就是考量如何让这些组件高可用起来!综合Kubernetes官方提供的资料以及一些proposal draft,我们知道完全从头搭建的hard way形式似乎不甚理智^0^,将一个由kubeadm创建的k8s cluster改造为一个ha的k8s cluster似乎更可行。下面是我的思路方案:

img{512x368}

前面提到过,我们的思路是基于kubeadm启动的kubernetes集群,通过逐步修改配置或替换,形成最终HA的k8s cluster。上图是k8s ha cluster的最终图景,我们可以看到:

  • kube-apiserver:得益于apiserver的无状态,每个master节点的apiserver都是active的,并处理来自Load Balance分配过来的流量;
  • etcd:状态的集中存储区。通过将多个master节点上的etcd组成一个etcd集群,使得apiserver共享集群状态和数据;
  • kube-controller-manager:kcm自带leader-elected功能,多个master上的kcm构成一个集群,但只有被elected为leader的kcm在工作。每个master节点上的kcm都连接本node上的apiserver;
  • kube-scheduler:scheduler自带leader-elected功能,多个master上的scheduler构成一个集群,但只有被elected为leader的scheduler在工作。每个master节点上的scheduler都连接本node上的apiserver;
  • kubelet: 由于master上的各个组件均以container的形式呈现,因此不承担workload的master节点上的kubelet更多是用来管理这些master组件容器。每个master节点上的kubelet都连接本node上的apiserver;
  • kube-proxy: 由于master节点不承载workload,因此master节点上的kube-proxy同样仅服务于一些特殊的服务,比如: kube-dns等。由于kubeadm下kube-proxy没有暴露出可供外部调整的配置,因此kube-proxy需要连接Load Balance暴露的apiserver的端口。

接下来,我们就来一步步按照我们的思路,对kubeadm启动的single-master node k8s cluster进行改造,逐步演进到我们期望的ha cluster状态。

三、第一步:使用kubeadm安装single-master k8s cluster

距离第一次使用kubeadm安装kubernetes 1.5.1集群已经有一些日子了,kubernetes和kubeadm都有了一些变化。当前kubernetes和kubeadm的最新release版都是1.6.2版本:

root@wudang:~# kubeadm version
kubeadm version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.2", GitCommit:"477efc3cbe6a7effca06bd1452fa356e2201e1ee", GitTreeState:"clean", BuildDate:"2017-04-19T20:22:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

root@wudang:~# docker images
REPOSITORY                                               TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/kube-proxy-amd64                v1.6.2              7a1b61b8f5d4        3 weeks ago         109 MB
gcr.io/google_containers/kube-controller-manager-amd64   v1.6.2              c7ad09fe3b82        3 weeks ago         133 MB
gcr.io/google_containers/kube-apiserver-amd64            v1.6.2              e14b1d5ee474        3 weeks ago         151 MB
gcr.io/google_containers/kube-scheduler-amd64            v1.6.2              b55f2a2481b9        3 weeks ago         76.8 MB
... ...

虽然kubeadm版本有更新,但安装过程没有太多变化,这里仅列出一些关键步骤,一些详细信息输出就在这里省略了。

我们先在shaolin node上安装相关程序文件:

root@shaolin:~# apt-get update && apt-get install -y apt-transport-https

root@shaolin:~# curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
OK

root@shaolin:~# cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
> deb http://apt.kubernetes.io/ kubernetes-xenial main
> EOF

root@shaolin:~# apt-get update

root@shaolin:~# apt-get install -y kubelet kubeadm kubectl kubernetes-cni

接下来,使用kubeadm启动集群。注意:由于在aliyun上flannel 网络插件一直不好用,这里还是使用weave network

root@shaolin:~/k8s-install# kubeadm init --apiserver-advertise-address 10.27.53.32
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[init] Using Kubernetes version: v1.6.2
[init] Using Authorization mode: RBAC
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.03.1-ce. Max validated version: 1.12
[preflight] Starting the kubelet service
[certificates] Generated CA certificate and key.
[certificates] Generated API server certificate and key.
[certificates] API Server serving cert is signed for DNS names [shaolin kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.27.53.32]
[certificates] Generated API server kubelet client certificate and key.
[certificates] Generated service account token signing key and public key.
[certificates] Generated front-proxy CA certificate and key.
[certificates] Generated front-proxy client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[apiclient] Created API client, waiting for the control plane to become ready
[apiclient] All control plane components are healthy after 17.045449 seconds
[apiclient] Waiting for at least one node to register
[apiclient] First node has registered after 5.008588 seconds
[token] Using token: a8dd42.afdb86eda4a8c987
[apiconfig] Created RBAC rules
[addons] Created essential addon: kube-proxy
[addons] Created essential addon: kube-dns

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run (as a regular user):

  sudo cp /etc/kubernetes/admin.conf $HOME/
  sudo chown $(id -u):$(id -g) $HOME/admin.conf
  export KUBECONFIG=$HOME/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

http://kubernetes.io/docs/admin/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join --token abcdefghijklmn 10.27.53.32:6443

root@shaolin:~/k8s-install# pods
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP            NODE
kube-system   etcd-shaolin                      1/1       Running   0          34s       10.27.53.32   shaolin
kube-system   kube-apiserver-shaolin            1/1       Running   0          35s       10.27.53.32   shaolin
kube-system   kube-controller-manager-shaolin   1/1       Running   0          23s       10.27.53.32   shaolin
kube-system   kube-dns-3913472980-tkr91         0/3       Pending   0          1m        <none>
kube-system   kube-proxy-bzvvk                  1/1       Running   0          1m        10.27.53.32   shaolin
kube-system   kube-scheduler-shaolin            1/1       Running   0          46s       10.27.53.32   shaolin

k8s 1.6.2版本的weave network的安装与之前稍有不同,因为k8s 1.6启用了更为安全的机制,默认采用RBAC对运行于cluster上的workload进行有限授权。我们要使用的weave network plugin的yaml为weave-daemonset-k8s-1.6.yaml

root@shaolin:~/k8s-install# kubectl apply -f https://git.io/weave-kube-1.6
clusterrole "weave-net" created
serviceaccount "weave-net" created
clusterrolebinding "weave-net" created
daemonset "weave-net" created

如果你的weave pod启动失败且原因类似如下日志:

Network 172.30.0.0/16 overlaps with existing route 172.16.0.0/12 on host.

你需要修改你的weave network的 IPALLOC_RANGE(这里我使用了172.32.0.0/16):

//weave-daemonset-k8s-1.6.yaml
... ...
spec:
  template:
    metadata:
      labels:
        name: weave-net
    spec:
      hostNetwork: true
      hostPID: true
      containers:
        - name: weave
          env:
            - name: IPALLOC_RANGE
              value: 172.32.0.0/16
... ...

master安装ok后,我们将wudang、emei两个node作为k8s minion node,来测试一下cluster的搭建是否是正确的,同时这一过程也在wudang、emei上安装上了kubelet和kube-proxy,这两个组件在后续的“改造”过程中是可以直接使用的:

以emei node为例:

root@emei:~# kubeadm join --token abcdefghijklmn 10.27.53.32:6443
[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.
[preflight] Running pre-flight checks
[preflight] WARNING: docker version is greater than the most recently validated version. Docker version: 17.03.1-ce. Max validated version: 1.12
[preflight] Starting the kubelet service
[discovery] Trying to connect to API Server "10.27.53.32:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.27.53.32:6443"
[discovery] Cluster info signature and contents are valid, will use API Server "https://10.27.53.32:6443"
[discovery] Successfully established connection with API Server "10.27.53.32:6443"
[bootstrap] Detected server version: v1.6.2
[bootstrap] The server supports the Certificates API (certificates.k8s.io/v1beta1)
[csr] Created API client to obtain unique certificate for this node, generating keys and certificate signing request
[csr] Received signed certificate from the API server, generating KubeConfig...
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"

Node join complete:
* Certificate signing request sent to master and response
  received.
* Kubelet informed of new secure connection details.

Run 'kubectl get nodes' on the master to see this machine join.

建立一个多pod的nginx服务,测试一下集群网络是否通!这里就不赘述了。

安装后的single-master kubernetes cluster的状态就如下图所示:

img{512x368}

四、第二步:搭建etcd cluster for ha k8s cluster

k8s集群状态和数据都存储在etcd中,高可用的k8s集群离不开高可用的etcd cluster。我们需要为最终的ha k8s cluster提供一个ha的etcd cluster,如何做呢?

当前k8s cluster中,shaolin master node上的etcd存储着k8s集群的所有数据和状态。我们需要在wudang和emei两个节点上也建立起etcd实例,与现存在 etcd共同构建成为高可用的且存储有cluster数据和状态的集群。我们将这一过程再细化为几个小步骤:

0、在emei、wudang两个节点上启动kubelet服务

etcd cluster可以采用完全独立的、与k8s组件无关的建立方法。不过这里我采用的是和master一样的方式,即采用由wudang和emei两个node上kubelet启动的etcd作为etcd cluster的两个member。此时,wudang和emei两个node的角色是k8s minion node,我们需要首先清理一下这两个node的数据:

root@shaolin:~/k8s-install # kubectl drain wudang --delete-local-data --force --ignore-daemonsets
node "wudang" cordoned
WARNING: Ignoring DaemonSet-managed pods: kube-proxy-mxwp3, weave-net-03jbh; Deleting pods with local storage: weave-net-03jbh
pod "my-nginx-2267614806-fqzph" evicted
node "wudang" drained

root@wudang:~# kubeadm reset
[preflight] Running pre-flight checks
[reset] Stopping the kubelet service
[reset] Unmounting mounted directories in "/var/lib/kubelet"
[reset] Removing kubernetes-managed containers
[reset] No etcd manifest found in "/etc/kubernetes/manifests/etcd.yaml", assuming external etcd.
[reset] Deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim]
[reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

root@shaolin:~/k8s-install # kubectl drain emei --delete-local-data --force --ignore-daemonsets
root@emei:~# kubeadm reset

root@shaolin:~/k8s-install# kubectl delete node/wudang
root@shaolin:~/k8s-install# kubectl delete node/emei

我们的小目标中:etcd cluster将由各个node上的kubelet自动启动;而kubelet则是由systemd在sys init时启动,且其启动配置如下:

root@wudang:~# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--kubeconfig=/etc/kubernetes/kubelet.conf --require-kubeconfig=true"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_EXTRA_ARGS

我们需要首先在wudang和emei node上将kubelet启动起来,我们以wudang node为例:

root@wudang:~# systemctl enable kubelet
root@wudang:~# systemctl start kubelet

查看kubelet service日志:

root@wudang:~# journalctl -u kubelet -f

May 10 10:58:41 wudang systemd[1]: Started kubelet: The Kubernetes Node Agent.
May 10 10:58:41 wudang kubelet[27179]: I0510 10:58:41.798507   27179 feature_gate.go:144] feature gates: map[]
May 10 10:58:41 wudang kubelet[27179]: error: failed to run Kubelet: invalid kubeconfig: stat /etc/kubernetes/kubelet.conf: no such file or directory
May 10 10:58:41 wudang systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
May 10 10:58:41 wudang systemd[1]: kubelet.service: Unit entered failed state.
May 10 10:58:41 wudang systemd[1]: kubelet.service: Failed with result 'exit-code'.

kubelet启动失败,因为缺少/etc/kubernetes/kubelet.conf这个配置文件。我们需要向shaolin node求援,我们需要将shaolin node上的同名配置文件copy到wudang和emei两个node下面,当然同时需要copy的还包括shaolin node上的/etc/kubernetes/pki目录:

root@wudang:~# kubectl --kubeconfig=/etc/kubernetes/kubelet.conf config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://10.27.53.32:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:shaolin
  name: system:node:shaolin@kubernetes
current-context: system:node:shaolin@kubernetes
kind: Config
preferences: {}
users:
- name: system:node:shaolin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

root@wudang:~# ls /etc/kubernetes/pki
apiserver.crt  apiserver-kubelet-client.crt  ca.crt  ca.srl              front-proxy-ca.key      front-proxy-client.key  sa.pub
apiserver.key  apiserver-kubelet-client.key ca.key  front-proxy-ca.crt  front-proxy-client.crt  sa.key

systemctl daemon-reload; systemctl restart kubelet后,再查看kubelet service日志,你会发现kubelet起来了!

以wudang node为例:

root@wudang:~# journalctl -u kubelet -f
-- Logs begin at Mon 2017-05-08 15:12:01 CST. --
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.213529   26907 factory.go:54] Registering systemd factory
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.213674   26907 factory.go:86] Registering Raw factory
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.213813   26907 manager.go:1106] Started watching for new ooms in manager
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.216383   26907 oomparser.go:185] oomparser using systemd
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.217415   26907 manager.go:288] Starting recovery of all containers
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.285428   26907 manager.go:293] Recovery completed
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.344425   26907 kubelet_node_status.go:230] Setting node annotation to enable volume controller attach/detach
May 11 10:37:07 wudang kubelet[26907]: E0511 10:37:07.356188   26907 eviction_manager.go:214] eviction manager: unexpected err: failed GetNode: node 'wudang' not found
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.358402   26907 kubelet_node_status.go:77] Attempting to register node wudang
May 11 10:37:07 wudang kubelet[26907]: I0511 10:37:07.363083   26907 kubelet_node_status.go:80] Successfully registered node wudang

此时此刻,我们先让wudang、emei node上的kubelet先连着shaolin node上的apiserver。

1、在emei、wudang两个节点上建立一个etcd cluster

我们以shaolin node上的/etc/kubernetes/manifests/etcd.yaml为蓝本,修改出wudang和emei上的etcd.yaml,主要的变化在于containers:command部分:

wudang上的/etc/kubernetes/manifests/etcd.yaml:

spec:
  containers:
  - command:
    - etcd
    - --name=etcd-wudang
    - --initial-advertise-peer-urls=http://10.24.138.208:2380
    - --listen-peer-urls=http://10.24.138.208:2380
    - --listen-client-urls=http://10.24.138.208:2379,http://127.0.0.1:2379
    - --advertise-client-urls=http://10.24.138.208:2379
    - --initial-cluster-token=etcd-cluster
    - --initial-cluster=etcd-wudang=http://10.24.138.208:2380,etcd-emei=http://10.27.52.72:2380
    - --initial-cluster-state=new
    - --data-dir=/var/lib/etcd
    image: gcr.io/google_containers/etcd-amd64:3.0.17

emei上的/etc/kubernetes/manifests/etcd.yaml:

spec:
  containers:
  - command:
    - etcd
    - --name=etcd-emei
    - --initial-advertise-peer-urls=http://10.27.52.72:2380
    - --listen-peer-urls=http://10.27.52.72:2380
    - --listen-client-urls=http://10.27.52.72:2379,http://127.0.0.1:2379
    - --advertise-client-urls=http://10.27.52.72:2379
    - --initial-cluster-token=etcd-cluster
    - --initial-cluster=etcd-emei=http://10.27.52.72:2380,etcd-wudang=http://10.24.138.208:2380
    - --initial-cluster-state=new
    - --data-dir=/var/lib/etcd
    image: gcr.io/google_containers/etcd-amd64:3.0.17

将这两个文件分别放入各自node的/etc/kubernetes/manifests目录后,各自node上的kubelet将会自动将对应的etcd pod启动起来!

root@shaolin:~# pods
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
kube-system   etcd-emei                         1/1       Running   0          11s       10.27.52.72     emei
kube-system   etcd-shaolin                      1/1       Running   0          25m       10.27.53.32     shaolin
kube-system   etcd-wudang                       1/1       Running   0          24s       10.24.138.208   wudang

我们查看一下当前etcd cluster的状态:

# etcdctl endpoint status --endpoints=10.27.52.72:2379,10.24.138.208:2379
10.27.52.72:2379, 6e80adf8cd57f826, 3.0.17, 25 kB, false, 17, 660
10.24.138.208:2379, f3805d1ab19c110b, 3.0.17, 25 kB, true, 17, 660

注:输出的列从左到右分别表示:endpoint URL, ID, version, database size, leadership status, raft term, and raft status.
因此,我们可以看出wudang(10.24.138.208)上的etcd被选为cluster leader了

我们测试一下etcd cluster,put一些key:

在wudang节点:(注意:export ETCDCTL_API=3)

root@wudang:~# etcdctl put foo bar
OK
root@wudang:~# etcdctl put foo1 bar1
OK
root@wudang:~# etcdctl get foo
foo
bar

在emei节点:

root@emei:~# etcdctl get foo
foo
bar

至此,当前kubernetes cluster的状态示意图如下:

img{512x368}

2、同步shaolin上etcd的数据到etcd cluster中

kubernetes 1.6.2版本默认使用3.x版本etcd。etcdctl 3.x版本提供了一个make-mirror功能用于在etcd cluster间同步数据,这样我们就可以通过etcdctl make-mirror将shaolin上etcd的k8s cluster数据同步到上述刚刚创建的etcd cluster中。在emei node上执行下面命令:

root@emei:~# etcdctl make-mirror --no-dest-prefix=true  127.0.0.1:2379  --endpoints=10.27.53.32:2379 --insecure-skip-tls-verify=true
... ...
261
302
341
380
420
459
498
537
577
616
655

... ...

etcdctl make-mirror每隔30s输出一次日志,不过通过这些日志无法看出来同步过程。并且etcdctl make-mirror似乎是流式同步:没有结束的边界。因此你需要手工判断一下数据是否都同步过去了!比如通过查看某个key,对比两边的差异的方式:

# etcdctl get --from-key /api/v2/registry/clusterrolebindings/cluster-admin

.. ..
compact_rev_key
122912

或者通过endpoint status命令查看数据库size大小,对比双方的size是否一致。一旦差不多了,就可以停掉make-mirror的执行了!

3、将shaolin上的apiserver连接的etcd改为连接etcd cluster,停止并删除shaolin上的etcd

修改shaolin node上的/etc/kubernetes/manifests/kube-apiserver.yaml,让shaolin上的kube0-apiserver连接到emei node上的etcd:

修改下面一行:
- --etcd-servers=http://10.27.52.72:2379

修改保存后,kubelet会自动重启kube-apiserver,重启后的kube-apiserver工作正常!

接下来,我们停掉并删除掉shaolin上的etcd(并删除相关数据存放目录):

root@shaolin:~# rm /etc/kubernetes/manifests/etcd.yaml
root@shaolin:~# rm -fr /var/lib/etcd

再查看k8s cluster当前pod,你会发现etcd-shaolin不见了。

至此,k8s集群的当前状态示意图如下:

img{512x368}

4、重新创建shaolin上的etcd ,并以member形式加入etcd cluster

我们首先需要在已存在的etcd cluster中添加etcd-shaolin这个member:

root@wudang:~/kubernetes-conf-shaolin/manifests# etcdctl member add etcd-shaolin --peer-urls=http://10.27.53.32:2380
Member 3184cfa57d8ef00c added to cluster 140cec6dd173ab61

然后,在shaolin node上基于原shaolin上的etcd.yaml文件进行如下修改:

// /etc/kubernetes/manifests/etcd.yaml
... ...
spec:
  containers:
  - command:
    - etcd
    - --name=etcd-shaolin
    - --initial-advertise-peer-urls=http://10.27.53.32:2380
    - --listen-peer-urls=http://10.27.53.32:2380
    - --listen-client-urls=http://10.27.53.32:2379,http://127.0.0.1:2379
    - --advertise-client-urls=http://10.27.53.32:2379
    - --initial-cluster-token=etcd-cluster
    - --initial-cluster=etcd-shaolin=http://10.27.53.32:2380,etcd-wudang=http://10.24.138.208:2380,etcd-emei=http://10.27.52.72:2380
    - --initial-cluster-state=existing
    - --data-dir=/var/lib/etcd
    image: gcr.io/google_containers/etcd-amd64:3.0.17

修改保存后,kubelet将自动拉起etcd-shaolin:

root@shaolin:~/k8s-install# pods
NAMESPACE     NAME                              READY     STATUS    RESTARTS   AGE       IP              NODE
kube-system   etcd-emei                         1/1       Running   0          3h        10.27.52.72     emei
kube-system   etcd-shaolin                      1/1       Running   0          8s        10.27.53.32     shaolin
kube-system   etcd-wudang                       1/1       Running   0          3h        10.24.138.208   wudang

查看etcd cluster状态:

root@shaolin:~# etcdctl endpoint status --endpoints=10.27.52.72:2379,10.24.138.208:2379,10.27.53.32:2379
10.27.52.72:2379, 6e80adf8cd57f826, 3.0.17, 11 MB, false, 17, 34941
10.24.138.208:2379, f3805d1ab19c110b, 3.0.17, 11 MB, true, 17, 34941
10.27.53.32:2379, 3184cfa57d8ef00c, 3.0.17, 11 MB, false, 17, 34941

可以看出三个etcd实例的数据size、raft status是一致的,wudang node上的etcd是leader!

5、将shaolin上的apiserver的etcdserver指向改回etcd-shaolin

// /etc/kubernetes/manifests/kube-apiserver.yaml

... ...
- --etcd-servers=http://127.0.0.1:2379
... ...

生效重启后,当前kubernetes cluster的状态如下面示意图:

img{512x368}

第二部分在这里




这里是Tony Bai的个人Blog,欢迎访问、订阅和留言!订阅Feed请点击上面图片

如果您觉得这里的文章对您有帮助,请扫描上方二维码进行捐赠,加油后的Tony Bai将会为您呈现更多精彩的文章,谢谢!

如果您喜欢通过微信App浏览本站内容,可以扫描下方二维码,订阅本站官方微信订阅号“iamtonybai”;点击二维码,可直达本人官方微博主页^_^:



本站Powered by Digital Ocean VPS。

选择Digital Ocean VPS主机,即可获得10美元现金充值,可免费使用两个月哟!

著名主机提供商Linode 10$优惠码:linode10,在这里注册即可免费获得。

阿里云推荐码:1WFZ0V立享9折!

View Tony Bai's profile on LinkedIn


文章

评论

  • 正在加载...

分类

标签

归档











更多