标签 iptables 下的文章

Hello,WireGuard

2020年1月28日,Linux之父Linus Torvalds正式将WireGuard merge到Linux 5.6版本内核主线

img{512x368}

图:WireGuard被加入linux kernel 5.6主线的commit log

这意味着在Linux 5.6内核发布时,linux在内核层面将原生支持一个新的VPN协议栈:WireGuard

img{512x368}

图:WireGuard Logo

一. VPN与WireGuard的创新

VPN,全称Virtual Private Network(虚拟专用网络)。提起VPN,大陆的朋友想到的第一件事就是fan qiang。其实fan qiang只是VPN的一个“小众”应用罢了^_^,企业网络才是VPN真正施展才能的地方。VPN支持在不安全的公网上建立一条加密的、安全的到企业内部网络的通道(隧道tunnel),这就好比专门架设了一个专用网络那样。在WireGuard出现之前,VPN的隧道协议主要有PPTPL2TPIPSec等,其中PPTP和L2TP协议工作在OSI模型的第二层,又称为二层隧道协议;IPSec是第三层隧道协议。

既然已经有了这么多的VPN协议,那么Why WireGuard?

WireGuard的作者Jason A. DonenfeldWireGuard官网给出了很明确地理由:

  • 简单、易用、无连接、无状态:号称目前最易用和最简单的VPN解决方案

WireGuard可以像SSH一样易于配置和部署。只需交换非常简单的公钥就可以建立VPN连接,就像交换SSH密钥一样,其余所有由WireGuard透明处理。并且WireGuard建立的VPN连接是基于UDP的,无需建立和管理连接,无需关心和管理状态的。

  • 先进加密协议

WireGuard充分利用安全领域和密码学在这些年的最新成果,使用noise frameworkCurve25519ChaCha20Poly1305BLAKE2SipHash24等构建WireGuard的安全方案。

  • 最小的攻击面(最少代码实现)

WireGuard的内核模块c代码仅不足5k行,便于代码安全评审。也使得WireGuard的实现更不容易被攻击(代码量少,理论上漏洞相对于庞大的代码集合而言也会少许多)。

  • 高性能

密码学最新成果带来的高速机密原语和WireGuard的内核驻留机制,使其相较于之前的VPN方案更具性能优势。

以上这些理由,同时也是WireGuard这个协议栈的特性。

这么说依然很抽象,我们来实操一下,体验一下WireGuard的简洁、易用、安全、高效。

二. WireGuard安装和使用

WireGuard将在linux 5.6内核中提供原生支持,也就是说在那之前,我们还无法直接使用WireGuard,安装还是不可避免的。在我的实验环境中有两台Linux VPS主机,都是ubuntu 18.04,内核都是4.15.0。因此我们需要首先添加WireGuard的ppa仓库:

sudo add-apt-repository ppa:wireguard/wireguard

更新源后,即可通过下面命令安装WireGuard:

sudo apt-get update

sudo apt-get install wireguard

安装的WireGuard分为两部分:

  • WireGuard内核模块(wireguard.ko),这部分通过动态内核模块技术DKMS安装到ubuntu的内核模块文件目录下:
$ ls /lib/modules/4.15.0-29-generic/updates/dkms/
wireguard.ko

  • 用户层的命令行工具

类似于内核netfilter和命令行工具iptables之间关系,wireguard.ko对应的用户层命令行工具wireguard-tools:wg、wg-quick被安装到/usr/bin下面了:

$ ls -t /usr/bin|grep wg|head -n 2
wg
wg-quick

1. peer to peer vpn

在两个linux Vps上都安装完WireGuard后,我们就可以在两个节点(peer)建立虚拟专用网络(VPN)了。我们分为称两个linux节点为peer1和peer2:

img{512x368}

图:点对点wireguard通信图

就像上图那样,我们只分别需要在peer1和peer2建立/etc/wireguard/wg0.conf

peer1的/etc/wireguard/wg0.conf

[Interface]
PrivateKey = {peer1's privatekey}
Address = 10.0.0.1
ListenPort = 51820

[Peer]
PublicKey = {peer2's publickey}
EndPoint = {peer2's ip}:51820
AllowedIPs = 10.0.0.2/32

peer2的/etc/wireguard/wg0.conf

[Interface]
PrivateKey = {peer2's privatekey}
Address = 10.0.0.2
ListenPort = 51820

[Peer]
PublicKey = {peer1's publickey}
EndPoint = {peer1's ip}:51820
AllowedIPs = 10.0.0.1/32

我们看到每个peer上WireGuard所需的配置文件wg0.conf包含两大部分:

  • [Interface]部分

    • PrivateKey – peer自身的privatekey

    • Address – peer的wg0接口在vpn网络中绑定的路由ip范围,在上述例子中仅绑定了一个ip地址

    • ListenPort – wg网络协议栈监听UDP端口

  • [Peer]部分(描述vpn网中其他peer信息,一个wg0配置文件中显然可以配置多个Peer部分)

    • PublicKey – 该peer的publickey

    • EndPoint – 该peer的wg网路协议栈地址(ip+port)

    • AllowedIPs – 允许该peer发送过来的wireguard载荷中的源地址范围。同时本机而言,这个字段也会作为本机路由表中wg0绑定的ip范围。

每个Peer自身的privatekey和publickey可以通过WireGuard提供的命令行工具生成:

$ wg genkey | tee privatekey | wg pubkey > publickey
$ ls
privatekey  publickey

注:这两个文件可以生成在任意路径下,我们要的是两个文件中内容。

在两个peer上配置完/etc/wireguard/wg0.conf配置文件后,我们就可以使用下面命令在peer1和peer2之间建立一条双向加密VPN隧道了:

peer1:

$ sudo wg-quick up wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.0.0.1 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip -4 route add 10.0.0.2/32 dev wg0

peer2:

$ sudo wg-quick up wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.0.0.2 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] ip -4 route add 10.0.0.1/32 dev wg0

执行上述命令,每个peer会增加一个network interface dev: wg0,并在系统路由表中增加一条路由,以peer1为例:

$ ip a

... ...

4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 10.0.0.1/32 scope global wg0
       valid_lft forever preferred_lft forever

$ ip route
default via 172.21.0.1 dev eth0 proto dhcp metric 100
10.0.0.2 dev wg0 scope link
... ...

现在我们来测试两个Peer之间的连通性。WireGuard的peer之间是对等的,谁发起的请求谁就是client端。我们在peer1上ping peer2,在peer2上我们用tcpdump抓wg0设备的包:

Peer1:

$ ping -c 3 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=34.9 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=34.7 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=34.6 ms

--- 10.0.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 34.621/34.781/34.982/0.262 ms

Peer2:

# tcpdump -i wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
13:29:52.659550 IP 10.0.0.1 > instance-cspzrq3u: ICMP echo request, id 20580, seq 1, length 64
13:29:52.659603 IP instance-cspzrq3u > 10.0.0.1: ICMP echo reply, id 20580, seq 1, length 64
13:29:53.660463 IP 10.0.0.1 > instance-cspzrq3u: ICMP echo request, id 20580, seq 2, length 64
13:29:53.660495 IP instance-cspzrq3u > 10.0.0.1: ICMP echo reply, id 20580, seq 2, length 64
13:29:54.662201 IP 10.0.0.1 > instance-cspzrq3u: ICMP echo request, id 20580, seq 3, length 64
13:29:54.662234 IP instance-cspzrq3u > 10.0.0.1: ICMP echo reply, id 20580, seq 3, length 64

我们看到peer1和peer2经由WireGuard建立的vpn实现了连通:在peer2上ping peer1(10.0.0.1)亦得到相同结果。

这时如果我们如果在peer2(vpn ip: 10.0.0.2)上启动一个http server(监听0.0.0.0:9090):

//httpserver.go
package main

import "net/http"

func index(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("hello, wireguard\n"))
}

func main() {
    http.Handle("/", http.HandlerFunc(index))
    http.ListenAndServe(":9090", nil)
}

那么我们在peer1(vpn ip:10.0.0.1)去访问这个server:

$ curl http://10.0.0.2:9090
hello, wireguard

在peer2(instance-cspzrq3u)上的tcpdump显示(tcp握手+数据通信+tcp拆除):

14:15:05.233794 IP 10.0.0.1.43922 > instance-cspzrq3u.9090: Flags [S], seq 1116349511, win 27600, options [mss 1380,sackOK,TS val 3539789774 ecr 0,nop,wscale 7], length 0
14:15:05.233854 IP instance-cspzrq3u.9090 > 10.0.0.1.43922: Flags [S.], seq 3504538202, ack 1116349512, win 27360, options [mss 1380,sackOK,TS val 2842719516 ecr 3539789774,nop,wscale 7], length 0
14:15:05.268792 IP 10.0.0.1.43922 > instance-cspzrq3u.9090: Flags [.], ack 1, win 216, options [nop,nop,TS val 3539789809 ecr 2842719516], length 0
14:15:05.268882 IP 10.0.0.1.43922 > instance-cspzrq3u.9090: Flags [P.], seq 1:78, ack 1, win 216, options [nop,nop,TS val 3539789809 ecr 2842719516], length 77
14:15:05.268907 IP instance-cspzrq3u.9090 > 10.0.0.1.43922: Flags [.], ack 78, win 214, options [nop,nop,TS val 2842719551 ecr 3539789809], length 0
14:15:05.269514 IP instance-cspzrq3u.9090 > 10.0.0.1.43922: Flags [P.], seq 1:134, ack 78, win 214, options [nop,nop,TS val 2842719552 ecr 3539789809], length 133
14:15:05.304147 IP 10.0.0.1.43922 > instance-cspzrq3u.9090: Flags [.], ack 134, win 224, options [nop,nop,TS val 3539789845 ecr 2842719552], length 0
14:15:05.304194 IP 10.0.0.1.43922 > instance-cspzrq3u.9090: Flags [F.], seq 78, ack 134, win 224, options [nop,nop,TS val 3539789845 ecr 2842719552], length 0
14:15:05.304317 IP instance-cspzrq3u.9090 > 10.0.0.1.43922: Flags [F.], seq 134, ack 79, win 214, options [nop,nop,TS val 2842719586 ecr 3539789845], length 0
14:15:05.339035 IP 10.0.0.1.43922 > instance-cspzrq3u.9090: Flags [.], ack 135, win 224, options [nop,nop,TS val 3539789880 ecr 2842719586], length 0

如果要拆除这个vpn,只需在每个peer上分别执行如下命令:

$ sudo wg-quick down wg0
[#] ip link delete dev wg0

2. peer to the local network of other peer

上面两个peer虽然实现了点对点的连通,但是如果我们想从peer1访问peer2所在的局域网中的另外一台机器(这显然是vpn最常用的应用场景),如下面示意图:

img{512x368}

图:从一个peer到另外一个peer所在局域网的节点的通信图

基于目前的配置是否能实现呢?我们来试试。首先我们在peer1上要将192.168.1.0/24网段的路由指到wg0上,这样我们在peer1上ping或curl 192.168.1.123:9090,数据才能被交给wg0处理并通过vpn网络送出,修改peer1上的wg0.conf:

// peer1's /etc/wireguard/wg0.conf

... ...
[Peer]
PublicKey = {peer2's publickey}
EndPoint = peer2's ip:51820
AllowedIPs = 10.0.0.2/32,192.168.1.0/24

重启peer1上的wg0使上述配置生效。然后我们尝试在peer1上ping 192.168.1.123:

$ ping -c 3 192.168.1.123
PING 192.168.1.123 (192.168.1.123) 56(84) bytes of data.

--- 192.168.1.123 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2038ms

我们在peer2上的tcpdump显示:

# tcpdump -i wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
14:33:38.393520 IP 10.0.0.1 > 192.168.1.123: ICMP echo request, id 30426, seq 1, length 64
14:33:39.408083 IP 10.0.0.1 > 192.168.1.123: ICMP echo request, id 30426, seq 2, length 64
14:33:40.432079 IP 10.0.0.1 > 192.168.1.123: ICMP echo request, id 30426, seq 3, length 64

我们看到peer2收到来自10.0.0.1的到192.168.1.123的ping包都没有对应的回包,通信失败。Why?我们分析一下。

peer2在51820端口收到WireGuard包后,去除wireguard包的包裹,露出真实数据包。真实数据包的目的ip地址为192.168.1.123,该地址并非peer2自身地址(其自身局域网地址为192.168.1.10)。既然不是自身地址,就不能送到上层协议栈(tcp)处理,那么另外一条路是forward(转发)出去。但是是否允许转发么?显然从结果来看,从wg0收到的消息无权转发,于是消息丢弃,这就是没有回包和通信失败的原因。

为了支持转发(这是vpn常用场景的功能哦),我们需要为peer2的wg0.conf增加些转发配置:

// peer2's  wg0.conf

[Interface]

... ...
PostUp   = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUT  ING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUT  ING -o eth0 -j MASQUERADE

... ...

重启peer2的wg0。在peer2的内核层我们也要开启转发开关:

// /etc/sysctl.conf

net.ipv4.ip_forward=1

net.ipv6.conf.all.forwarding=1

执行下面命令临时生效:

# sysctl -p
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

接下来,我们再来测试一下连通性。我们在peer1上再次尝试ping 192.168.1.123

$ ping -c 3 192.168.1.123
PING 192.168.1.123 (192.168.1.123) 56(84) bytes of data.
64 bytes from 192.168.1.123: icmp_seq=1 ttl=46 time=200 ms
64 bytes from 192.168.1.123: icmp_seq=2 ttl=46 time=200 ms
64 bytes from 192.168.1.123: icmp_seq=3 ttl=46 time=200 ms

--- 192.168.1.123 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 200.095/200.239/200.396/0.531 ms

这回通了!peer2上的Tcpdump输出中也看到了回包:

14:49:58.808467 IP 10.0.0.1 > 192.168.1.123: ICMP echo request, id 402, seq 1, length 64
14:49:58.974035 IP 192.168.1.123 > 10.0.0.1: ICMP echo reply, id 402, seq 1, length 64
14:49:59.809747 IP 10.0.0.1 > 192.168.1.123: ICMP echo request, id 402, seq 2, length 64
14:49:59.975240 IP 192.168.1.123 > 10.0.0.1: ICMP echo reply, id 402, seq 2, length 64
14:50:00.810802 IP 10.0.0.1 > 192.168.1.123: ICMP echo request, id 402, seq 3, length 64
14:50:00.976202 IP 192.168.1.123 > 10.0.0.1: ICMP echo reply, id 402, seq 3, length 64

我们在192.168.1.123上运行上面的那个httpserver程序,再在peer1上用curl访问这个程序:

$ curl 192.168.1.123:9090
hello, wireguard

我们看到httpserver的应答成功返回。peer2上的tcpdump也抓到了整个通信过程:

14:50:36.437259 IP 10.0.0.1.47918 > 192.168.1.123.9090: Flags [S], seq 3235649864, win 27600, options [mss 1380,sackOK,TS val 101915019 ecr 0,nop,wscale 7], length 0
14:50:36.593554 IP 192.168.1.123.9090 > 10.0.0.1.47918: Flags [S.], seq 2420552016, ack 3235649865, win 28960, options [mss 1460,sackOK,TS val 2323314775 ecr 101915019,nop,wscale 7], length 0
14:50:36.628315 IP 10.0.0.1.47918 > 192.168.1.123.9090: Flags [.], ack 1, win 216, options [nop,nop,TS val 101915210 ecr 2323314775], length 0
14:50:36.628379 IP 10.0.0.1.47918 > 192.168.1.123.9090: Flags [P.], seq 1:84, ack 1, win 216, options [nop,nop,TS val 101915210 ecr 2323314775], length 83
14:50:36.784550 IP 192.168.1.123.9090 > 10.0.0.1.47918: Flags [.], ack 84, win 227, options [nop,nop,TS val 2323314822 ecr 101915210], length 0
14:50:36.784710 IP 192.168.1.123.9090 > 10.0.0.1.47918: Flags [P.], seq 1:134, ack 84, win 227, options [nop,nop,TS val 2323314822 ecr 101915210], length 133
14:50:36.820339 IP 10.0.0.1.47918 > 192.168.1.123.9090: Flags [.], ack 134, win 224, options [nop,nop,TS val 101915401 ecr 2323314822], length 0
14:50:36.820383 IP 10.0.0.1.47918 > 192.168.1.123.9090: Flags [F.], seq 84, ack 134, win 224, options [nop,nop,TS val 101915401 ecr 2323314822], length 0
14:50:36.977226 IP 192.168.1.123.9090 > 10.0.0.1.47918: Flags [F.], seq 134, ack 85, win 227, options [nop,nop,TS val 2323314870 ecr 101915401], length 0
14:50:37.011927 IP 10.0.0.1.47918 > 192.168.1.123.9090: Flags [.], ack 135, win 224, options [nop,nop,TS val 101915594 ecr 2323314870], length 0

3. WireGuard的用户层实现

在linux上,我们务必使用WireGuard的内核模式,这显然是最高效的。在macOS、Windows上,WireGuard无法以内核模块驻留模式运行,但WireGuard项目提供了WireGuard的用户层实现。其作者Jason A. Donenfeld亲自实现了Go语言版本的wireguard-go。macOS上使用的就是wireguard的Go实现。我们可以使用brew在macOS上按照WireGuard:

$brew install wireguard-tools

配置好/etc/wireguard/wg0.conf后(和linux上的配置方式一致),同样可以通过wg-quick命令启动wireguard:

$sudo wg-quick up wg0

wg-quick实际上会通过wireguard-go来实现linux wireguard在内核中完成的功能:

$ps -ef|grep wireguard

    0 57783     1   0  3:18下午 ttys002    0:00.01 wireguard-go utun

三. WireGuard性能如何

关于WireGuard性能如何,官方给出了一个性能基准测试的对比数据(相较于其他vpn网络栈):

img{512x368}

图:WireGuard性能与其他vpn网络栈的对比(来自官方截图)

我们看到和IPSec、OpenVPN相比,无论从吞吐还是延迟,WireGuard都领先不少。

我们这里用microsoft开源的带宽测试工具ethr来直观看一下走物理网络和走WireGuard VPN的带宽差别。

在peer2上运行:

$ ethr -s

然后在peer1上分别通过物理网络和VPN网络向peer2发起请求:

  • peer1 -> peer2 (物理网络)
$ ethr -c  peer2's ip
Connecting to host [peer2 ip], port 9999
[  6] local 172.21.0.5 port 46108 connected to  peer2 ip port 9999
- - - - - - - - - - - - - - - - - - - - - - -
[ ID]   Protocol    Interval      Bits/s
[  6]     TCP      000-001 sec     1.54M
[  6]     TCP      001-002 sec     1.54M
[  6]     TCP      002-003 sec     1.54M
[  6]     TCP      003-004 sec     1.54M
[  6]     TCP      004-005 sec     1.54M

.... ...

  • peer1 -> peer2 (vpn网络)
$ ethr -c 10.0.0.2
Connecting to host [10.0.0.2], port 9999
[  6] local 10.0.0.1 port 36010 connected to 10.0.0.2 port 9999
- - - - - - - - - - - - - - - - - - - - - - -
[ ID]   Protocol    Interval      Bits/s
[  6]     TCP      000-001 sec     1.79M
[  6]     TCP      001-002 sec      640K
[  6]     TCP      002-003 sec     1.15M
[  6]     TCP      003-004 sec      512K
[  6]     TCP      004-005 sec     1.02M
[  6]     TCP      005-006 sec     1.02M
[  6]     TCP      006-007 sec     1.02M

我们看到走vpn的带宽相当于走物理网络的66%(1.02/1.54)左右。这里peer1(腾讯云)、peer2(百度云)之间走的是互联网,而在局域网测试的效果可能更好(留给大家^_^)。

四. 小结

经过上面的实验,我们看到了WireGuard的配置的确十分简单,这也是我目前使用过的配置过程最为简单的vpn。随着linux kernel 5.6内置对WireGuard的原生支持,WireGuard在vpn领域势必会有更为广泛的应用。

在容器网络方面,目前WireGuard已经给出了跨容器的网络通信方案,基于wireguard的k8s cni网络插件wormhole可以让pod之间通过wireguard实现的overlay网络通信。

国外的tailscale公司正在实现一种基于Wireguard的mesh vpn网络,该网络以WireGuard为数据平面的承载体,该公司主要实现控制平面。该公司目前聚集了一些Go核心开发人员,这里就包括著名的go核心开发团队成员、net/http包的最初作者和当前维护者的Brad Fitzpatrick。

五. 参考资料


我的网课“Kubernetes实战:高可用集群搭建、配置、运维与应用”在慕课网上线了,感谢小伙伴们学习支持!

我爱发短信:企业级短信平台定制开发专家 https://tonybai.com/
smspush : 可部署在企业内部的定制化短信平台,三网覆盖,不惧大并发接入,可定制扩展; 短信内容你来定,不再受约束, 接口丰富,支持长短信,签名可选。

著名云主机服务厂商DigitalOcean发布最新的主机计划,入门级Droplet配置升级为:1 core CPU、1G内存、25G高速SSD,价格5$/月。有使用DigitalOcean需求的朋友,可以打开这个链接地址:https://m.do.co/c/bff6eed92687 开启你的DO主机之路。

Gopher Daily(Gopher每日新闻)归档仓库 – https://github.com/bigwhite/gopherdaily

我的联系方式:

微博:https://weibo.com/bigwhite20xx
微信公众号:iamtonybai
博客:tonybai.com
github: https://github.com/bigwhite

微信赞赏:
img{512x368}

商务合作方式:撰稿、出书、培训、在线课程、合伙创业、咨询、广告合作。

再谈Docker容器单机网络:利用iptables trace和ebtables log

这大半年一直在搞Kubernetes。每次搭建Kubernetes集群,或多或少都会被Kubernetes的“网络插件们”折腾折腾。因此,要说目前Kubernetes中最难搞的是什么?个人觉得莫过于其Pod网络了,至少也是最难搞的之一。除此之外,以Service和Pod为中心的Kubernetes架构还大量利用iptables规则来实现Service的反向代理和负载均衡,这又与Docker原生容器单机网络实现所基于的linux bridgeiptables规则糅合在一起,让troubleshooting时的难度又增加了一些。

去年曾经花过一段研究Docker网络,但现在看来当时在某些关键环节的理解上还有些模糊,于是花了周末的闲暇时间对Docker容器单机网络做了一次再理解。这次重新认识利用上了iptables的Trace功能以及数据链路层的ebtables,让我可以更清晰地看到单机容器网络的网络数据流流向。同时,有了容器网络理解这个基础,对后续解决K8s Pod网络问题也是大有裨益的。

本文从某个角度来说也可以理解为自我答疑,我不会从最最基础的Docker网络结构说起,对Docker容器单机网络结构不了解的童鞋,可以先看看我之前写的《理解Docker单机容器网络》和《理解Docker容器网络之Linux Network Namespace》两篇文章。

一、实验环境

1、主机环境和工具版本

Docker的默认单机容器网络从最初的版本开始就几乎没有变过,因此理论上下面的分析适用于Docker的大部分版本。我的实验环境如下:

Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-63-generic x86_64)

# docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:18 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

# iptables --version
iptables v1.6.0
# ebtables --version
ebtables v2.0.10-4 (December 2011)

2、容器网络及拓扑

我们需要制作一个用于实验的容器镜像。因为这里仅用ping包进行测试,这里我们仅基于ubuntu:14.04 base image制作一个简单的安装有必要网络工具的image:

//Dockerfile

From ubuntu:14.04
RUN apt-get update && apt-get install -y curl iptables
ENTRYPOINT ["tail", "-f", "/var/log/bootstrap.log"]

// 制作镜像:

# docker build -t foo:latest ./

启动两个容器:

# docker run --name c1 -d --cap-add=NET_ADMIN foo:latest
7a01a19d9328b39f094c9a9c76340d179baaf93afb52189816bcc79f8319cb64
# docker run --name c2 -d --cap-add=NET_ADMIN foo:latest
94a2f1841f6d95fd0682299b17c0aedb60c1047786c8e75b0f1ab7316a995409

容器启动后的网络信息汇总如下:

# ifconfig -a
docker0   Link encap:Ethernet  HWaddr 02:42:ff:27:17:4d
          inet addr:192.168.0.1  Bcast:0.0.0.0  Mask:255.255.240.0
          ... ...

eth0      Link encap:Ethernet  HWaddr 00:16:3e:06:3a:3a
          inet addr:10.171.77.0  Bcast:10.171.79.255  Mask:255.255.248.0
          ... ...

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          ... ...

veth0594f4b Link encap:Ethernet  HWaddr 96:5b:d4:80:73:5f
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          ... ...

veth57a3dec Link encap:Ethernet  HWaddr 02:52:e9:60:ea:b1
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          ... ...

为了方便大家理解,这里附上一幅简易的容器网络拓扑:

img{512x368}

二、调试工具配置

Docker单机容器网络默认使用的是桥接网络,所有启动的容器均桥接在Docker引擎创建的docker0 linux bridge上,因此内核对Linux bridge的处理逻辑是理解Docker容器网络的关键。

与硬件网桥/交换机不同的是,Linux Bridge还具备三层网络,即IP层的功能,也就是docker0既是一个网桥也是一个具备三层转发功能的网卡设备。传统意义上,按照iso网络七层规范,iptables工作在三层,而网桥是一个二层(数据链路层)设备,但Linux协议栈针对网桥设备的实现却在网络层的规则链(ebtables)中串接了iptables的规则链处理,即在二层也可以处理ip包,这是为了实现桥接透明防火墙的需要。但实现也会保证每个packet数据包仅会走一次iptable的某个chain,要么在linker layer走,要么在network layer走,不会出现在linker layer走一次,又在network layer重复走一次的情况。关于这种基于linux bridge的ebtables和iptables的交互规则,在netfilter官网的一篇名为《ebtables/iptables interaction on a Linux-based bridge》文档中有详细说明,这篇文章也是后续分析的一个重要参考。下面这幅图也是文章中提到的那幅netfilter数据流全图,后续在分析时会反复回到这幅图(后续简称为:全图):

img{512x368}
建议:右键在新标签中打开图片看大图

关于数据包在iptables的各条chain的流经图可以参见下面:

img{512x368}

1、iptables TRACE target的设置

在本次实验中,我们主要需要查看数据包的流转路径,因此我们需要针对iptables的data flow进行跟踪。之前,我曾使用过iptables提供的LOG target或mark set&match方式来跟踪iptables中的数据流,但这两种方式都不理想,需要针对特定流程插入LOG target或match在入口包设定好的mark,对iptables规则的侵入较大,调试和观察也较为复杂;iptables自身提供了TRACE功能,一旦设定,当数据包匹配到任意chain上任意table的处理规则时,iptables会在系统日志(/var/log/syslog)中自动输出此时的数据包状态日志。

我们来为iptables规则添加TRACE,TRACE target只能在iptables的raw表中添加,raw表中有两条iptables built-in chain: PREROUTING和OUTPUT,分别代表网卡数据入口和本地进程下推数据的出口。TRACE target就添加在这两条chain上,步骤如下:

# iptables -t raw -A OUTPUT -p icmp -j TRACE
# iptables -t raw -A PREROUTING -p icmp -j TRACE

注意:我们采用icmp协议(ping协议)进行测试,因此我们只TRACE icmp协议的请求和应答包。

2、ebtables的调试设置

我们的重点在iptables,为ebtables只是辅助,帮助我们看清数据包到底是在哪一层被hook进iptables的规则链中进行处理的。因此我们在全图中的每个ebtables的built-in chain上都加上LOG(ebtables目前还不支持TRACE):

# ebtables -t broute -A BROUTING -p ipv4 --ip-proto 1 --log-level 6 --log-ip --log-prefix "TRACE: eb:broute:BROUTING" -j ACCEPT
# ebtables -t nat -A OUTPUT -p ipv4 --ip-proto 1 --log-level 6 --log-ip --log-prefix "TRACE: eb:nat:OUTPUT"  -j ACCEPT
# ebtables -t nat -A PREROUTING -p ipv4 --ip-proto 1 --log-level 6 --log-ip --log-prefix "TRACE: eb:nat:PREROUTING" -j ACCEPT
# ebtables -t filter -A INPUT -p ipv4 --ip-proto 1 --log-level 6 --log-ip --log-prefix "TRACE: eb:filter:INPUT" -j ACCEPT
# ebtables -t filter -A FORWARD -p ipv4 --ip-proto 1 --log-level 6 --log-ip --log-prefix "TRACE: eb:filter:FORWARD" -j ACCEPT
# ebtables -t filter -A OUTPUT -p ipv4 --ip-proto 1 --log-level 6 --log-ip --log-prefix "TRACE: eb:filter:OUTPUT" -j ACCEPT
# ebtables -t nat -A POSTROUTING -p ipv4 --ip-proto 1 --log-level 6 --log-ip --log-prefix "TRACE: eb:nat:POSTROUTING" -j ACCEPT

注意:这里--ip-proto 1 表示仅match icmp packet。

3、iptables和ebtables规则全文

启动两个容器并添加上述规则后,当前的的iptables规则如下:(通过iptables-save输出的按table组织的rules)

# iptables-save
# Generated by iptables-save v1.6.0 on Sun Nov  5 14:50:46 2017
*raw

: PREROUTING ACCEPT [1564539:108837380]
:OUTPUT ACCEPT [1504962:130805835]
-A PREROUTING -p icmp -j TRACE
-A OUTPUT -p icmp -j TRACE
COMMIT
# Completed on Sun Nov  5 14:50:46 2017
# Generated by iptables-save v1.6.0 on Sun Nov  5 14:50:46 2017
*filter
:INPUT ACCEPT [1564535:108837044]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [1504968:130806627]

: DOCKER - [0:0]

: DOCKER-ISOLATION - [0:0]

: DOCKER-USER - [0:0]

-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Sun Nov  5 14:50:46 2017
# Generated by iptables-save v1.6.0 on Sun Nov  5 14:50:46 2017
*nat

: PREROUTING ACCEPT [280:14819]
:INPUT ACCEPT [278:14651]
:OUTPUT ACCEPT [639340:38370263]

: POSTROUTING ACCEPT [639342:38370431]

: DOCKER - [0:0]

-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 192.168.0.0/20 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Sun Nov  5 14:50:46 2017

而ebtables的规则如下:

# ebtables-save
# Generated by ebtables-save v1.0 on Sun Nov  5 16:51:50 CST 2017
*nat
: PREROUTING ACCEPT
:OUTPUT ACCEPT
: POSTROUTING ACCEPT
-A PREROUTING -p IPv4 --ip-proto icmp --log-level info --log-prefix "TRACE: eb:nat:PREROUTING" --log-ip -j ACCEPT
-A OUTPUT -p IPv4 --ip-proto icmp --log-level info --log-prefix "TRACE: eb:nat:OUTPUT" --log-ip -j ACCEPT
-A POSTROUTING -p IPv4 --ip-proto icmp --log-level info --log-prefix "TRACE: eb:nat:POSTROUTING" --log-ip -j ACCEPT

*broute
:BROUTING ACCEPT
-A BROUTING -p IPv4 --ip-proto icmp --log-level info --log-prefix "TRACE: eb:broute:BROUTING" --log-ip -j ACCEPT

*filter
:INPUT ACCEPT
:FORWARD ACCEPT
:OUTPUT ACCEPT
-A INPUT -p IPv4 --ip-proto icmp --log-level info --log-prefix "TRACE: eb:filter:INPUT" --log-ip -j ACCEPT
-A FORWARD -p IPv4 --ip-proto icmp --log-level info --log-prefix "TRACE: eb:filter:FORWARD" --log-ip -j ACCEPT
-A OUTPUT -p IPv4 --ip-proto icmp --log-level info --log-prefix "TRACE: eb:filter:OUTPUT" --log-ip -j ACCEPT

对于iptables,我们还可以通过iptables命令输出另外一种组织形式的规则列表,我们这里列出filter和nat这两个重要的table的规则(输出规则number,便于后续match分析时查看):

# iptables -nL --line-numbers -v -t filter
Chain INPUT (policy ACCEPT 2558K packets, 178M bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       10   840 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
2       10   840 DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0
3        7   588 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
4        3   252 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
5        0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
6        3   252 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 2460K packets, 214M bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       10   840 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       10   840 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

# iptables -nL --line-numbers -v -t nat
Chain PREROUTING (policy ACCEPT 884 packets, 46522 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1      881 46270 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 881 packets, 46270 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 1048K packets, 63M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 1048K packets, 63M bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MASQUERADE  all  --  *      !docker0  192.168.0.0/20       0.0.0.0/0

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

三、Container to Container

下面,我们分三种情况来看看容器网络的数据包是如何流动的,首先是Container to Container。

img{512x368}

我们在容器C1中执行ping 3次 C2的命令:

# docker exec c1 ping -c 3 192.168.0.3
PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.
64 bytes from 192.168.0.3: icmp_seq=1 ttl=64 time=0.226 ms
64 bytes from 192.168.0.3: icmp_seq=2 ttl=64 time=0.159 ms
64 bytes from 192.168.0.3: icmp_seq=3 ttl=64 time=0.185 ms

--- 192.168.0.3 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.159/0.190/0.226/0.027 ms

在容器c1(192.168.0.2)中,icmp request由ping程序(c1 namespace中的local process)发出。c1 network namespace中的路由表如下:

# docker exec c1 netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG        0 0          0 eth0
192.168.0.0     0.0.0.0         255.255.240.0   U         0 0          0 eth0

由于目标容器地址为192.168.0.3,在容器c1的直连网络上,走第二条直连路由(非默认路由),数据包通过eth0发出。

由于c1 namespace中的eth0通过veth机制连接在host namespace的docker0 bridge的一个Slave port上,因此上述数据包通过docker0 bridge的slave port: veth0594f4b流入docker0 bridge。

这里再强调一下linux bridge设备。Linux下的Bridge是一种虚拟设备,它依赖于一个或多个从设备。它不是内核虚拟出的和从设备同一层次的镜像设备,而是内核虚拟出的一个高一层次的设备,并把从设备虚拟化为端口port,同时处理各个从设备的数据收发及转发。bridge设备是建立在从设备之上的(这些从设备可以是实际设备,也可以是vlan设备等),并且我们可以为bridge准备一个IP(bridge设备的MAC地址是它所有从设备中最小的MAC地址),这样该主机就可以通过这个bridge设备与网络中的其它主机通信了。另外一旦某个网络设备被“插到”linux bridge上,这个网络设备将会变为bridge的从设备,被虚拟化为端口port,从设备的IP及MAC都不再可用,好似被bridge剥夺了被内核网络栈处理的资格;它们被设置为接收任何包,对其流入的数据包的处理交由bridge完成,并最终由bridge设备来决定数据包的去向:接收到本机、转发或丢弃。

因此,位于host namespace的docker0 bridge从slave port: veth0594f4b收到icmp request后,我们不会看到veth0594f4b这一netdev被内核网络栈程序单独处理(比如:单独走一遍ebtables和iptables chains),而是进入bridge处理逻辑(此时可以回顾一下上面的全图)。由于数据包已经进入到了host namespace,因此我们可以通过ebtables和iptables输出的Trace和log来跟踪数据包流转的路径了:

1、start -> bridgecheck -> linker layer

TRACE: eb:broute:BROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1
TRACE: eb:nat:PREROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1

从最初的trace log来看,在bridge check之后(发现it is a linux bridge),数据包进入到linker layer中;并且在linker layer的BROUTING built-in chain之后,数据包没有被转移到上面的network layer,而是继续linker layer的行程:进入linker layer的nat:PREROUTING中。

2、call iptables chain rules in linker layer

结合全图中的图示和日志输出,在linker layer的nat:PREROUTING之后,linker layer调用了上层iptables的处理规则:raw:PREROUTING和nat:PREROUTING:

TRACE: raw:PREROUTING:policy:2 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1
TRACE: nat:PREROUTING:policy:2 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1

Trace target在数据包match table、chains的policy或rules时会输出日志,日志格式:”TRACE:tablename:chainname:type:rulenum”。当匹配到的是普通rules时,type=”rule”;当碰到一个user-defined chain的return target时,type=”return”;当匹配到built-in chain(比如:PREROUTING、INPUT、OUTPUT、FORWARD和POSTROUTING)的default policy时,type=”policy”。

从上面的日志输出来看,似乎PREROUTING chain的raw table中的Trace target不能被trace自身match,因此trace log输出的是匹配raw table built-in chain: PREROUTING的default policy: ACCEPT,num=2(policy和rules整体排序后的序号);在PREROUTING chain的nat表中匹配时,Trace也仅匹配到了default policy,rule 1(target: Docker)没有匹配上;

这里有一点奇怪的是mangle table没有任何输出,即便是default policy的也没有,原因暂不明。

3、bridge decision

根据全图和后续的日志,我们得到了bridge decision的结果:继续在linker layer上处理数据包,一路向右。不过在处理的路径上依旧调用了iptables的rules:

TRACE: eb:filter:FORWARD IN=veth0594f4b OUT=veth57a3dec MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1
TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1
TRACE: filter:DOCKER-USER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1
TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1
TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1
TRACE: filter:FORWARD:rule:4 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1
TRACE: filter:DOCKER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1
TRACE: filter:FORWARD:rule:6 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1

bridge decision决定的依据或则规则是什么呢?《ebtables/iptables interaction on a Linux-based bridge》一文给了我们一些答案:

The bridge's decision for a frame can be one of these:

* bridge it, if the destination MAC address is on another side of the bridge;
* flood it over all the forwarding bridge ports, if the position of the box with the destination MAC is unknown to the bridge;
* pass it to the higher protocol code (the IP code), if the destination MAC address is that of the bridge or of one of its ports;
* ignore it, if the destination MAC address is located on the same side of the bridge.

不过即便按照这几条规则,我依然有一定困惑,那就是真实的处理是:依旧在linker layer,但掺杂了上层网络层的处理规则。

另外,你可能会发现iptables log里MAC值的格式很怪异(比如:MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00),非常long。其实这个MAC值是一个组合:Souce MAC, Destination MAC和 frame type的组合。

02:42:c0:a8:00:03: Destination MAC=00:60:dd:45:67:ea
02:42:c0:a8:00:02: Source MAC=00:60:dd:45:4c:92
08:00 : Type=08:00 (ethernet frame carried an IPv4 datagram)

4、eb:nat:POSTROUTING -> nat:POSTROUTING -> egress(qdisc)

最后packet进入linker layer的POSTROUTING built-in chain:

TRACE: eb:nat:POSTROUTING IN= OUT=veth57a3dec MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1
TRACE: nat:POSTROUTING:policy:2 IN= OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47066 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=1

iptables nat:POSTROUTING没有匹配上docker引擎增加的那条target为DOCKER的rule,于是输出了default policy的日志。

进入到egress(qdisc)后,相当于数据包到了bridge上的另一个slave port(veth57a3dec)上,此时数据包必须被送回网络上,于是进入到容器C2的eth0中。离开了host namespace,我们的日志便追踪不到了。

容器c2因为所在的network namespace是独立于host namespace的,因此有自己的iptables规则(如果未设置,均为默认accept),不受host namespace中的iptables的影响。

5、”消失”的iptable的nat:PREROUTING和nat:POSTROUTING

C2容器回复ping response的路径与request甚为相似,这里一次性将全部日志列出:

TRACE: eb:broute:BROUTING IN=veth57a3dec OUT= MAC source = 02:42:c0:a8:00:03 MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.3 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:nat:PREROUTING IN=veth57a3dec OUT= MAC source = 02:42:c0:a8:00:03 MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.3 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: raw:PREROUTING:policy:2 IN=docker0 OUT= PHYSIN=veth57a3dec MAC=02:42:c0:a8:00:02:02:42:c0:a8:00:03:08:00 SRC=192.168.0.3 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5962 PROTO=ICMP TYPE=0 CODE=0 ID=90 SEQ=1

TRACE: eb:filter:FORWARD IN=veth57a3dec OUT=veth0594f4b MAC source = 02:42:c0:a8:00:03 MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.3 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth57a3dec PHYSOUT=veth0594f4b MAC=02:42:c0:a8:00:02:02:42:c0:a8:00:03:08:00 SRC=192.168.0.3 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5962 PROTO=ICMP TYPE=0 CODE=0 ID=90 SEQ=1
TRACE: filter:DOCKER-USER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth57a3dec PHYSOUT=veth0594f4b MAC=02:42:c0:a8:00:02:02:42:c0:a8:00:03:08:00 SRC=192.168.0.3 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5962 PROTO=ICMP TYPE=0 CODE=0 ID=90 SEQ=1
TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth57a3dec PHYSOUT=veth0594f4b MAC=02:42:c0:a8:00:02:02:42:c0:a8:00:03:08:00 SRC=192.168.0.3 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5962 PROTO=ICMP TYPE=0 CODE=0 ID=90 SEQ=1
TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth57a3dec PHYSOUT=veth0594f4b MAC=02:42:c0:a8:00:02:02:42:c0:a8:00:03:08:00 SRC=192.168.0.3 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5962 PROTO=ICMP TYPE=0 CODE=0 ID=90 SEQ=1
TRACE: filter:FORWARD:rule:3 IN=docker0 OUT=docker0 PHYSIN=veth57a3dec PHYSOUT=veth0594f4b MAC=02:42:c0:a8:00:02:02:42:c0:a8:00:03:08:00 SRC=192.168.0.3 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5962 PROTO=ICMP TYPE=0 CODE=0 ID=90 SEQ=1

TRACE: eb:nat:POSTROUTING IN= OUT=veth0594f4b MAC source = 02:42:c0:a8:00:03 MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.3 IP DST=192.168.0.2, IP tos=0x00, IP proto=1

仔细观察,我们发现虽然与request的路径类似,但依旧有不同:iptable的nat:PREROUTING和nat:POSTROUTING消失了。Why?iptables就是这么设计的。iptables会跟踪connection的state,当一个connection的首个包经过一次后,connection的state由NEW变成了ESTABLISHED;对于ESTABLISHED的connection的后续packets,内核会自动按照该connection的首个包在nat:PREROUTING和nat:POSTROUTING环节的处理方式进行处理,而不再流经这两个链中的nat表逻辑。而ebtables中似乎没有这个逻辑。

后续的ping的第二个、第三个流程也印证了上述设计,这里仅列出ping request packet 2:

TRACE: eb:broute:BROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1
TRACE: eb:nat:PREROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1
TRACE: raw:PREROUTING:policy:2 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47310 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=2
TRACE: eb:filter:FORWARD IN=veth0594f4b OUT=veth57a3dec MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1
TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47310 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=2
TRACE: filter:DOCKER-USER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47310 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=2
TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47310 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=2
TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47310 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=2
TRACE: filter:FORWARD:rule:3 IN=docker0 OUT=docker0 PHYSIN=veth0594f4b PHYSOUT=veth57a3dec MAC=02:42:c0:a8:00:03:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.3 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=47310 DF PROTO=ICMP TYPE=8 CODE=0 ID=90 SEQ=2
TRACE: eb:nat:POSTROUTING IN= OUT=veth57a3dec MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:c0:a8:00:03 proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.3, IP tos=0x00, IP proto=1

全部日志内容请参见:docker-bridge-network-demo-iptables-trace-log.txt文件,这里不赘述。

四、Local Process to Container

img{512x368}

很多”疑难”环节在上面的container to container数据流分析时已经做了解惑,因此后续local process to container和container to external流程将不会再细致描述,说明会略微泛泛一些,不那么细致。

我们在host上执行ping C1三次:

# ping -c 3 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.160 ms
64 bytes from 192.168.0.2: icmp_seq=2 ttl=64 time=0.105 ms
64 bytes from 192.168.0.2: icmp_seq=3 ttl=64 time=0.131 ms

--- 192.168.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.105/0.132/0.160/0.022 ms

1、local process -> routing decision -> iptables OUTPUT chain

ping request数据包从本地的ping process发出,根据目的地址路由后,选择docker0作为OUT设备:

TRACE: raw:OUTPUT:policy:2 IN= OUT=docker0 SRC=192.168.0.1 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18692 DF PROTO=ICMP TYPE=8 CODE=0 ID=30245 SEQ=1 UID=0 GID=0
TRACE: mangle:OUTPUT:policy:1 IN= OUT=docker0 SRC=192.168.0.1 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18692 DF PROTO=ICMP TYPE=8 CODE=0 ID=30245 SEQ=1 UID=0 GID=0
TRACE: nat:OUTPUT:policy:2 IN= OUT=docker0 SRC=192.168.0.1 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18692 DF PROTO=ICMP TYPE=8 CODE=0 ID=30245 SEQ=1 UID=0 GID=0
TRACE: filter:OUTPUT:policy:1 IN= OUT=docker0 SRC=192.168.0.1 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18692 DF PROTO=ICMP TYPE=8 CODE=0 ID=30245 SEQ=1 UID=0 GID=0

奇怪的是这次mangle chain居然有trace log输出:(。

2、进入linker layer:iptables POSTROUTING -> ebtables OUTPUT -> ebtables POSTROUTING

由于是OUT是bridge设备,因此要进入到ebtable中走一遭:

TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=192.168.0.1 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18692 DF PROTO=ICMP TYPE=8 CODE=0 ID=30245 SEQ=1 UID=0 GID=0
TRACE: nat:POSTROUTING:policy:2 IN= OUT=docker0 SRC=192.168.0.1 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=18692 DF PROTO=ICMP TYPE=8 CODE=0 ID=30245 SEQ=1 UID=0 GID=0
TRACE: eb:nat:OUTPUT IN= OUT=veth57a3dec MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.1 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:filter:OUTPUT IN= OUT=veth57a3dec MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.1 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:nat:POSTROUTING IN= OUT=veth57a3dec MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.1 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:nat:OUTPUT IN= OUT=veth0594f4b MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.1 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:filter:OUTPUT IN= OUT=veth0594f4b MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.1 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:nat:POSTROUTING IN= OUT=veth0594f4b MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=192.168.0.1 IP DST=192.168.0.2, IP tos=0x00, IP proto=1

icmp的response和container to container类似,入口走的是linker layer(由于是桥设备),在bridge decision后,走到INPUT chain:

TRACE: eb:broute:BROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:ff:27:17:4d proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.1, IP tos=0x00, IP proto=1
TRACE: eb:nat:PREROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:ff:27:17:4d proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.1, IP tos=0x00, IP proto=1
TRACE: raw:PREROUTING:policy:2 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56535 PROTO=ICMP TYPE=0 CODE=0 ID=30245 SEQ=1
TRACE: mangle:PREROUTING:policy:1 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56535 PROTO=ICMP TYPE=0 CODE=0 ID=30245 SEQ=1
TRACE: eb:filter:INPUT IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:ff:27:17:4d proto = 0x0800 IP SRC=192.168.0.2 IP DST=192.168.0.1, IP tos=0x00, IP proto=1
TRACE: mangle:INPUT:policy:1 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56535 PROTO=ICMP TYPE=0 CODE=0 ID=30245 SEQ=1
TRACE: filter:INPUT:policy:1 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=192.168.0.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=56535 PROTO=ICMP TYPE=0 CODE=0 ID=30245 SEQ=1

以上我们可以与到非桥设备的ping做比对,我们在host上ping 另外一个LAN中的host:

# ping -c 1 10.28.61.30
PING 10.28.61.30 (10.28.61.30) 56(84) bytes of data.
64 bytes from 10.28.61.30: icmp_seq=1 ttl=57 time=1.09 ms

--- 10.28.61.30 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.093/1.093/1.093/0.000 ms

得到的trace log如下:

icmp request:

TRACE: raw:OUTPUT:policy:2 IN= OUT=eth0 SRC=10.171.77.0 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=4494 DF PROTO=ICMP TYPE=8 CODE=0 ID=30426 SEQ=1 UID=0 GID=0
TRACE: mangle:OUTPUT:policy:1 IN= OUT=eth0 SRC=10.171.77.0 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=4494 DF PROTO=ICMP TYPE=8 CODE=0 ID=30426 SEQ=1 UID=0 GID=0
TRACE: nat:OUTPUT:policy:2 IN= OUT=eth0 SRC=10.171.77.0 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=4494 DF PROTO=ICMP TYPE=8 CODE=0 ID=30426 SEQ=1 UID=0 GID=0
TRACE: filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=10.171.77.0 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=4494 DF PROTO=ICMP TYPE=8 CODE=0 ID=30426 SEQ=1 UID=0 GID=0
TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=10.171.77.0 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=4494 DF PROTO=ICMP TYPE=8 CODE=0 ID=30426 SEQ=1 UID=0 GID=0
TRACE: nat:POSTROUTING:policy:2 IN= OUT=eth0 SRC=10.171.77.0 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=4494 DF PROTO=ICMP TYPE=8 CODE=0 ID=30426 SEQ=1 UID=0 GID=0

icmp response:

TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=10.171.77.0 LEN=84 TOS=0x00 PREC=0x00 TTL=57 ID=61118 PROTO=ICMP TYPE=0 CODE=0 ID=30426 SEQ=1
TRACE: mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=10.171.77.0 LEN=84 TOS=0x00 PREC=0x00 TTL=57 ID=61118 PROTO=ICMP TYPE=0 CODE=0 ID=30426 SEQ=1
TRACE: mangle:INPUT:policy:1 IN=eth0 OUT= MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=10.171.77.0 LEN=84 TOS=0x00 PREC=0x00 TTL=57 ID=61118 PROTO=ICMP TYPE=0 CODE=0 ID=30426 SEQ=1
TRACE: filter:INPUT:policy:1 IN=eth0 OUT= MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=10.171.77.0 LEN=84 TOS=0x00 PREC=0x00 TTL=57 ID=61118 PROTO=ICMP TYPE=0 CODE=0 ID=30426 SEQ=1

可以对照着全图看出在request出去时,发现OUT设备不是bridge,直接走network layer的iptables rules,并从xfrm lookup出去,走到egress(qdisc); response回来时,进行bridge check后,发现IN设备eth0不是bridge,因此直接上到network layer,走iptable chain rules到local process。ebtable的log一行也没有输出。

后续的两个icmp request&response大致相同,并且依旧不走nat PREROUTING和nat POSTROUTING,因为不再是NEW connection。

五、Container to External

img{512x368}

我们在c1 容器中ping 外部的一个节点三次:

# docker exec c1 ping -c 3 10.28.61.30
PING 10.28.61.30 (10.28.61.30) 56(84) bytes of data.
64 bytes from 10.28.61.30: icmp_seq=1 ttl=56 time=1.32 ms
64 bytes from 10.28.61.30: icmp_seq=2 ttl=56 time=1.30 ms
64 bytes from 10.28.61.30: icmp_seq=3 ttl=56 time=1.21 ms

--- 10.28.61.30 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.219/1.280/1.323/0.060 ms

1、start -> bridgecheck -> linker layer

和Container to Container的开端很类似,在bridge check后,数据流进入linker layer(docker0 is a bridge),并在该层进行iptables PREROUTING rules的处理,直到bridge decision之前:

TRACE: eb:broute:BROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:ff:27:17:4d proto = 0x0800 IP SRC=192.168.0.2 IP DST=10.28.61.30, IP tos=0x00, IP proto=1
TRACE: eb:nat:PREROUTING IN=veth0594f4b OUT= MAC source = 02:42:c0:a8:00:02 MAC dest = 02:42:ff:27:17:4d proto = 0x0800 IP SRC=192.168.0.2 IP DST=10.28.61.30, IP tos=0x00, IP proto=1
TRACE: raw:PREROUTING:policy:2 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: mangle:PREROUTING:policy:1 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: nat:PREROUTING:policy:2 IN=docker0 OUT= PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1

2、ebtable filter:INPUT -> routing decision -> iptables FORWARD

目的地址为外部host ip,需要三层介入转发,于是数据包经由eb:filter:INPUT向上走到达network layer的routing decision,根据路由表,将包转发到eth0:

TRACE: mangle:FORWARD:policy:1 IN=docker0 OUT=eth0 PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=eth0 PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: filter:DOCKER-USER:return:1 IN=docker0 OUT=eth0 PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=eth0 PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=eth0 PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=eth0 PHYSIN=veth0594f4b MAC=02:42:ff:27:17:4d:02:42:c0:a8:00:02:08:00 SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1

3、iptables nat:POSTROUTING match rule 1

由于要流出到主机外,因此在最后iptables nat:POSTROUTING中,数据包匹配到rule 1,即做MASQUERADE,将数据包源地址更换为host ip:10.171.77.0。

TRACE: mangle:POSTROUTING:policy:1 IN= OUT=eth0 PHYSIN=veth0594f4b SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1
TRACE: nat:POSTROUTING:rule:1 IN= OUT=eth0 PHYSIN=veth0594f4b SRC=192.168.0.2 DST=10.28.61.30 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=57351 DF PROTO=ICMP TYPE=8 CODE=0 ID=94 SEQ=1

4、iptables prerouting、forward、postrouting -> ebtabls output、postrouting

返回的应答由于IN设备为eth0,因此直接上到network layer进行iptable chain的处理。在路由后,OUT设备为docker0(bridge设备),因此在最后的环节需要下降到linker layer做output和postrouting处理:

TRACE: raw:PREROUTING:policy:2 IN=eth0 OUT= MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=10.171.77.0 LEN=84 TOS=0x00 PREC=0x00 TTL=57 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: mangle:PREROUTING:policy:1 IN=eth0 OUT= MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=10.171.77.0 LEN=84 TOS=0x00 PREC=0x00 TTL=57 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: mangle:FORWARD:policy:1 IN=eth0 OUT=docker0 MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: filter:FORWARD:rule:1 IN=eth0 OUT=docker0 MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: filter:DOCKER-USER:return:1 IN=eth0 OUT=docker0 MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: filter:FORWARD:rule:2 IN=eth0 OUT=docker0 MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: filter:DOCKER-ISOLATION:return:1 IN=eth0 OUT=docker0 MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: filter:FORWARD:rule:3 IN=eth0 OUT=docker0 MAC=00:16:3e:06:3a:3a:00:2a:6a:aa:12:7c:08:00 SRC=10.28.61.30 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: mangle:POSTROUTING:policy:1 IN= OUT=docker0 SRC=10.28.61.30 DST=192.168.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=56 ID=58706 PROTO=ICMP TYPE=0 CODE=0 ID=94 SEQ=1
TRACE: eb:nat:OUTPUT IN= OUT=veth0594f4b MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=10.28.61.30 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:filter:OUTPUT IN= OUT=veth0594f4b MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=10.28.61.30 IP DST=192.168.0.2, IP tos=0x00, IP proto=1
TRACE: eb:nat:POSTROUTING IN= OUT=veth0594f4b MAC source = 02:42:ff:27:17:4d MAC dest = 02:42:c0:a8:00:02 proto = 0x0800 IP SRC=10.28.61.30 IP DST=192.168.0.2, IP tos=0x00, IP proto=1

后续的请求和应答基本类似,少的还是nat PREROUTING和nat POSTROUTING,因为不再是NEW connection。

六、小结

个人赶脚:iptables的规则还是太复杂了,再加上bridge的ebtable规则,让人有些眼花缭乱。尤其是kube-proxy的规则又与docker的规则鞣合在一起,iptables的rules列表就显得更为冗长和复杂了。但目前kube-proxy稳定版依然以iptables为主要实现机制,不过kube-proxy对ipvs的支持也已经在路上了(kubernetes 1.8中ipvs处于alpha阶段),希望后续我们能有更多的选择。

此次实验全部日志内容参见:docker-bridge-network-demo-iptables-trace-log.txt文件

七、参考资料


微博:@tonybai_cn
微信公众号:iamtonybai
github.com: https://github.com/bigwhite

如发现本站页面被黑,比如:挂载广告、挖矿等恶意代码,请朋友们及时联系我。十分感谢! Go语言第一课 Go语言精进之路1 Go语言精进之路2 商务合作请联系bigwhite.cn AT aliyun.com

欢迎使用邮件订阅我的博客

输入邮箱订阅本站,只要有新文章发布,就会第一时间发送邮件通知你哦!

这里是 Tony Bai的个人Blog,欢迎访问、订阅和留言! 订阅Feed请点击上面图片

如果您觉得这里的文章对您有帮助,请扫描上方二维码进行捐赠 ,加油后的Tony Bai将会为您呈现更多精彩的文章,谢谢!

如果您希望通过微信捐赠,请用微信客户端扫描下方赞赏码:

如果您希望通过比特币或以太币捐赠,可以扫描下方二维码:

比特币:

以太币:

如果您喜欢通过微信浏览本站内容,可以扫描下方二维码,订阅本站官方微信订阅号“iamtonybai”;点击二维码,可直达本人官方微博主页^_^:
本站Powered by Digital Ocean VPS。
选择Digital Ocean VPS主机,即可获得10美元现金充值,可 免费使用两个月哟! 著名主机提供商Linode 10$优惠码:linode10,在 这里注册即可免费获 得。阿里云推荐码: 1WFZ0V立享9折!


View Tony Bai's profile on LinkedIn
DigitalOcean Referral Badge

文章

评论

  • 正在加载...

分类

标签

归档



View My Stats