There are many different ways to connect docker containers running on different hosts. Recently I’m playing a bit with SDN hence I tried to use openvswitch for that purpose. The situation in my case is that I have two following hosts:

  1. host docker-A: debian (stretch/sid), docker 1.6.1, IP: 10.168.113.51
  2. host docker-B: ubuntu (15.04 vivid), docker 1.5.0, IP: 192.168.121.166

Each host is running in different part of the network though both hosts are fully reachable for each other.

To start I install openvswitch on both hosts:

# apt-get install openvswitch-switch

Then on both hosts I start a “debian” container that is used for experimenting:

# docker run -ti --name=debian -d debian /bin/bash

The idea is to connect both containers in L2 so they can ping each other. The concept is outlined in following diagram:

docker-openvswitch

To configure openvswitch I need a proper namespace configuration for docker containers. Unfortunately docker is not making entries in /var/netns. To fix that i created following script:

#!/bin/sh
pid=$(docker inspect -f '{{.State.Pid}}' $1)
name=$(docker inspect -f '{{.Name}}' $1 | cut -c2-)
#hostname=$(docker inspect -f '{{.Config.Hostname}}' $1)
#id=$(docker inspect -f '{{.Id}}' $1| cut -c1-12)
mkdir -p /var/run/netns
ln -sf /proc/$pid/ns/net /var/run/netns/$name

I saved it as docker-netns and run on both hosts the same command:

 # ./docker-netns debian

The script created proper links in /var/netns hence let me use ip netns command:

# ip netns
debian

Having network namespaces in place I was able to configure the openvswicth and connect it across both hosts. I run this on docker-A:

ip link add veth-debian type veth peer name eth-debian
ip link set eth-debian netns debian
ip netns exec debian ip link set dev eth-debian name eth1
ip link set veth-debian up
ip netns exec debian ip link set eth1 up 
ip netns exec debian ip addr add 10.0.0.1/24 dev eth1

ovs-vsctl add-br br-docker
ovs-vsctl add-port br-docker veth-debian
ovs-vsctl add-port br-docker gre0 -- set interface gre0 type=gre options:remote_ip=192.168.121.166

And the same commands with different IPs in docker-B:

ip link add veth-debian type veth peer name eth-debian
ip link set eth-debian netns debian
ip netns exec debian ip link set dev eth-debian name eth1
ip link set veth-debian up
ip netns exec debian ip link set eth1 up 
ip netns exec debian ip addr add 10.0.0.2/24 dev eth1

ovs-vsctl add-br br-docker
ovs-vsctl add-port br-docker veth-debian
ovs-vsctl add-port br-docker gre0 -- set interface gre0 type=gre options:remote_ip=10.168.113.51

Now I’m able to attach to debian containers on both hosts and ping each other container.
On docker-A:

# docker attach debian

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
17: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:05 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.5/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:5/64 scope link 
       valid_lft forever preferred_lft forever
19: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether da:28:f6:dd:e6:3a brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::d828:f6ff:fedd:e63a/64 scope link 
       valid_lft forever preferred_lft forever

root@39f3cf37bf8d:/# ping -c1 10.0.0.2
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: icmp_seq=0 ttl=64 time=2.584 ms
--- 10.0.0.2 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.584/2.584/2.584/0.000 ms

And the same on host docker-B:

# docker attach debian

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
26: eth0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:11:00:06 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.6/16 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:6/64 scope link 
       valid_lft forever preferred_lft forever
28: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether f6:a1:0a:c8:76:3b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::f4a1:aff:fec8:763b/64 scope link 
       valid_lft forever preferred_lft forever

root@a114e84f0100:/# ping -c1 10.0.0.1
PING 10.0.0.1 (10.0.0.1): 48 data bytes
56 bytes from 10.0.0.1: icmp_seq=0 ttl=64 time=2.667 ms
--- 10.0.0.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max/stddev = 2.667/2.667/2.667/0.000 ms

A side note: to detach from the container without stopping the container use ^+p, ^+q sequence. You’re out of luck if your window manager (or other fancy desktop environment) hijacks this combination.

How does it work?

Small explanation of few commands:

ip link add veth-debian type veth peer name eth-debian

Cerates a veth interface pair: veth-debian and eth-debian.

ip link set eth-debian netns debian

Puths eth-debian end into netns debian associated with debian container.

ip netns exec debian ip link set dev eth-debian name eth1

Sets the eth-debian end a new name eth1. Since it resides inside different namespace (associated with debian docker container) this name has to be unique within that namespace. As there is already a eth0 inside docker container I chose eth1 name.

ip link set veth-debian up

Bring up the vet-debian end.

ip netns exec debian ip link set eth1 up 
ip netns exec debian ip addr add 10.0.0.1/24 dev eth1

Bring up the veth inside the container and assign an address to it. We have to do that from the host as default capabilities does not permit network management from within the container.

ovs-vsctl add-br br-docker
ovs-vsctl add-port br-docker veth-debian

Create br-docker switch (bridge) and add a port to it. This port is connected to vet-debian interface. This way we connected the eth1 inside container to openvswicth br-docker.

ovs-vsctl add-port br-docker gre0 -- set interface gre0 type=gre options:remote_ip=192.168.121.166

Add another port to br-docekr switch. This port is associated with gre tunnel that is configured to connect to remote host.
Having simmetrical configuration on both hosts we’re able to interconnect docker containers directly in virtual L2 a de facto overlay network.

Advertisements