Multi Container Host Networking: Seamless Communication with VxLAN and Docker
In this hands-on demo, we will explore how to set up communication between two hosts using Virtual Extensible LAN (VxLAN) and Docker. The goal is to create a VxLAN overlay network tunnel between the hosts' containers, allowing them to communicate with each other. Let's dive into the steps involved in this process.
What are we going to cover in this hands-on demo?
Creating two VMs and installing Docker: We will use two virtual machines (VMs) and install Docker to run the containers.
Creating separate subnets and assigning static IP addresses: To simplify the setup, we'll create separate subnets for each VM and assign static IP addresses.
Creating a VxLAN bridge: We'll utilize the Linux "ip link vxlan" feature to create a VxLAN bridge.
Binding the VxLAN to the Docker bridge: We'll bind the VxLAN to the Docker bridge to establish the tunnel.
Verifying communication between containers: Finally, we'll test the communication between containers on different hosts.
Let's get started! To facilitate effective communication between hosts, we need to deploy two VMs using any hypervisor or virtualization technology. It is crucial to ensure that both VMs are connected to the same network.
Step 1: Creating the VMs
We begin by creating two Lubuntu VMs using UTM / Multipass. These VMs will serve as the hosts for our containers.
Step 2: Installing necessary tools and Docker
In this step, we will install several necessary tools and Docker on our Ubuntu VMs. Here's a brief overview of the tools we are going to install:
net-tools: This package includes important network management tools such asifconfig,netstat,route, etc.iputils-ping: This package provides thepingcommand, which is used to test the reachability of a network host.bridge-utils: This package provides utilities for configuring Ethernet bridging on Linux.
To install these tools and Docker, execute the following commands:
apt-get update
apt-get install net-tools
apt-get install iputils-ping
apt-get install bridge-utils
apt-get install -y docker.io
Additionally, if the IP address of one of the VMs is the same as the other, you'll need to modify it to avoid conflicts. For example, you can change the IP address of one of the VMs to 192.168.64.7/24 using the following commands:
ifconfig eth0 192.168.64.7/24
route add default gw 192.168.64.1 eth0
To configure DNS resolution, open the resolv.conf file:
nano /etc/resolv.conf
Delete the existing content and add the following lines:
nameserver 8.8.8.8
nameserver 8.8.4.4
Now, let's proceed to set up the containers and their communication.
Step 3: Running Docker Containers on the VxLAN Network Now for host1(amicable-hyena),
# create a separate docker bridge network
docker network create --subnet 172.18.0.0/16 vxlan-net
# list all networks in docker
docker network ls
# The output should include the newly created vxlan-net network.
NETWORK ID NAME DRIVER SCOPE
53be5ce8e682 bridge bridge local
757eb3a14b73 host host local
c1c0e4f01fa6 none null local
08bdd2dc3a82 vxlan-net bridge local
# Check interfaces
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 36:db:7c:3f:11:58 brd ff:ff:ff:ff:ff:ff
inet 192.168.64.9/24 brd 192.168.64.255 scope global dynamic enp0s2
valid_lft 85395sec preferred_lft 85395sec
inet6 fde4:cfbc:2cca:cd78:34db:7cff:fe3f:1158/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 2591923sec preferred_lft 604723sec
inet6 fe80::34db:7cff:fe3f:1158/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:aa:59:e4:e3 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: br-08bdd2dc3a82: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:4b:42:67:6b brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-08bdd2dc3a82
valid_lft forever preferred_lft forever
For Host 2(affluent-toad),
docker network create --subnet 172.18.0.0/16 vxlan-net
docker network ls
cdc5ceed0de2 bridge bridge local
11ed966e63df host host local
5ea8856ad30c none null local
55316818ea9f vxlan-net bridge local
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether f2:5f:8d:d3:9f:c9 brd ff:ff:ff:ff:ff:ff
inet 192.168.64.10/24 brd 192.168.64.255 scope global dynamic enp0s2
valid_lft 85276sec preferred_lft 85276sec
inet6 fde4:cfbc:2cca:cd78:f05f:8dff:fed3:9fc9/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 2591908sec preferred_lft 604708sec
inet6 fe80::f05f:8dff:fed3:9fc9/64 scope link
valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:b3:cb:9a:b9 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
4: br-55316818ea9f: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:2c:83:cd:df brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-55316818ea9f
valid_lft forever preferred_lft forever
Step 4: Run docker container Let's run docker container on top of newly created docker bridge network and try to ping docker bridge
For Host 1,
# running alpine container with "sleep 3000" and a static ip
docker run -d --net vxlan-net --ip 172.18.0.11 alpine sleep 3000
# check the container running or not
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d2b96eddea7a alpine "sleep 3000" About a minute ago Up About a minute funny_jepsen
# check the IPAddress to make sure that the ip assigned properly
docker inspect d2 | grep IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "",
"IPAddress": "172.18.0.11",
# ping the docker bridge ip to see whether the traffic can pass
ping 172.18.0.1 -c 2
PING 172.18.0.1 (172.18.0.1) 56(84) bytes of data.
64 bytes from 172.18.0.1: icmp_seq=1 ttl=64 time=0.166 ms
64 bytes from 172.18.0.1: icmp_seq=2 ttl=64 time=0.050 ms
--- 172.18.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1018ms
rtt min/avg/max/mdev = 0.050/0.108/0.166/0.058 ms
For Host 2,
docker run -d --net vxlan-net --ip 172.18.0.12 alpine sleep 3000
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
9d2e17598b8b alpine "sleep 3000" 3 seconds ago Up 2 seconds eloquent_black
docker inspect 9d | grep IPAddress
"SecondaryIPAddresses": null,
"IPAddress": "",
"IPAddress": "172.18.0.12",
ping 172.18.0.1 -c 2
PING 172.18.0.1 (172.18.0.1) 56(84) bytes of data.
64 bytes from 172.18.0.1: icmp_seq=1 ttl=64 time=0.193 ms
64 bytes from 172.18.0.1: icmp_seq=2 ttl=64 time=0.067 ms
--- 172.18.0.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1017ms
rtt min/avg/max/mdev = 0.067/0.130/0.193/0.063 ms
Step 5: Testing Communication Now, let's access one of the running containers and test communication between hosts. Note that containers within the same host should communicate, but container-to-container communication between hosts will fail at this stage because there is no tunnel or anything to carry the traffic.
docker exec -it d2 sh
apk update
apk add net-tools
apk add iputils-ping
ping 172.18.0.12 -c 2
ping: -c: Try again
ping -c 2 192.168.64.10
PING 192.168.64.10 (192.168.64.10) 56(84) bytes of data.
64 bytes from 192.168.64.10: icmp_seq=1 ttl=63 time=3.93 ms
64 bytes from 192.168.64.10: icmp_seq=2 ttl=63 time=0.967 ms
--- 192.168.64.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.967/2.447/3.928/1.480 ms
Step 6: Creating the VxLAN Tunnel To establish communication between the containers, we need to create a VxLAN tunnel and attach it to the Docker bridge. Make sure the VNI (Virtual Network Identifier) ID is the same for both hosts.
For Host 1,
# check the bridges list on the hosts
brctl show
bridge name bridge id STP enabled interfaces
br-08bdd2dc3a82 8000.02424b42676b no
docker0 8000.0242aa59e4e3 no
# create a vxlan
# 'vxlan-demo' is the name of the interface, type should be vxlan
# VNI ID is 100
# dstport should be 4789 which a udp standard port for vxlan communication
# 192.168.64.10 is the ip of another host
ip link add vxlan-demo type vxlan id 100 remote 192.168.64.10 dstport 4789 dev enp0s2
# check interface list if the vxlan interface created
ip a | grep vxla
7: vxlan-demo: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
# make the interface up
ip link set vxlan-demo up
# now attach the newly created vxlan interface to the docker bridge we created
brctl addif br-08bdd2dc3a82 vxlan-demo
# check the route to ensure everything is okay. here '172.18.0.0' part is our concern part.
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.64.1 0.0.0.0 UG 100 0 0 enp0s2
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-08bdd2dc3a82
192.168.64.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s2
192.168.64.1 0.0.0.0 255.255.255.255 UH 100 0 0 enp0s2
For Host 2,
brctl show
bridge name bridge id STP enabled interfaces
br-55316818ea9f 8000.02422c83cddf no
docker0 8000.0242b3cb9ab9
ip link add vxlan-demo type vxlan id 100 remote 192.168.64.9 dstport 4789 dev enp0s2
ip a | grep vxlan
7: vxlan-demo: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
ip link set vxlan-demo up
brctl addif br-55316818ea9f vxlan-demo
route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.64.1 0.0.0.0 UG 100 0 0 enp0s2
172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0
172.18.0.0 0.0.0.0 255.255.0.0 U 0 0 0 br-55316818ea9f
192.168.64.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s2
192.168.64.1 0.0.0.0 255.255.255.255 UH 100 0 0 enp0s2
Step 7: Testing Communication between Containers Now that the VxLAN overlay network tunnel has been created, let's test the communication between the containers on different hosts.
docker exec -it a9 bash
ping 192.168.64.10 -c 2
You can find result of ping from below images. [Note: Re ran docker container in both host as previous container already ran for 3000 seconds]
You should see successful ping responses, indicating that communication between the containers on different hosts is now established.
Congratulations! You have successfully set up communication between hosts using VxLAN and Docker.
Reference: The demonstration code and steps mentioned in this blog post were adapted from the following GitHub repository: vxlan-docker-hands-on
Feel free to explore the repository for more in-depth details and examples.
Thank you for reading this blog post, and I hope you found it informative and useful.