Network

When job is interactive or requests to open some ports in the container, GoDocker executors must allocate an IP address to the container and define some access rules to make those ports reachable.

Kubernetes

Kubernetes provide an IP address per container. It is possible to access container ports using their original port (22 for SSH for example) as long as kubernetes allocated IP are reachable via local network.

Mesos / Docker swarm

Mesos and Docker makes use of port mapping. Containers are launched behind a bridge, with local IP address not reachable from other hosts. Port mapping maps job ports (ssh or job custom ports) to ports available on host (internal docker port 22 mapped to host port 30000 for example). Host ports can be reached from other hosts on local network.

GoDocker API and web interface gives the mapping of each port of the job/container.

Public/Internet access

Compute nodes will usually have local network addresses (as well as Kubernetes IPs). This means that job/container ports are not directly reachable from external network.

To do so, users should connect to a gateway connected to both networks (user ssh to gateway then ssh to the container). It is also possible to create a dynamic proxy gateway with a watcher plugin.

Watcher plugins are triggered during the life cycle of the job , with all job information. A plugin could for example regenerate/reload dynamically a HAProxy or a nginx proxy with container ports to provide remote access to the containers.

CNI networks

With CNI network plugins support (go-docker >= 1.2), executors can give an IP to the job/containers, avoiding port mapping. See README.md for more info.

Weave and Calico are supported via network plugins.

System (docker, mesos, ...) needs to be setup for CNI support with a default network available that should match the GoDocker public network.

More info for Mesos:

https://github.com/apache/mesos/blob/master/docs/cni.md

Simple port mapping

In /etc/cni/net.d, create a bridge.conf and a portmapper.conf

# cat /etc/cni/net.d/bridge.conf

{
"name": "cni-test",
"type": "bridge",
"bridge": "mesos-cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
    "type": "host-local",
    "subnet": "192.168.0.0/16",
    "routes": [
    { "dst":
      "0.0.0.0/0" }
    ]
  }
}

# cat /etc/cni/net.d/portmapper.conf

{
  "name" : "port-mapper-test",
  "type" : "mesos-cni-port-mapper",
  "excludeDevices" : ["mesos-cni0"],
  "chain": "MESOS-TEST-PORT-MAPPER",
  "delegate": {
      "type": "bridge",
      "bridge": "mesos-cni0",
      "isGateway": true,
      "ipMasq": true,
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.0.0/16",
        "routes": [
        { "dst":
          "0.0.0.0/0" }
        ]
      }
  }
}

In cni directory, put the following cni binaries: bridge ,host-local, mesos-cni-port-mapper

bridge and host-local come from cni: https://github.com/containernetworking/plugins/releases,

mesos-cni-port-mapper is included in mesos, just copy the binary in cni binaries directory.

In go-d.ini set mesos/port_mapper_network_name field to port-mapper-test

Weave

To access to the container (ssh or other open ports), Weave network must be exposed on the user gateway (via "weave expose").

Calico

To access to the container (ssh or other open ports), Calico network must be exposed on the user gateway. 

See more info at https://github.com/projectcalico/calico-containers/blob/master/docs/FAQ.md and https://github.com/projectcalico/calico-containers/blob/v0.21.0/README.md

Example:

sudo ./calicoctl profile calico-net-1 rule add inbound allow tcp to ports 22
sudo ./calicoctl pool add 192.168.0.0/16 --nat-outgoing

Mesos configuration

Mesos slave must be on reachable IP address (not localhost) and must add isolators: cgroups/cpu,cgroups/mem,network/cni

Usual setup will be for isolation: filesystem/linux,docker/runtime,cgroups/cpu,cgroups/mem,network/cni

Test on mesos

sudo mesos-execute --command="ifconfig && sleep 30" -docker_image=centos:latest --master=127.0.0.1:5050 --name=test0 --networks=calico-net-1

$ sudo ./calicoctl endpoint show --detailed
+----------+-----------------+--------------------------------------+----------------------------------+-----------------+-------------------+--------------+--------+
| Hostname | Orchestrator ID | Workload ID | Endpoint ID | Addresses | MAC | Profiles | State |
+----------+-----------------+--------------------------------------+----------------------------------+-----------------+-------------------+--------------+--------+
| tifenn | cni | 838fbea7-e87b-4f53-a85a-d10e86f5f67a | 8fbbdcb06a9311e68a99fc3fdb46dee9 | 192.168.0.17/32 | de:e3:68:26:5d:b6 | calico-net-1 | active |
+----------+-----------------+--------------------------------------+----------------------------------+-----------------+-------------------+--------------+--------+