Compose v3.2 endpoint_mode not working with dnsrr

[mbentley]

Description
When deploying an app with docker stack deploy that has endpoint_mode set to dnsrr, it fails to create the service correctly.

Steps to reproduce the issue:

1. Deploy a v3.2 compose file via docker stack deploy:

version: "3.2"

services:

  elasticsearch:
    command: elasticsearch -Enetwork.host=0.0.0.0 -Ediscovery.zen.ping.unicast.hosts=elasticsearch
    environment:
      ES_JAVA_OPTS: -Xms1g -Xmx1g
    image: elasticsearch:5
    ulimits:
      memlock: -1
      nofile:
        hard: 65536
        soft: 65536
      nproc: 65538
    volumes:
      - esdata:/usr/share/elasticsearch/data
    networks:
      - elk
    deploy:
      mode: replicated
      replicas: 1
      endpoint_mode: dnsrr
    healthcheck:
      test: curl -s http://localhost:9200/_cluster/health?local=true | grep green
      interval: 30s
      retries: 3

volumes:
  esdata:
    driver: local

networks:
  elk:
    driver: overlay

2. docker stack deploy -c docker-compose.yml elasticsearch
3. See that the endpoint isn't correct:

$ docker service inspect elasticsearch_elasticsearch | jq '.[]|.Spec.EndpointSpec'
{
  "Mode": "vip"
}

Describe the results you received:
It should use dnsrr but doesn't.

From the dockerd logs in debug:
docker stack deploy:

"form data: {\"EndpointSpec\":{},\"Labels\":{\"com.docker.stack.namespace\":\"hromis\"},\"Mode\":{\"Replicated\":{\"Replicas\":1}},\"Name\":\"hromis_elasticsearch\",\"Networks\":[
{\"Aliases\":[\"elasticsearch\"],\"Target\":\"hromis_elk\"}],\"TaskTemplate\":{\"ContainerSpec\":{\"Args\":[\"elasticsearch\",\"-Enetwork.host=0.0.0.0\",\"-Ediscovery.zen.ping.unicast.hosts=elasticsearch\"],\"Env\":[\"ES_JAVA_OPTS=-Xms1g-
Xmx1g\"],\"Healthcheck\":{\"Interval\":3e+10,\"Retries\":3,\"Test\":[\"CMD-SHELL\",\"curl -s http://localhost:9200/_cluster/health?local=true | grep green\"]},\"Image\":\"elasticsearch:5\",\"Labels\":{\"com.docker.stack.namespace\":\"hrom
is\"},\"Mounts\":[{\"Source\":\"hromis_esdata\",\"Target\":\"/usr/share/elasticsearch/data\",\"Type\":\"volume\",\"VolumeOptions\":{\"DriverConfig\":{\"Name\":\"local\"},\"Labels\":{\"com.docker.stack.namespace\":\"hromis\"}}}]},\"ForceUp
date\":0,\"Placement\":{},\"Resources\":{}}}"

docker service create:
(Using a shorter example of docker service create --name elasticsearch --env ES_JAVA_OPTS="-Xms1g -Xmx1g" --replicas 1 --endpoint-mode=dnsrr elasticsearch:5 elasticsearch -Enetwork.host=0.0.0.0 -Ediscovery.zen.ping.unicast.hosts=elasticsearch)

"form data: {\"EndpointSpec\":{\"Mode\":\"dnsrr\"},\"Labels\":{},\"Mode\":{\"Replicated\":{\"Replicas\":1}},\"Name\":\"elasticsearch\",\"RollbackConfig\":{\"FailureAction\":\"pause\",\"MaxFailureRatio\":0,\"Parallelism\":1},\"TaskTemplate\":{\"ContainerSpec\":{\"Args\":[\"elasticsearch\",\"-Enetwork.host=0.0.0.0\",\"-Ediscovery.zen.ping.unicast.hosts=elasticsearch\"],\"DNSConfig\":{},\"Env\":[\"ES_JAVA_OPTS=-Xms1g -Xmx1g\"],\"Image\":\"elasticsearch:5\"},\"ForceUpdate\":0,\"Placement\":{},\"Resources\":{\"Limits\":{},\"Reservations\":{}},\"RestartPolicy\":{}},\"UpdateConfig\":{\"FailureAction\":\"pause\",\"MaxFailureRatio\":0,\"Parallelism\":1}}"

Describe the results you expected:
dnsrr should be used as the compose file specifies

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

$ docker version
Client:
 Version:      17.04.0-ce
 API version:  1.28
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Wed Apr  5 23:33:17 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.04.0-ce
 API version:  1.28 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Wed Apr  5 23:33:17 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

$ docker info
Containers: 3
 Running: 0
 Paused: 0
 Stopped: 3
Images: 217
Server Version: 17.04.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: dkybeesgz1xbhl8v7cabihzt1
 Is Manager: true
 ClusterID: momqttjyo3qo49jj5012xies5
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 192.168.100.10
 Manager Addresses:
  192.168.100.10:2377
Runtimes: runc
Default Runtime: runc
Init Binary:
containerd version: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.59
Operating System: Alpine Linux v3.5
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.954GiB
Name: alpine
ID: ANNE:4FUU:XCPW:GM7X:3OBN:E4KZ:VY63:ZB6H:NS3W:MAOI:54XZ:RBHL
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 31
 Goroutines: 141
 System Time: 2017-04-12T16:50:05.396247149-05:00
 EventsListeners: 0
Username: mbentley
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

/cc @dnephin @ahromis

[dnephin]

This was fixed in #32333, will be in 17.05

[cr0hn]

This error still exits:

I tried the fixed endpoint_mode into version 17.05 and it raises the error:

> docker stack deploy -c /tmp/GNKqMUPcJDdkdRamXojAe.yml asdf
endpoint_mode Additional property endpoint_mode is not allowed

Compose content

networks:
  IDSFREE_SCAN_xkCXb_hSBwq: {external: true}
  mynet: {internal: true}
  registry: {driver: overlay, internal: true}
  traefik: {internal: true}
services:
  docker-empty:
    deploy:
      endpoint_mode: dnsrr
      replicas: 1
      restart_policy: {condition: on-failure}
    image: registry:2
    networks: [IDSFREE_SCAN_xkCXb_hSBwq]
  docker-registry:
    deploy:
      endpoint_mode: dnsrr
      replicas: 1
      restart_policy: {condition: on-failure}
    image: registry:2
    networks: [registry, IDSFREE_SCAN_xkCXb_hSBwq]
  redis:
    deploy: {endpoint_mode: dnsrr}
    image: redis
    networks: [IDSFREE_SCAN_xkCXb_hSBwq]
  registry-ui:
    deploy:
      endpoint_mode: dnsrr
      labels: [traefik.port=80, 'traefik.frontend.rule=Host:registry.91k.es', traefik.docker.network=traefik]
      replicas: 1
      restart_policy: {condition: on-failure}
    environment: [ENV_DOCKER_REGISTRY_HOST=docker-registry, ENV_DOCKER_REGISTRY_PORT=5000]
    image: konradkleine/docker-registry-frontend:v2
    networks: [registry, traefik, IDSFREE_SCAN_xkCXb_hSBwq]
version: '3'

Docker version

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 10
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 79
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: 30znk5pf8ttrc3c1icrkpdlmf
 Is Manager: true
 ClusterID: n1dbqd4aibhkhry76rk11fbhl
 Managers: 1
 Nodes: 2
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 172.16.1.227
 Manager Addresses:
  172.16.1.227:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-1016-aws
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.952GiB
Name: idsFreeMaster
ID: 4IPU:EKIY:XGOI:OQA7:YQN3:DOBO:D4SG:35Y6:WJ4M:IQOK:73WW:QASP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[ggogel]

I just experienced this bug. If you set endpoint_mode=dnsrr before replicas: 2 it will somehow use both! When I run nslookup I receive both IPs of the containers and the virtual IP.

"Mode": {
                "Replicated": {
                    "Replicas": 4
                }
            },
            "EndpointSpec": {
                "Mode": "vip"
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "dnsrr"
            }
        },

It sets it correctly when ``endpoint_mode: dnsrris defined **after**replicas: 2`

"Mode": {
                "Replicated": {
                    "Replicas": 4
                }
            },
            "EndpointSpec": {
                "Mode": "dnsrr"
            }
        },
        "Endpoint": {
            "Spec": {
                "Mode": "dnsrr"
            }

I needed to remove the stack completely and redeploy it to make the changes actually apply.