Docker Service Update Failed: How to Modify Running Swarm Services

# Docker Service Update Failed: How to Modify Running Swarm Services

You tried to update a Swarm service, but it failed:

Error response from daemon: service myapp update failed: no suitable node

Or:

Error response from daemon: rpc error: code = 2 desc = image not found

Or updates stuck with tasks pending:

docker service ps myapp
ID      NAME        NODE        DESIRED STATE   CURRENT STATE   ERROR
abc     myapp.1     node-1      Running         Running         
def     myapp.2     node-2      Running         Pending          "no suitable node"

Updating Swarm services requires careful coordination. Let me show you how to diagnose and fix update failures.

Understanding Service Updates

docker service update modifies running services: - Changes image version - Updates resource limits - Adds/removes constraints - Modifies environment variables - Changes replicas count

Updates use rolling updates by default—tasks are replaced incrementally.

Step 1: Check Current Service State

Before updating, understand the current state:

```bash # List all services docker service ls

# Inspect specific service docker service inspect myapp --pretty

# Check task status docker service ps myapp

# Check service details docker service inspect myapp --format '{{json .Spec}}' ```

Step 2: Verify Image Availability

New images must be accessible:

```bash # Pull new image to verify it exists docker pull myapp:v2.0

# Or on all nodes (for local images) docker pull myapp:v2.0 # Or use a registry that all nodes can access ```

Image not found error:

```bash # Error: image not found # Solution: pull or build image first docker build -t myapp:v2.0 . docker push registry.example.com/myapp:v2.0

# Then update docker service update --image registry.example.com/myapp:v2.0 myapp ```

Step 3: Handle Constraint Failures

Updates may fail if constraints block placement:

```bash # Check current constraints docker service inspect myapp --format '{{json .Spec.TaskTemplate.Placement}}'

# Check node labels docker node ls --format "{{.Hostname}} {{.Status}}" docker node inspect node-1 --format '{{json .Spec.Labels}}' ```

Add missing labels:

```bash # Add label to node docker node update --label-add zone=west node-1

# Retry update docker service update --image myapp:v2.0 myapp ```

Or remove constraints:

# Remove constraint
docker service update --constraint-rm node.labels.zone myapp

Step 4: Fix Resource Limit Issues

Insufficient resources cause pending tasks:

```bash # Check current resource limits docker service inspect myapp --format '{{json .Spec.TaskTemplate.Resources}}'

# Check node resources docker node inspect node-1 --format '{{.Description.Resources.MemoryBytes}}' docker node inspect node-1 --format '{{.Description.Resources NanoCPUs}}'

# Reduce limits if needed docker service update --limit-memory 256M myapp ```

Resource exhaustion:

```bash # Node might be running too many tasks docker node ps node-1

# Drain some services from the node docker service update --constraint-add node.hostname!=node-1 other-service

# Then retry update docker service update --image myapp:v2.0 myapp ```

Step 5: Handle Rolling Update Issues

Rolling updates can get stuck:

```bash # Check update status docker service inspect myapp --format '{{.UpdateStatus}}'

# Check update config docker service inspect myapp --format '{{json .Spec.UpdateConfig}}' ```

Update stuck with failed tasks:

```bash # View failed tasks docker service ps myapp --no-trunc

# Roll back to previous version docker service rollback myapp

# Or force complete update docker service update --update-parallelism 1 --update-delay 10s --image myapp:v2.0 myapp ```

Adjust update parameters:

# Slower, more careful update
docker service update \
  --update-parallelism 1 \
  --update-delay 30s \
  --update-failure-action rollback \
  --image myapp:v2.0 \
  myapp

Step 6: Network Update Issues

Changing networks can fail:

```bash # Error adding network docker service update --network-add new-network myapp

# Network must exist docker network create -d overlay new-network

# Then add to service docker service update --network-add new-network myapp ```

Remove network:

docker service update --network-rm old-network myapp

Step 7: Environment Variable Updates

Updating environment variables:

```bash # Add environment variable docker service update --env-add VAR=value myapp

# Remove environment variable docker service update --env-rm VAR myapp

# Update existing variable docker service update --env-add VAR=newvalue myapp # Replaces existing ```

Step 8: Secret and Config Updates

Adding/removing secrets:

```bash # Add secret (must exist) docker secret create new_secret - docker service update --secret-add new_secret myapp

# Add secret with custom target docker service update \ --secret-add source=new_secret,target=/app/secret.txt \ myapp

# Remove secret docker service update --secret-rm old_secret myapp ```

Config updates work similarly:

docker config create new_config ./config.json
docker service update --config-add new_config myapp

Step 9: Handle Health Check Failures

Services with health checks may fail updates:

```bash # Check health check configuration docker service inspect myapp --format '{{json .Spec.TaskTemplate.ContainerSpec.Healthcheck}}'

# Tasks failing health check docker service ps myapp --no-trunc | grep unhealthy

# Temporarily disable health check for update docker service update --health-check-interval 60s --health-check-retries 10 myapp

# Or remove health check docker service update --health-check-none myapp ```

Step 10: Force Update When Stuck

If updates completely stuck:

```bash # Force immediate update docker service update --force myapp

# This recreates all tasks immediately # Bypasses rolling update ```

Or scale down then up:

```bash # Scale to zero docker service scale myapp=0

# Wait for tasks to stop docker service ps myapp

# Scale back up docker service scale myapp=3

# Or update while scaling docker service update --replicas 3 --image myapp:v2.0 myapp ```

Step 11: Debug Update Failures

When updates keep failing:

```bash # Check service events docker service inspect myapp --format '{{json .UpdateStatus}}'

# Check task errors docker service ps myapp --no-trunc

# Check node availability docker node ls

# Check daemon logs on nodes journalctl -u docker.service | grep -i service

# Run test container with same config docker run -d --name test \ --network mynet \ --env VAR=value \ myapp:v2.0 docker logs test ```

Complete Update Workflow

```bash # 1. Verify current state docker service inspect myapp

# 2. Pull new image docker pull myapp:v2.0

# 3. Test image locally docker run --rm myapp:v2.0 --help

# 4. Update with rollback safety docker service update \ --image myapp:v2.0 \ --update-failure-action rollback \ --update-parallelism 1 \ --update-delay 30s \ myapp

# 5. Monitor update progress docker service ps myapp watch -n 2 'docker service ps myapp | head -10'

# 6. Verify success docker service inspect myapp --format '{{.UpdateStatus.State}}' ```

Common Update Error Patterns

Update Options Reference

docker service update \
  --image IMAGE              # Change image
  --replicas N               # Scale replicas
  --env-add VAR=val          # Add env var
  --env-rm VAR               # Remove env var
  --secret-add SECRET        # Add secret
  --secret-rm SECRET         # Remove secret
  --config-add CONFIG        # Add config
  --config-rm CONFIG         # Remove config
  --network-add NET          # Add network
  --network-rm NET           # Remove network
  --constraint-add CONSTR    # Add constraint
  --constraint-rm CONSTR     # Remove constraint
  --limit-memory MEM         # Set memory limit
  --limit-cpu CPU            # Set CPU limit
  --update-parallelism N     # Concurrent updates
  --update-delay SECONDS     # Delay between updates
  --update-failure-action rollback/continue  # On failure
  --force                    # Force immediate update

Prevention Best Practices

1. 1.Test images locally before service update
2. 2.Use rollback on failure (--update-failure-action rollback)
3. 3.Update slowly (low parallelism, longer delay)
4. 4.Verify node capacity before increasing resources
5. 5.Ensure secrets/configs exist before referencing
6. 6.Monitor updates actively with docker service ps

Quick Reference

Service update failures are usually due to missing resources, constraint mismatches, or unavailable images. Verify prerequisites exist, use rolling updates with rollback safety, and monitor the update process.