docker swarm database connection reset by peer

Solution 1:

I've got the same error deploying Docker Swarm stack of Spring Boot app and PostgreSQL. After battling with this for about a week, I've figured out that issue was in firewall dropping connections between containers because of inactivity. Quick answer, run following cmd on linux machine:

sudo sysctl -w \
net.ipv4.tcp_keepalive_time=600 \
net.ipv4.tcp_keepalive_intvl=60 \
net.ipv4.tcp_keepalive_probes=3

As well, I've included following tomcat connection pool properties:

tomcat:
  max-active: 10
  initial-size: 5
  max-idle: 8
  min-idle: 5
  test-on-borrow: true
  test-while-idle: true
  test-on-return: false
  test-on-connect: true
  validation-query: SELECT 1
  validation-interval: 30000
  max-wait: 30000
  min-evictable-idle-time-millis: 60000
  time-between-eviction-runs-millis: 5000
  remove-abandoned: true
  remove-abandoned-timeout: 60

Solution came from this blogpost: DEALING WITH NODENOTAVAILABLE EXCEPTIONS IN ELASTICSEARCH

Solution 2:

There is another way to prevent closing idle connection. The problem is related to default swarm service discovery which closes the idle connection after 15 minutes.
Explicit specified the dnsrr endpoint mode resolves the problem, e.g.:

version: '3.3'

services:
  foo-service:
    image: example/foo-service:latest
    hostname: foo-service
    networks:
      - foo_network
    deploy:
      endpoint_mode: dnsrr
      # ...

networks:
  foo_network:
    external: true
    driver: overlay