Prometheus Blackbox Exporter TCP probe returns value of 0, even when port is open (Iperf3 server)

I have a strange problem that I've been digging into the last couple of days.

The setup

I have an Iperf3 server running in Docker on this internal address and port: 10.10.2.2:5201

I also have Prometheus and Grafana running in Docker, to monitor different services.

Everything is working, including Prometheus Endpoint exporter and Prometheus Blackbox exporter for HTTP and HTTPS.

However, I also want to monitor this Iperf3 server using a simple TCP probe with Blackbox exporter.

Here are my config files for both Prometheus in general and the Blackbox exporter.

Prometheus.yml: (Blackbox job under scrape configs)

  - job_name: 'blackbox-tcp'
    scrape_interval: 60s
    metrics_path: /probe
    params:
      module: [tcp_connect]  # Look for TCP response
    static_configs:
      - targets:
        - http://10.10.2.2:5201       # Iperf3 server
        - http://10.10.2.4:9201       # Portainer agent
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.10.2.4:9095  # The blackbox exporter's real hostname:port.

Blackbox.yml: (Defined modules)

modules:
  http_2xx:
    prober: http
    http:
      method: GET
  http_post_2xx:
    prober: http
    http:
      method: POST
  tcp_connect:
    prober: tcp
    timeout: 5s
  pop3s_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^+OK"
      tls: true
      tls_config:
        insecure_skip_verify: false
  ssh_banner:
    prober: tcp
    tcp:
      query_response:
      - expect: "^SSH-2.0-"
  icmp:
    prober: icmp

Just to show that Prometheus is up and running, here is a section of both the HTTP and TCP probes having connection to the exporter:

Finally, I'm able to probe the TCP via Netcat and Telnet, to confirm that the service on 10.10.2.2:5201 is actually running:

$ if nc -w 3 -z 10.10.2.2 5201; then echo "1"; fi
1

$ if echo -e '\x1dclose\x0d' | telnet 10.10.2.2 5201 > /dev/null; then echo "1"; fi
1

The problem

And here is the trouble: When I try to monitor the target 10.10.2.2:5201 in Grafana, it shows a query value of 0, aka Down.

Also, when I try to run the Query Inspector, or run the query directly in Prometheus, I get a confirmation that it returns a value of 0.

I believe I've done everything as documented, and this thread also confirms that my config should be correct for the TCP probes.

I'm simply lost about how I get Prometheus to return a query value of 1, indicating that the port is open and the Iperf3 service in this case is running?

I've wondered if there are some send and expect commands that I need to add in Blackbox.yml under query_response, but I haven't had any luck. Any input and help is greatly appreciated!

Solution 1:

I was having exactly the same problem until I realised I had to remove the http:// part from the target. The below should work for you.

 - job_name: 'blackbox-tcp'
    scrape_interval: 60s
    metrics_path: /probe
    params:
      module: [tcp_connect]  # Look for TCP response
    static_configs:
      - targets:
        - 10.10.2.2:5201       # Iperf3 server
        - 10.10.2.4:9201       # Portainer agent
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 10.10.2.4:9095  # The blackbox exporter's real hostname:port.