Telegraf works manually but not the service - Run Telegraf in background
I have a virtual machine running on Ubuntu 20.04, I installed Telegraf, InfluxDB and Grafana for monitoring VMs and cluster's nodes.
Everythings work except when I try to make telegraf working in background.
The command to start manually Telegraf which is working :
telegraf --config http://XXX.XX.X.X:8086/api/v2/telegrafs/XXXXXXXXXXX
returns me :
2021-03-14T00:02:27Z I! Starting Telegraf 1.17.3
2021-03-14T00:02:27Z I! Loaded inputs: cpu disk diskio mem net processes swap system
2021-03-14T00:02:27Z I! Loaded aggregators:
2021-03-14T00:02:27Z I! Loaded processors:
2021-03-14T00:02:27Z I! Loaded outputs: influxdb_v2
2021-03-14T00:02:27Z I! Tags enabled: host=XXXXX
2021-03-14T00:02:27Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"XXXXX", Flush Interval:10s
So everythings good so far.
But impossible to run this service (/lib/systemd/system/telegraf.service):
[Unit]
Description=The plugin-driven server agent for reporting metrics into InfluxDB
Documentation=https://github.com/influxdata/telegraf
After=network.target
[Service]
EnvironmentFile=-/etc/default/telegraf
User=telegraf
ExecStart=/usr/bin/telegraf -config /etc/telegraf/telegraf.conf -config-directory /etc/telegraf/telegraf.d $TELEGRAF_OP>
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartForceExitStatus=SIGPIPE
KillMode=control-group
[Install]
WantedBy=multi-user.target
It's starting well but after checking the service status I'm getting :
14T00:10:46Z E! [agent] Error writing to outputs.influxdb: could not write any address
14T00:10:56Z W! [outputs.influxdb] Metric buffer overflow; 25 metrics have been dropped
14T00:10:56Z E! [outputs.influxdb] When writing to [http://XXX.XX.X.X:8086]: 401 Unauthorized
14T00:10:56Z E! [agent] Error writing to outputs.influxdb: could not write any address
14T00:11:06Z W! [outputs.influxdb] Metric buffer overflow; 25 metrics have been dropped
14T00:11:06Z E! [outputs.influxdb] When writing to [http://XXX.XX.X.X:8086]: 401 Unauthorized
14T00:11:06Z E! [agent] Error writing to outputs.influxdb: could not write any address
14T00:11:16Z W! [outputs.influxdb] Metric buffer overflow; 25 metrics have been dropped
14T00:11:16Z E! [outputs.influxdb] When writing to [http://XXX.XX.X.X:8086]: 401 Unauthorized
14T00:11:16Z E! [agent] Error writing to outputs.influxdb: could not write any address
And InfluxDB does not get any data and so Grafana.
Can anyone help me solve this service problem and get Telegraf working in background pls ?
This information was hard to find! I had exactly the same problem and I found the solution here: https://github.com/influxdata/telegraf/issues/8395.
So all you have to do is to add the INFLUX_TOKEN variable to /etc/default/telegraf such as:
INFLUX_TOKEN=mytoken