Systemd permission denied on creating a folder

I am trying to install InfluxDB 2.0 and followed the instruction here in Centos 7. So, my service files is: /lib/systemd/system/influxdb.service and its content:

[Unit]
Description=InfluxDB 2.0 service file.
Documentation=https://v2.docs.influxdata.com/v2.0/get-started/
After=network-online.target

[Service]
User=influxdb
Group=influxdb
ExecStart=/usr/local/bin/influxd
Restart=on-failure

[Install]
WantedBy=multi-user.target

And lsing the file:

[vagrant@localhost ~]$ ls /lib/systemd/system/inf* -l
-rw-r--r--. 1 root root 268 Oct 17 09:54 /lib/systemd/system/influxdb.service

The Influxdb binaries are here:

[vagrant@localhost ~]$ ls /usr/local/bin/inf* -l
-rwxr-xr-x. 1 influxdb influxdb 33235536 Oct 17 09:46 /usr/local/bin/influx
-rwxr-xr-x. 1 influxdb influxdb 95804192 Oct 17 09:46 /usr/local/bin/influxd

Systemd service status is:

[vagrant@localhost ~]$ systemctl status influxdb.service
● influxdb.service - InfluxDB 2.0 service file.
   Loaded: loaded (/usr/lib/systemd/system/influxdb.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-10-17 11:39:54 UTC; 16min ago
     Docs: https://v2.docs.influxdata.com/v2.0/get-started/
 Main PID: 13398 (influxd)
   CGroup: /system.slice/influxdb.service
           └─13398 /usr/local/bin/influxd

However, the service is not working and looking the sudo journalctl | grep influxdb | less , I found this:

Oct 17 09:46:27 localhost.localdomain sudo[16616]:  vagrant : TTY=pts/0 ; PWD=/opt/influxdb-2.0.0-rc.1_linux_amd64 ; USER=root ; COMMAND=/bin/cp ./{influx, influxd} /usr/local/bin
Oct 17 09:46:37 localhost.localdomain sudo[16619]:  vagrant : TTY=pts/0 ; PWD=/opt/influxdb-2.0.0-rc.1_linux_amd64 ; USER=root ; COMMAND=/bin/cp {influx, influxd} /usr/local/bin
Oct 17 09:46:52 localhost.localdomain sudo[16634]:  vagrant : TTY=pts/0 ; PWD=/opt/influxdb-2.0.0-rc.1_linux_amd64 ; USER=root ; COMMAND=/bin/cp influx influxd /usr/local/bin
Oct 17 09:50:31 localhost.localdomain sudo[16653]:  vagrant : TTY=pts/0 ; PWD=/usr/lib/systemd/system ; USER=root ; COMMAND=/sbin/useradd -rs /bin/false influxdb
Oct 17 09:50:31 localhost.localdomain useradd[16655]: new group: name=influxdb, GID=993
Oct 17 09:50:31 localhost.localdomain useradd[16655]: new user: name=influxdb, UID=996, GID=993, home=/home/influxdb, shell=/bin/false
Oct 17 09:50:42 localhost.localdomain sudo[16661]:  vagrant : TTY=pts/0 ; PWD=/usr/lib/systemd/system ; USER=root ; COMMAND=/bin/touch /lib/systemd/system/influxdb.service
Oct 17 09:51:14 localhost.localdomain sudo[16677]:  vagrant : TTY=pts/0 ; PWD=/usr/lib/systemd/system ; USER=root ; COMMAND=/bin/vim influxdb.service
Oct 17 09:58:42 localhost.localdomain sudo[16708]:  vagrant : TTY=pts/0 ; PWD=/usr/lib/systemd/system ; USER=root ; COMMAND=/bin/systemctl start influxdb.service
Oct 17 09:58:56 localhost.localdomain influxd[16716]: ts=2020-10-17T09:58:56.784692Z lvl=error msg="Failed opening bolt" log_id=0PuVoU40000 error="unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied"
Oct 17 09:58:56 localhost.localdomain influxd[16716]: Error: unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied
Oct 17 09:58:56 localhost.localdomain influxd[16716]: --bolt-path string                                              path to boltdb database (default "/home/influxdb/.influxdbv2/influxd.bolt")
Oct 17 09:58:56 localhost.localdomain influxd[16716]: --engine-path string                                            path to persistent engine files (default "/home/influxdb/.influxdbv2/engine")
Oct 17 09:58:56 localhost.localdomain systemd[1]: influxdb.service: main process exited, code=exited, status=1/FAILURE
Oct 17 09:58:56 localhost.localdomain systemd[1]: Unit influxdb.service entered failed state.
Oct 17 09:58:56 localhost.localdomain systemd[1]: influxdb.service failed.
Oct 17 09:58:57 localhost.localdomain systemd[1]: influxdb.service holdoff time over, scheduling restart.
Oct 17 09:59:10 localhost.localdomain influxd[16731]: ts=2020-10-17T09:59:10.994806Z lvl=error msg="Failed opening bolt" log_id=0PuVpL_W000 error="unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied"
Oct 17 09:59:10 localhost.localdomain influxd[16731]: Error: unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied
Oct 17 09:59:11 localhost.localdomain influxd[16731]: --bolt-path string                                              path to boltdb database (default "/home/influxdb/.influxdbv2/influxd.bolt")
Oct 17 09:59:11 localhost.localdomain influxd[16731]: --engine-path string                                            path to persistent engine files (default "/home/influxdb/.influxdbv2/engine")
Oct 17 09:59:11 localhost.localdomain systemd[1]: influxdb.service: main process exited, code=exited, status=1/FAILURE
Oct 17 09:59:11 localhost.localdomain systemd[1]: Unit influxdb.service entered failed state.
Oct 17 09:59:11 localhost.localdomain systemd[1]: influxdb.service failed.
Oct 17 09:59:11 localhost.localdomain systemd[1]: influxdb.service holdoff time over, scheduling restart.
Oct 17 09:59:25 localhost.localdomain influxd[16745]: ts=2020-10-17T09:59:25.253903Z lvl=error msg="Failed opening bolt" log_id=0PuVqDHG000 error="unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied"
Oct 17 09:59:25 localhost.localdomain influxd[16745]: Error: unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied
Oct 17 09:59:25 localhost.localdomain influxd[16745]: --bolt-path string                                              path to boltdb database (default "/home/influxdb/.influxdbv2/influxd.bolt")
Oct 17 09:59:25 localhost.localdomain influxd[16745]: --engine-path string                                            path to persistent engine files (default "/home/influxdb/.influxdbv2/engine")
Oct 17 09:59:25 localhost.localdomain systemd[1]: influxdb.service: main process exited, code=exited, status=1/FAILURE
Oct 17 09:59:25 localhost.localdomain systemd[1]: Unit influxdb.service entered failed state.
Oct 17 09:59:25 localhost.localdomain systemd[1]: influxdb.service failed.
Oct 17 09:59:25 localhost.localdomain systemd[1]: influxdb.service holdoff time over, scheduling restart.
Oct 17 09:59:39 localhost.localdomain influxd[16759]: ts=2020-10-17T09:59:39.280461Z lvl=error msg="Failed opening bolt" log_id=0PuVr440000 error="unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied"
Oct 17 09:59:39 localhost.localdomain influxd[16759]: Error: unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied
Oct 17 09:59:39 localhost.localdomain influxd[16759]: --bolt-path string                                              path to boltdb database (default "/home/influxdb/.influxdbv2/influxd.bolt")
Oct 17 09:59:39 localhost.localdomain influxd[16759]: --engine-path string                                            path to persistent engine files (default "/home/influxdb/.influxdbv2/engine")
Oct 17 09:59:39 localhost.localdomain systemd[1]: influxdb.service: main process exited, code=exited, status=1/FAILURE
Oct 17 09:59:39 localhost.localdomain systemd[1]: Unit influxdb.service entered failed state.
Oct 17 09:59:39 localhost.localdomain systemd[1]: influxdb.service failed.
Oct 17 09:59:39 localhost.localdomain systemd[1]: influxdb.service holdoff time over, scheduling restart.
Oct 17 09:59:53 localhost.localdomain influxd[16778]: ts=2020-10-17T09:59:53.202433Z lvl=error msg="Failed opening bolt" log_id=0PuVrvSW000 error="unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied"
Oct 17 09:59:53 localhost.localdomain influxd[16778]: Error: unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied
Oct 17 09:59:53 localhost.localdomain influxd[16778]: --bolt-path string                                              path to boltdb database (default "/home/influxdb/.influxdbv2/influxd.bolt")
Oct 17 09:59:53 localhost.localdomain influxd[16778]: --engine-path string                                            path to persistent engine files (default "/home/influxdb/.influxdbv2/engine")
Oct 17 09:59:53 localhost.localdomain systemd[1]: influxdb.service: main process exited, code=exited, status=1/FAILURE
Oct 17 09:59:53 localhost.localdomain systemd[1]: Unit influxdb.service entered failed state.
Oct 17 09:59:53 localhost.localdomain systemd[1]: influxdb.service failed.
Oct 17 09:59:53 localhost.localdomain systemd[1]: influxdb.service holdoff time over, scheduling restart.
Oct 17 10:00:07 localhost.localdomain influxd[16795]: ts=2020-10-17T10:00:07.316899Z lvl=error msg="Failed opening bolt" log_id=0PuVsma0000 error="unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied"
Oct 17 10:00:07 localhost.localdomain influxd[16795]: Error: unable to create directory /home/influxdb/.influxdbv2/influxd.bolt: mkdir /home/influxdb: permission denied

It is failing on creation of the folders as per the log above. I ran this: sudo mkhomedir_helper influxdb to create the home folder for influxdb and the result of checking permission is this:

[vagrant@localhost ~]$ sudo ls -l /home
total 0
drwxr-xr-x. 4 influxdb influxdb  95 Oct 17 11:39 influxdb
drwx------. 4 vagrant  vagrant  109 Oct 17 11:54 vagrant

I cannot figure out where I am missing to give enough permission for the user (created: sudo useradd -rs /bin/false influxdb) to run the binary as a service?

UPDATE: Audit log added

type=SERVICE_START msg=audit(1602934669.699:3137): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934683.369:3138): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_START msg=audit(1602934683.556:3139): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934683.556:3140): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1602934683.558:3141): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934697.637:3142): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_START msg=audit(1602934697.879:3143): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934697.879:3144): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1602934697.882:3145): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934711.388:3146): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_START msg=audit(1602934711.561:3147): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934711.561:3148): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1602934711.565:3149): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934725.261:3158): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_START msg=audit(1602934725.700:3159): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934725.700:3160): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1602934725.700:3161): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934739.450:3162): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_START msg=audit(1602934739.558:3163): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934739.559:3164): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1602934739.561:3165): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934753.257:3166): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
type=SERVICE_START msg=audit(1602934753.585:3167): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934753.585:3168): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1602934753.588:3169): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_STOP msg=audit(1602934783.087:3186): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=SERVICE_START msg=audit(1602934794.151:3193): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=influxdb comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
type=USER_AUTH msg=audit(1602937018.434:3233): pid=13575 uid=1000 auid=1000 ses=7 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="influxdb" exe="/usr/bin/su" hostname=localhost.localdomain addr=? terminal=pts/0 res=failed'
type=USER_CHAUTHTOK msg=audit(1602937096.193:3238): pid=13581 uid=0 auid=1000 ses=7 subj=unconfined_u:unconfined_r:passwd_t:s0-s0:c0.c1023 msg='op=PAM:chauthtok grantors=? acct="influxdb" exe="/usr/bin/passwd" hostname=localhost.localdomain addr=? terminal=pts/0 res=failed'

Help? Thanks

Solution 1:

I wasn't sure about this question until yesterday when I ran into a similar issue and traced it to SELinux.

InfluxDB does not have an SELinux policy, neither shipped by Red Hat nor included with InfluxDB itself. Such services run under a generic policy which, among other things, prohibits the service to access /home as system services are expected to place their files elsewhere.

Try to move the InfluxDB home directory from /home/influxdb to /var/lib/influxdb. You can do it like this:

1. Run sudo mv /home/influxdb /var/lib/influxdb to move the files.
2. Run sudo restorecon -rv /var/lib/influxdb to fix the SELinux contexts.
3. Run sudo vipw to edit the passwd file, and change the influx user's home directory from /home/influxdb to /var/lib/influxdb.

Now try to start the service again. If it doesn't work, you may also need to re-run influx setup to get rid of any old configuration that references /home.

Solution 2:

Same issue. It seems /home/influxdb folder is needed with the right permissions. Following worked for me

sudo mkdir /home/influxdb
sudo chown -R influxdb:influxdb /home/influxdb

Alternatively modifying the useradd command to following might also work (I haven't tried it) and this would be a cleaner fix:

sudo useradd -d /home/influxdb -rs /bin/false influxdb