Foreman displays a server as "out of sync", what could be the reason and what is the right way to troubleshoot such an issue?

If you're on Foreman 1.8 or lower, your host will be shown as 'out of sync' only when Foreman has not received any reports for that host for 5 minutes longer than the setting 'puppet_interval' (check it in Administer > Settings). By default this would be 35 minutes. There's an 'outofsync_interval' setting that allows you to customize this on Foreman 1.9.

In order to start troubleshooting this (missing reports), I would look at /var/log/foreman/production.log on your Foreman boxes and check if you receive the report from that host when Puppet finishes its run.

I found the reason for the out-of-sync notifications in Foreman. It seems like in these servers there was a stuck process of puppet agent -t, is looks like so:

root  7531 21487  0 Aug09 ?   00:00:03 puppet agent: applying configurat

Once I killed this process and ran puppet agent -t again - problem solved.

Hosts may be out of sync due to host/server restart or network issues or puppet version mismatch between the server and agent.

For the version mismatch, you can upgrade puppet agent to match the server version and sync host via "puppet agent -t" .

If you have remote-execution and remote-execution-ssh plugin for foreman enabled, you can run the "puppet agent -t" command from Foreman console for out of sync host. Below are the steps :

1. Enabling remote-execution plugins in foreman server , may require server restart

   sudo foreman-installer --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh
   

2. Syncing host

   • Goto “Hosts → All Hosts” and select the check box next to the out of sync host
   • A “Select Action” button will appear on top right of the hosts table. Select the “Schedule Remote Job” option from the action list.
   • In the displayed form, keep the default values and copy “sudo puppet agent -t” (without quotes) in “command” text box. You may need to provide full path to puppet (eg. /opt/puppetlabs/puppet/bin/puppet) if puppet binary is not in host's PATH. Click submit.
   • Once the command executes remotely in the host, it will sync with the server and Foreman should update the host status to sync after few minutes.