sudden "Could not evaluate: Field 'key' is required" errors when generating exported SSH keys
To help debugging/understanding what has happened, there's three things you can do (at least):
- Run the agent in debug mode: puppet agent -tv --debug . It's a bit of a firehose, but there might be some clues in there (it's a bit of a long shot - the problem is probably on the server
- Review the puppetmaster/server logs; I recommend tailing them while you run the agent on the problem node(s), so you can be sure you're seeing the output of those runs. It's entirely possible there's a clue in there (the server emits lots of warnings about things that are not-quite-right that are not visible on the agents)
- Go spelunking into the HSQLDB database; there should be some rows in a table somewhere related to the various ssh keys. See what info is there, particularly if some are working and some are not (compare the rows).
In the end it worked by deleting the PuppetDB data directory and re-running puppet on all nodes. I suspect some corrupt HSQLDB files.