Stackdriver Extractor fails
I'm trying to get stackdriver working with GCE. I went through the standard instructions to install (executing stack-install.sh --write-gcm), however it doesn't seem to work at all. When I add monitoring alerts to watch processes, I get:
Process monitoring not available for this instance
In /var/log/messages I see:
Mar 28 19:47:59 myhost stackdriver-agent: which: no java in (/root/.gvm/bin:/usr/local/sbin:/sbin
:/usr/sbin:/bin:/usr/bin:/go/bin:/bin)
Mar 28 19:47:59 myhost stackdriver-agent: Starting stackdriver-collectd: option = PIDFile; value
= /var/run/stackdriver-agent.pid;
Mar 28 19:47:59 myhost stackdriver-agent: option = Interval; value = 60.000000;
Mar 28 19:47:59 myhost stackdriver-agent: Created new plugin context.
Mar 28 19:47:59 myhost stackdriver-agent: [ OK ]
Mar 28 19:47:59 myhost collectd[1934]: Initialization complete, entering read-loop.
Mar 28 19:47:59 myhost systemd: Started LSB: start and stop Stackdriver Agent.
Mar 28 19:47:59 myhost collectd[1934]: match_throttle_metadata_keys: 1 history entries, 1 distinc
t keys, 71 bytes server memory.
Mar 28 19:47:59 myhost collectd[1934]: tcpconns plugin: Reading from netlink succeeded. Will use
the netlink method from now on.
Mar 28 19:47:59 myhost collectd[1934]: write_gcm: Asking metadata server for auth token
I have verified that the service is enabled. When I check the permissions, it says I don't need to need special credentials since I am using GCE. However, when I restart the stackdriver-extractor service, I get:
Please set STACKDRIVER_API_KEY in /etc/sysconfig/stackdriver
And so if I set that... I get:
stackdriver-extractor[3020]: Starting stackdriver-extractor: Unable to determine extractor endpoint!
stackdriver-extractor[3020]: Unable to generate config file
I feel like something is wrong at the project level where this service is not getting the key information automatically like it is supposed to.
I dug into the scripts and found where it gets it...
curl -f -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/project/attributes/stackdriver-agent-key
But this is getting a 404.
Also, when I check the scopes:
# curl --silent -f -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/scopes
https://www.googleapis.com/auth/cloud.useraccounts.readonly
https://www.googleapis.com/auth/devstorage.read_only
https://www.googleapis.com/auth/logging.write
https://www.googleapis.com/auth/monitoring.write
Any ideas? My ultimate problem is that I can't create alerts on the process counts, but I think this is where Stackdriver gets it from.
To fix this problem, I fixed the API key. I accidentally left out the double quotes when I edited the file.
That brings me to the next point. Why did I edit the file? They provide a nice easy utility for adding an API key. We should use it.
/opt/stackdriver/stack-config --api-key DEADBEEFDEADBEEFDEADBEEFDEADBEEF...
It will even restart the services for you.
As for how I got to this place to begin with... I accidentally copy/pasted a partial API key. It wasn't obvious because I missed a little bit at the end. Little bits matter.
As for why I could not get this working by changing the scope to all API access, I don't know. I thought that would work as I really don't want to have to mess around with any more statefulness in my VM disk than I have to. But I'll take a working solution at this point.
One final bit of advice... if you are here, you might need to also find your API key. You can find it here:
https://app.google.stackdriver.com/settings/accounts/agent/
If one isn't there, you can create it.
Good luck!
Just FYI the use of API keys is kind of depreciated/Legacy. You should enable the Google Monitoring API and just install the agent.