Monitoring AWS Systems Behind ElasticBeanStalk

So I'm getting a company set up in the Amazon Cloud -- creating IAAS protocol/solutions/standardized implementation, etc while also being the SysAdmin for individual systems, app environments, and day-to-day uptime.

One of the biggest issues I'm having is tracking various system/application logs, as well as logging/monitoring/archiving system metrics like memory usage, cpu usage, etc etc In a centralized fashion. E.g. --> Nagios + Urchin.

The BIGGEST impediment to my endeavors is the following:

The company application is deployed in the form of a Java *.WAR file, uploaded to an Elastic BeanStalk application environment, load balancing and auto-scaling between 3(min) and 10(max) servers, and the EC2's that run the application are fired up and disposed of ad-hoc.

That is to say, I can't monitor the individual EC2's for very long because so many are being terminated then auto-provisioned/auto-scaled on the fly -- so I'd constantly be having to "monitor what I'm monitoring", and continuously remove/add EC2 machine addresses to my monitoring lists.

IS there some sort of way to use monitoring tools like Zabbix or Nagios to monitor the ElasticBeanStalk, and have it automatically add on new EC2's, and remove terminated/failed EC2's from its monitoring list automatically?

Furthermore, is there anything I can do with GrayLog to achieve similar results with the aggregation/centralization of my application logs from multiple EC2 instances into ONE consolidated set of logs/events? If not GrayLog, is there ANYTHING LIKE GrayLog that can automatically detect what EC2 members are being added/removed from the environment, and collect the logs from them automatically?

Any and all advice or direction is appreciated.

Thanks much, and cheers!!

Solution 1:

If you are deploying a WAR on elastic beanstalk you can install the metrics by creating a configuration file in the .ebextensions folder under WEB-INF. See the following link for more information on configuring and instance using this: - http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers.html

To install disk / memory metrics you need to install the "Amazon CloudWatch Monitoring Scripts for Linux" - see http://aws.amazon.com/code/8720044071969977

files:
  "/opt/aws/cwms/CloudWatchMonitoringScripts.zip":
    mode: "000777"
    owner: ec2-user
    group: ec2-user
    source:  http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip
container_commands:
  01_unzip_cloud_watch_zip: 
    command: unzip -d /opt/aws/cwms /opt/aws/cwms/CloudWatchMonitoringScripts.zip
    ignoreErrors: true
  02_update_password_file:
    command: sed -i 's/Key=$/Key=<VALUE OF YOUR SECRET KEY>/;s/KeyId=$/KeyId=<VALUE OF YOUR ACCESS ID>/' /opt/aws/cwms/awscreds.conf
  03_update_crontab:    
    command: echo "*/1 * * * * /opt/aws/cwms/mon-put-instance-data.pl --mem-util --disk-path=/ --disk-space-util --from-cron" | crontab - -u ec2-user

Basically what this script does is download the Linux based CloudWatchMonitoringScripts.zip into a folder such as /opt/aws/cwms (this can be anywhere). The commands then unzip the file, update the access / secret key (using the "sed" command) and finally creating the crontab tab.

Be careful of the crontab tab section, as it could potentially wipe you existing crontab entries.

UPDATE (FEB 2016)

Here's an updated script which is working for me quite nicely as of Feb 2016 (see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html).

sources: 
  /opt/cloudwatch: http://ec2-downloads.s3.amazonaws.com/cloudwatch-samples/CloudWatchMonitoringScripts-v1.1.0.zip

commands:
  00-installpackages:
    command: yum install -y perl-Switch perl-Sys-Syslog perl-LWP-Protocol-https

container_commands:
  01-setupcron:
    command: |
      echo '* * * * * root perl /opt/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl `{"Fn::GetOptionSetting" : { "OptionName" : "CloudWatchMetrics", "DefaultValue" : "--mem-used --memory-units=megabytes --mem-util --disk-space-util --disk-space-used --disk-space-avail --disk-path=/" }}` >> /var/log/cwpump.log 2>&1' > /etc/cron.d/cwpump
  02-changeperm:
    command: chmod 644 /etc/cron.d/cwpump
  03-changeperm:
    command: chmod u+x /opt/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl

option_settings:
  "aws:autoscaling:launchconfiguration" :
    IamInstanceProfile : "MonitorRole"
  "aws:elasticbeanstalk:customoption" :
    CloudWatchMetrics : "--mem-used --memory-units=megabytes --mem-util --disk-space-util --disk-space-used --disk-space-avail --disk-path=/"

NOTE: You must have an IAM role called MonitorRule in place. It's role policy should be as follows (also see http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/customize-containers-cw.html):-

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "cloudwatch:PutMetricData",
        "ec2:DescribeTags"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

Solution 2:

For Elastic beanstack's disk usage, you can enable the RootFileSysmtemUtil cloud watch instance metrics. This is available under Elastic beanstalk's Health configuration section. There is a "Health Reporting" section under "Health" configuration. Please select the RootFileSysmtemUtil option and save the configuration.

Now if you go to CloudWatch's ElasticBeanstack metrics, you can see the new metrics.