How to determine bottlenecks on Ubuntu LAMP environment

I would recommend to first install a logging framework such as munin, so that you have a some data available on your base load and during the spikes on your server. Then, I would monitor the queries on your SQL server (especially the slow queries).

Using this information, you can already determine if how extraordinary these situations are or whether they simply add the last nail to a highly loaded machine and what happens, such as you are running out of RAM, certain processes are hammering the disk at times, or your CPU is the issue.

This is a very broad question. To measure anything, first you need a baseline performance indicator. Install munin and munin-node and evaluate your baseline. collectd and the venerable SNMP are also options but not very user-friendly.

Think about your application. Is it database-heavy? Lots of IO? CPU (video encoding)? Perform some normal web application tasks and see where your bottlenecks even exist, first. Use other tools like siege, ab, or JMeter to automate these tasks.

After you have established some baseline metrics and breaking points, you can see where to improve. As stated, this is a broad question and therefore has a rather wide-spectrum answer:

• I/O bound: Are you hitting the database needlessly? What's your query cache look like? Do you need to optimize for writes or reads? Do you need a separate DB server?

• Memory bound: Assuming you're not database-I/O bound and have got as much as you can in memory, is your dataset too large for available memory? What is the cache hit rate? Are you using a separate cache framework like memcached?

• CPU bound: Is your application able to take advantage of multiple cores/workers? Do you suffer from thread lock or contention issues? Consider using a lightweight webserver like nginx instead of Apache, or a reverse proxy setup. There may also be relevant sysctl.conf (kernel) parameters you can change.

• Network bound: highly unlikely at this stage of the game.

Other points and considerations:

• Some quick optimization on your DB can be done with mysqltuner.pl.

• Try and simulate real traffic as closely as possible (replaying access logs is a good start).

• If you ever hit the swap area on disk in a production DB setting, you are in for a world of hurt.

tl;dr Graph everything you think is relevant, hit your server hard, tune, repeat.

What I have found best is to perform a capacity analysis before deployment. However there is a really great article on exactly what you are looking for, I hope this helps.

http://blog.inarow.net/post/227533559/four-steps-to-diagnose-your-lamp-application

1) Track MySQL Queries taking over a second 2) Monitor PHP Memory Usage & Log Apache Delivery Times 3) Log PHP Errors 4) Take Snapshots at an OS Level

I'd also like to add JMeter into this. http://jakarta.apache.org/jmeter/