When is the right time to add a separate database server, add more web servers?

Usually web projects start small, everything is on the one server. But if site becomes popular one server won't be enough.
So, since I don't have any experience in this, I'd like to get an idea of:
1. what kind of load means that I need to move my MySQL DB to a separate server and how to measure this load?
2. what kind of load means that I need to add another web server and how to measure it?
3. approximately, how many users can one server serve?

I'm talking about simple PHP app on a LAMP server like this:
* CPU: Athlon 3800+
* CPU Details: 2 x 2.0 GHz
* RAM: 1 GB RAM
* Hard Disks: 2 x 160 GB (RAID 1 Software)

it all depends... on size of your dataset, usage pattern.

measure end-user experience. set acceptable page load time, test that every 5-10 minutes, plot charts. it all depends what your business can tolerate and when it becomes unacceptable.

enable slow_query in mysql, probably you do not want to have more then 5-10 queries per day that take more then 10 seconds. or maybe you have some nightly reporting and you can tolerate those?

probably you want to generate some statistics - for instance with munin with cpu usage, iostats, system load, network traffic, number of http processess, number of mysql threads, page generation time and see how that changes during the day / week. you can pinpoint peak time with that and can see how for instance adding more memory / tuning mysql changes your page load time.

in your case load average > 2 or 3 means system is overloaded [ but that's highly discussable ]

look here for some tips.

The time to start planning for more servers and a scalable architecture is when you sit down and draw up your specs (you want to have a code base with modular functionality, so it's easy to pull things apart, but at least having a "web front-end" and a "database back-end" is some level ov modularity).

Once you have that in place and you're running on one server (database and front-end), you can monitor how long different operations take and the load on the machine. Once load starts creeping up over "number of CPUs", you're definitely into "we should split this up"-land. If you have saved historical data, you can extrapolate how things will look in the future and decide on adding more capacity at a time, where the capacity will be in roughly as it's needed.

On a unix machine, the "load average" is (approximately) the average length of the run queue (that is, processes that are ready to run, but waiting for a time slot; usually NOT scaled by the number of CPUs physically in the box). If this is persistently higher than the number of CPUs you have, this is at least one indicator that the box is over-worked. It's not the only measure, but as a hard-and-fast guide, it's not too bad and teh system computes it for you.

Another thing to look at besides load, is your need for 100% uptime. If you need your site to remain up during server reboots, then you need some redundancy / clustering.

We have our webservers running as Virtual Machines, so we only use two (about 200 sites) for redundancy, and can increase the "server resources" if necessary to handle the load, without needing to introduce a third webserver (up to a point).

If you need to increase performance, the easiest thing is to separate out your MySQL on to a seperate box. However, that also means that your sites will be down twice as much (ie - they will be down when either server is rebooted/fails)