Monitor and graph CPU usage per process and per thread

Ouch.. that's quite a task.

Software like Munin, cacti or zabbix would probably be most suited and would easily cover the 1st and 3rd points. I suspect Munin is possibly the easiest of the three for those points. The 2nd and 4th are slightly trickier. I'm not sure of any monitoring system that might provide you with total CPU time for a period. Usually system metric collection is focussed around the idea of "Whats going wrong" rather than system accounting. Hopefully someone else might have a suggestion.

As a starting point for point 4, you need to enable the process accounting features, something that's been integral to Linux for a while. There is some good detail here: http://shibuvarkala.blogspot.com/2009/04/howto-enable-process-accounting-in.html that will get you started with enabling process accounting and reading information it returns.

One reason for suggesting Munin is that it's relatively trivial to write plugins for, and I suspect you'll need to for what you're trying to achieve.

Wish I could be more help.