Solution 1:

IE8 includes a trident token in the User-Agent string regardless of compatibility mode.

See MSDN for more details: http://blogs.msdn.com/ie/archive/2009/01/09/the-internet-explorer-8-user-agent-string-updated-edition.aspx

IE7 on Windows Vista
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)

IE8 on Windows Vista (Compatibility View)
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0)

IE8 on Windows Vista
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)

Solution 2:

You need to check the value of document.documentMode which will have a value of 8 in true IE8 mode and 7 in IE7 mode. The user agent string will be identical in both cases and will still state IE8. Note that this JavaScript variable is only available in IE8.