What determines whether a browser will open or download a .txt file?

Solution 1:

The behavior of the browser when you send it a file is going to be based on the content-type header sent by your server. If you aren't explicitly setting it yourself, your web server may attempt to make a best guess based on the first few bytes of the file.

See http://httpd.apache.org/docs/2.2/mod/mod_mime_magic.html