Markdown to LaTeX conversion with a custom preamble using Pandoc

As you have discovered, --include-in-header adds text into the preamble specified in Pandoc's LaTeX template. There are a few ways to do what you are trying to do.

1. If you would like a completely custom preamble, you need to specify a template file using

   pandoc -o output.tex --template=FILE input.txt
   

   The template can have variables (such as $title$ and, more importantly, $body$) and conditionals. If you would like some inspiration, you could check out the default template using the command

   pandoc -D latex
   

2. If you want to use a new template once and for all, you can make one, call it default.latex, and put it in the templates directory (~/.pandoc/templates/ on a unix machine). In this case, you need to specify that you want to use a template by calling

   pandoc -o output.tex --standalone input.txt
   

3. If you would rather not deal with templates at all, you can just run

   pandoc -o output.tex input.txt
   

   and the result will be a bare LaTeX document, that is, without a preamble, \begin{document} or \end{document}. Then you can add a preamble yourself. Note that any metadata (title, author) will be lost when using this method.

Full details on how to make and use templates can be found in Pandoc's excellent man page.

See also this answer on tex.stackexchange Adding headers and footers using Pandoc

With pandoc 1.12.x and it’s new YAML metadata capabilities you could add all the information and all LaTeX-code you need in your markdown document like this:

---
title: Test
author: Author Name
header-includes: |
    \usepackage{fancyhdr}
    \pagestyle{fancy}
    \fancyhead[CO,CE]{This is fancy}
    \fancyfoot[CO,CE]{So is this}
    \fancyfoot[LE,RO]{\thepage}
abstract: This is a pandoc test . . . 
---

# This is a test

Lorem ipsum....

That way you don't have to modify the template, simply calling pandoc doc.md -o doc.pdf will suffice.