C# httpwebrequest and javascript

Solution 1:

Just an idea but there is a way to have .net load a webpage as if it were in a browser: using System.Windows.Forms

you could Load the webpage into a WebBrowser control

WebBrowser wb = new WebBrowser();
wb.ScrollBarsEnabled = false;
wb.ScriptErrorsSuppressed = true;
wb.Navigate(url);
while (wb.ReadyState != WebBrowserReadyState.Complete) { Application.DoEvents(); }
wb.Document.DomDocument.ToString()

This will probably give you the pre ajax DOM but maybe there is a way to let it run the ajax first.

Solution 2:

If I correctly interpret your question, there is no simple solution for your problem.

You are scraping the HTML from a server and since your C# code is not a real web browser, it doesn't execute client scripts.

This way you can't access information which the HTML you fetch doesn't contain.

Edit: I don't know how complex these AJAX calls from the original web site are, but you could use Firebug or Fiddler for IE to see how the requests are made in order to call these AJAX calls in your C# application too. So you could add the pieces of information you'll need. But it's only a theoretical solution.

Solution 3:

When you open a web page in a web browser, it is the browser that executes the javascript and downloads additional resources used by the page (images, scripts, etc). HttpWebRequest by itself will not do any of this, it will only download the html for the page you requested. It will never execute any of the javascript/ajax code on it's own.

Solution 4:

Use HttpWebRequest to download the page, programatically search the source code for the relevant ajax information and then use a new HttpWebRequest to pull that data down.