C# Spider with HttpWebRequest
I was recently tasked to write code to facilitate html post requests to a website that hits a cgi application which in turn sets a configuration file that sends text streams to be displayed on TVs at the Kennedy Space Center. This component was to be built in support of the Weather Warning Appplication.
Weather warnings are a big deal at KSC due to the vastness of the center and many people work outdoors for a good part of the day. Long story short the code below details a simple web request call. I actually make an initial call to a login page and retain the cookie to maintain state on all subsequent requests. After this is done I continue hitting the required config. html form pages passing in the required form field values via the strPost variable. I hope this code helps anyone struggling with this.
FYI: All params are brought in from web.config.
(Notice the ConfigurationSettings.AppSettings["Key"]; calls)
The values in relation to each key look something like this:
loginId=foo&password=bar
{
//JFB - Calculate minutes for warning display.
DateTime currentDateTime = DateTime.Now;
//JFB - Difference in days, hours, and minutes.
TimeSpan tvTimeSpan = tvExpiration - currentDateTime;
//JFB - Difference in minutes.
int differenceInMinutes = (int) tvTimeSpan.TotalMinutes;
String url = ConfigurationSettings.AppSettings["URL"];
String strPost = ConfigurationSettings.AppSettings["LoginPassword"];
StreamWriter myWriter = null;
CookieContainer myContainer = new CookieContainer();
//Request #1 (the login)
HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(url);
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
objRequest.ContentType = "application/x-www-form-urlencoded";
objRequest.CookieContainer = new CookieContainer();
try
{
myWriter = new StreamWriter(objRequest.GetRequestStream());
myWriter.Write(strPost);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
myWriter.Close();
}
HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
//retain the cookies
foreach (Cookie cook in objResponse.Cookies)
{
myContainer.Add(cook);
}
//Check out the html.
using (StreamReader sr =
new StreamReader(objResponse.GetResponseStream()) )
{
String test = sr.ReadToEnd();
// Close and clean up the StreamReader
sr.Close();
}
//Request #2 (select the proper submenu)
objRequest = (HttpWebRequest)WebRequest.Create(url);
strPost = strPost = ConfigurationSettings.AppSettings["EditMenu"];
objRequest = (HttpWebRequest)WebRequest.Create(url);
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
objRequest.ContentType = "application/x-www-form-urlencoded";
objRequest.CookieContainer = myContainer;
try
{
myWriter = new StreamWriter(objRequest.GetRequestStream());
myWriter.Write(strPost);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
myWriter.Close();
}
objResponse = (HttpWebResponse)objRequest.GetResponse();
//Check out the html.
using (StreamReader sr =
new StreamReader(objResponse.GetResponseStream()) )
{
String test = sr.ReadToEnd();
// Close and clean up the StreamReader
sr.Close();
}
}
this is shailesh, i found your site when i search webreqest method h iin goole
help in goolge i try to use your C# webrequest spider for address book graber in y
yahoo website.
Please help me for same
i try to develop contact list grabber same as http://www.webdataextractor.net this site right now i have complited ,yahoo, msn contact list grabber .right now i am developing gmail contact list grabber.
do you have any idea about streamread and networkstream.
i have use in MSN contact list grabber but it's take 3 min to grabs contact
Thanks
Shailesh
Thank you for a great entry. I have been able to take your code and use it right away in one of my own applications.
Good Job!
If so, is there any way to get an example c# solution that has everything that this is missing, such as the missing "using" statements and such? I know I'm asking alot :(
If not, Thanks for the code :)
I'm not sure I understand what you are asking for with regard to missing using statements; but yes, this code will submit information in a html form and pull back the resulting page.
Thanks for the incredibly quick response, and for confirming that fact :)
Also, very nice blog here :)
Actually in my example there are only 2 settings which you may not even need. You'll probably have to create your own. It was intranet so probably would be useless. I added the two example entries below to give you an idea.
<add key="URL" value="https://127.0.0.1/acm/acm.cgi"/>
<add key="LoginPassword" value="loginuser=theUser&loginpassword=thePassword"/>
Thanks again for your time, patience, and assistance :)
You don't need to close the StreamReader since your code is inside a 'using' statement. 'using' will invoke sr.Dispose().
I hear you. But is doesn't hurt ;). Thanks for the kind words.
Thanks.
---------------------------
using System;
using System.Collections.Generic;
using System.Text;
using System.IO;
using System.Net;
namespace httpWebReqAndResp
{
class Program
{
static void Main(string[] args)
{
string strLoginName = "xxxxxxxxxx";
string strPassword = "xxxxxxxxxx";
try
{
string strURL = "http://login.yahoo.com/config/login?";
string strPostData = String.Format("login={0}&passwd={1}",
strLoginName.Trim(), strPassword.Trim());
// Setup the http request.
HttpWebRequest webReq = WebRequest.Create(strURL) as HttpWebRequest;
webReq.Method = "POST";
webReq.ContentLength = strPostData.Length;
webReq.ContentType = "application/x-www-form-urlencoded";
webReq.AllowAutoRedirect = true;
webReq.CookieContainer = new CookieContainer();
// Post to the login form.
StreamWriter streamRequestWriter = new StreamWriter(webReq.GetRequestStream());
streamRequestWriter.Write(strPostData);
streamRequestWriter.Close();
// Get the response.
HttpWebResponse webResp = (HttpWebResponse)webReq.GetResponse();
// Have some cookies.
CookieCollection cookieCollection = webResp.Cookies;
// Read the response
Stream datastream = webResp.GetResponseStream();
StreamReader reader = new StreamReader(datastream);
String strResponseFromServer = reader.ReadToEnd();
Console.WriteLine(strResponseFromServer);
Console.ReadLine();
}
catch (WebException e)
{
Console.WriteLine("\nMain 1 Exception raised!");
Console.WriteLine("\nMessage 1:{0}", e.Message);
Console.WriteLine("\nStatus 1:{0}", e.Status);
Console.WriteLine("Press any key to continue..........");
Console.ReadLine();
}
catch (Exception e)
{
Console.WriteLine("\nMain 2 Exception raised!");
Console.WriteLine("Source 2:{0} ", e.Source);
Console.WriteLine("Message 2:{0} ", e.Message);
Console.WriteLine("Press key to continue..........");
Console.ReadLine();
}
}
}
}
Thanks for the code, saves me from having to think and write. I know, i'm lazy! But that's why Google is there for, isn't it?
Good luck with your projects!
I have a URL like https://username@test.com:password@test.com/filename.ext
i have to get username. password and filename from it and need to authenicate it with the website user/pass. Once authnticated i need to provide them with the file mentioned in the URl .I am unable to parse the URL.
Please help me out.
char[] delimiterChars = { '@' };
string text = "https://username@test.com:password@test.com/filename.ext";
System.Console.WriteLine("Original text: '{0}'", text);
string[] urlParts = text.Split(delimiterChars);
System.Console.WriteLine("{0} parts in text:", urlParts.Length);
foreach (string s in urlParts)
{
System.Console.WriteLine(s);
}
string url = "https://username:password@server/filename.xml.zip";
WebRequest request = WebRequest.Create(strURI);
response = request.GetResponse();
I am getting error on GetResponse. like
InnerException = {"No connection could be made because the target machine actively refused it"}
I am sure my username. password are good.
its also helped to post the more than one request with maintaining the same session state thank u
The response from the login page of a site is <script>loaction.replace("homepage.aspx")</script>
When i read the response and write it in my page, the url searches for the home.aspx page on my server and gives resource not found error.
Any idea how to handle this?
I recommend parsing out the aspx file and if no dir structure is present prefix it with the url. All you are getting is relocated via javascript after a successful login I'm guessing?
Hope this helps.
location.replace("homepage.aspx"). It also returned few cookies.
Like you said, I tried appending the URL with in the javascript, but
the remote web server treated this as an entirely new request, redirecting back to the
login page.
I did send the cookies through the request.
Thank you
Satish
Boolean Certificate_isValid(object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) {
X509Store store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
store.Open(OpenFlags.OpenExistingOnly | OpenFlags.ReadOnly | OpenFlags.IncludeArchived | OpenFlags.MaxAllowed);
return store.Certificates.Contains(certificate);
}
...
ServicePointManager.ServerCertificateValidationCallback = Certificate_isValid;
-AH
I have a question concerning what to do AFTER you login to a page.
For example, I want to login to www.foxsheets.com (a sports info site) and once logged in navigate to a Yankees matchup page (containing sports stats about the game) and get the content of the page. I would then have a local copy of the Yankees game data to read off-line.
I have been able to login, but once I am logged in if I try to navigate to another url in the domain it loses the fact that I am logged in and the website reports back "you are not logged in". Here is what I thought would work:
public static string LoginToFormUsingPostAndGetContentOfSomeUrl(string loginUrl, string strPost, string someUrl, string strGet)
{
StreamWriter myWriter = null;
CookieContainer myContainer = new CookieContainer();
HttpWebRequest objRequest = (HttpWebRequest)WebRequest.Create(loginUrl);
objRequest.Method = "POST";
objRequest.ContentLength = strPost.Length;
objRequest.ContentType = "application/x-www-form-urlencoded";
objRequest.CookieContainer = new CookieContainer();
try
{
myWriter = new StreamWriter(objRequest.GetRequestStream());
myWriter.Write(strPost);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
myWriter.Close();
}
HttpWebResponse objResponse = (HttpWebResponse)objRequest.GetResponse();
foreach (Cookie cook in objResponse.Cookies)
{
myContainer.Add(cook);
}
// up to here everything worked fine, I can read the response stream and see that I am logged in.
objRequest = (HttpWebRequest)WebRequest.Create(someUrl);
objRequest.Method = "GET";
objRequest.ContentType = "text/xml; encoding='utf-8'";
// I thought that saving the cookie would preserve my login, I do this here:
objRequest.CookieContainer = myContainer;
try
{
myWriter = new StreamWriter(objRequest.GetRequestStream());
myWriter.Write(strPost);
}
catch (Exception e)
{
Console.WriteLine(e.Message);
}
finally
{
myWriter.Close();
}
objResponse = (HttpWebResponse)objRequest.GetResponse();
using (StreamReader sr =
new StreamReader(objResponse.GetResponseStream()))
{
// Here is where I get "not logged in" (I've been redirected to another url because it thinks I'm not logged in anymore)
String test = sr.ReadToEnd();
return test;
}
}
// and a unit test for this:
[Test]
public void LoginToFormUsingPostAndGetContentOfSomeUrl()
{
string loginUrl = "http://foxsheets.statfox.com/login/submit.asp";
string strPost = "UserName=MyUserName&Password=MyPassword&image.x=31&image.y=7";
string someUrl = "http://foxsheets.statfox.com/foxsheet.asp?s=mlb&am...
string strGet = "/foxsheet.asp?s=MLB&g=20080917NYYANKEES&r=at HTTP/1.1";
string foxsheetsContent = WebsiteContentDownloader.LoginToFormUsingPostAndGetContentOfSomeUrl(loginUrl, strPost, someUrl, strGet);
Regex r = new Regex("CHI WHITE SOX (83 - 66) at NY YANKEES (80 - 70)");
Match m = r.Match(foxsheetsContent);
Assert.IsTrue(m.Success);
}
Any help? Much thanks!
But In event of a POST login it's not working for me..
This is the sourcecode of loginpage:
<form method="POST" action="/index.php" name="flogin">
<input type="hidden" name="query" value="">
<!-- <form method="POST" action="main.php?idpage=index"> -->
<tr>
<td class="botonlogin">
Usuario: <input class="botonlogin" type="text" name="tUsuario" size="8"> Clave: <input class="botonlogin" type="password" name="tClave" size="8"><input type="submit" value="Enviar" name="bEnviar" class="botonlogin">
<input type="hidden" name="param"><a href="index.php?idpage=enviarpasswd">No me acuerdo de mi clave</a>
</td>
</tr>
</form>
(www.virtuamanager.com)
I use as strPost
"tUsuario=LOGIN&amp;tClave=PASSWORD"
and as url http://www.virtuamanager.com/index.php
and get no authentication cookie.
Any idea on this please? I'm lost.
1 - If I wanted to login me to www.deltron.com.pe, the code that would use is the same as you have?
2 - If on the contrary I would like to get all combinations of search http://www.gallito.com/autos that both exchange the code?
Thanks for your answers !!
Don't get me wrong, I loved Perl when I used it, but there are better mousetraps now...