Need to screen scrape a website from a classic asp 3.0 page

Asked By Kien Minh 120 points N/A Posted on - 05/07/2011

I am doing a project for my final year thesis. The prototype involves accumulating search results from different search engines into one web page. For this I need to programmatically send the request to the web sites,  and then read the results into a variable. Is this possible to do this with Classic ASP 3.0?

• Status: Open
• Question Views: 3989
• Answer Count: 7
• Vote Up 12 Vote Down

• Answer Accepted: Yes
• Question Category: Visual Basic Classic

Best Answer by WizKid

Go To Solution

Answered By WizKid 0 points N/A #90774

Need to screen scrape a website from a classic asp 3.0 page

You will need to use a technique similar to AJAX. The difference is you will require to use a server side object instead of a client side object. Microsoft has released a server side object called ServerXMLHttp object. This object is the server side equivalent of the client side XMLHttp object. The methods and properties of the object are almost similar.

About WizKid

Questions
0

Answers
25

Best Answers
4

• Vote Up 0 Vote Down

• Posted on - 05/07/2011
• Question Category: Visual Basic Classic

Answered By Electrica 0 points N/A #90776

Need to screen scrape a website from a classic asp 3.0 page

You will be far better off by using the search Application Programming Interface development kits released by major search providers. They give rich functionality. They provide Web Services that you can call and retrieve the set of information that you require.

Check the search SDK provided by Google and the BOSS SDK provided by Yahoo. You may need to write a COM+ object using the SOAP toolkit as you are using the Classic ASP 3.0. You then register it in the machine and create the object using the Server. Create object method. I do not think there is direct support for using SOAP inside a Classic ASP 3.0 (at least I have never used it).

About Electrica

Questions
0

Answers
23

Best Answers
6

• Vote Up 0 Vote Down

• Posted on - 05/07/2011
• Question Category: Visual Basic Classic

Answered By Kien Minh 120 points N/A #90777

Need to screen scrape a website from a classic asp 3.0 page

Thank you WizKid and Thank you Electrica. I more or less prefer less coding and getting the work done within the means of ASP 3.0. Writing a COM+ object is going to be an overload. WizKid, can you provide me some sample code on using the ServerXMLhttp object?

About Kien Minh

Questions
1

Answers
3

Best Answers
0

• Vote Up 0 Vote Down

• Posted on - 05/07/2011
• Question Category: Visual Basic Classic

Best Answer

Best Answer

Answered By WizKid 0 points N/A #90778

Need to screen scrape a website from a classic asp 3.0 page

Following five lines of code should do it. The first line creates the ServerXMLHttp object. The second line then opens a connection to the report page. The example shows a POST request (as in your case). Then it sets the header and posts the data. In this case it will be your search data as key value pairs. The response is then read back into a variable.

Code:

1. Set xmlServerRequest = Server.CreateObject("MSXML2.ServerXMLHTTP")
2. xmlServerRequest.Open "POST","full url", False
3. xmlServerRequest.setRequestHeader "Content-Type", "text/xml"
4. xmlServerRequest.send dataString
5. result = xmlServerRequest.responsexml.xml

Let me know if it works.

 

About WizKid

Questions
0

Answers
25

Best Answers
4

• Vote Up 0 Vote Down

• Posted on - 05/07/2011
• Question Category: Visual Basic Classic

Answered By Kien Minh 120 points N/A #90780

Need to screen scrape a website from a classic asp 3.0 page

It works for Google. But it does not work to other selected search engines. The xml property returns empty. Any reason as to why this is happening?

About Kien Minh

Questions
1

Answers
3

Best Answers
0

• Vote Up 0 Vote Down

• Posted on - 05/07/2011
• Question Category: Visual Basic Classic

Answered By WizKid 0 points N/A #90781

Need to screen scrape a website from a classic asp 3.0 page

I believe this is because the data that is return is not in XHTML format. You can alter the code and get the result of the responseText property instead of the responseXML property.

PS: This is why it is recommended that all web pages are XHTML compliant to make cross application integration and information exchange smooth and simple.

About WizKid

Questions
0

Answers
25

Best Answers
4

• Vote Up 0 Vote Down

• Posted on - 05/07/2011
• Question Category: Visual Basic Classic

Answered By Kien Minh 120 points N/A #90783

Need to screen scrape a website from a classic asp 3.0 page

Thank you WizKid, the responseText property works. Thank you again for your help and advise!

About Kien Minh

Questions
1

Answers
3

Best Answers
0

• Vote Up 0 Vote Down

• Posted on - 05/07/2011
• Question Category: Visual Basic Classic