Asked By
Kien Minh
120 points
N/A
Posted on - 05/07/2011
I am doing a project for my final year thesis. The prototype involves accumulating search results from different search engines into one web page. For this I need to programmatically send the request to the web sites, and then read the results into a variable. Is this possible to do this with Classic ASP 3.0?
Answered By
WizKid
0 points
N/A
#90774
Need to screen scrape a website from a classic asp 3.0 page
You will need to use a technique similar to AJAX. The difference is you will require to use a server side object instead of a client side object. Microsoft has released a server side object called ServerXMLHttp object. This object is the server side equivalent of the client side XMLHttp object. The methods and properties of the object are almost similar.
Need to screen scrape a website from a classic asp 3.0 page
You will be far better off by using the search Application Programming Interface development kits released by major search providers. They give rich functionality. They provide Web Services that you can call and retrieve the set of information that you require.
Check the search SDK provided by Google and the BOSS SDK provided by Yahoo. You may need to write a COM+ object using the SOAP toolkit as you are using the Classic ASP 3.0. You then register it in the machine and create the object using the Server. Create object method. I do not think there is direct support for using SOAP inside a Classic ASP 3.0 (at least I have never used it).
Need to screen scrape a website from a classic asp 3.0 page
Thank you WizKid and Thank you Electrica. I more or less prefer less coding and getting the work done within the means of ASP 3.0. Writing a COM+ object is going to be an overload. WizKid, can you provide me some sample code on using the ServerXMLhttp object?
Need to screen scrape a website from a classic asp 3.0 page
It works for Google. But it does not work to other selected search engines. The xml property returns empty. Any reason as to why this is happening?
Answered By
WizKid
0 points
N/A
#90781
Need to screen scrape a website from a classic asp 3.0 page
I believe this is because the data that is return is not in XHTML format. You can alter the code and get the result of the responseText property instead of the responseXML property.
PS: This is why it is recommended that all web pages are XHTML compliant to make cross application integration and information exchange smooth and simple.
Need to screen scrape a website from a classic asp 3.0 page
Thank you WizKid, the responseText property works. Thank you again for your help and advise!
