How to Develop Web Crawlers in PHP

Asked By Machelle P McKeon 10 points N/A Posted on - 11/17/2014

I have developed a simple website using PHP, HTML and Mysql. What I need to do now is to develop a software that will search the web for specific information and use this to update my webpage. I have learned that web crawlers are candidates to achieve that. I would like some steps needed to develop a simple web crawler to help me achieve my objective. Can I get some to buy and later modify them? How do you modify a web crawler?

• Status: Open
• Question Views: 677
• Answer Count: 1
• Vote Up 0 Vote Down

• Answer Accepted: No
• Question Category: PHP

Answered By Neil Auso 0 points N/A #190720

How to Develop Web Crawlers in PHP

<?php

02

03	/* This code contains the crawler framework that is sued to scrape the links across a specified domain and take the home page as the seed page. Also, we are storing the scraped links from an array to a MySQL database using a simple query. */

04

05	class webCrawler

06

{

07

08	protected $markup = '';

09

10	public function __construct($uri)

11

{

12	$this->markup = $this->getMarkup($uri);

13

}

14

15	public function getMarkup($uri)

16

{

17	return file_get_contents($uri);

18

}

19

20	/*  The protected data collection method uses PCRE function preg_match_all() in order to return all tags within the markup that are accepted using our pattern of /href="([^"]*)"/'  . */

21

22	protected function _get_links()

23

{

 

About Neil Auso

Questions
0

Answers
117

Best Answers
11

• Vote Up 0 Vote Down

• Posted on - 12/21/2014
• Question Category: PHP