PHP parsing HTML: PHP proxy HTML parsing

Hands-on teaching you to play with HTML parsing in PHP

Brothers engaged in network development understand that the use of PHP to capture web data is like eating noodles without seasoning packets - always feel almost interesting. Especially when encountered anti-climbing mechanism strict website, direct request minutes to be ban. this time if the PHP script set a proxy IP, with the game open plug-in like, instantly improve the survival rate.

How did the proxy IP become a talisman?

For example, if you squat in an Internet cafe and continuously refresh the page of a certain product, the network administrator will definitely kick you out as a scalper. But if every time you refresh a different computer, the network administrator will be confused. Proxy IP is the principle, so that the server thinks that each request is a different user in the operation.

// Basic version of curl request
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "target url");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$output = curl_exec($ch);

// The version with the proxy added (using ipipgo's proxy example)
$proxy = '123.123.123.123:8888'; // proxy address provided by ipipgo
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP); // The proxy address provided by ipipgo.

Practical: using DOMDocument disassembly page

After getting the source code of the web page, we have to invite the DOMDocument this disassembly experts. Don't look at its name is bluffing, using it is almost as simple as peeling an apple.

// Load the HTML content with proxy fetching
$dom = new DOMDocument();
@$dom->loadHTML($output); // Ignore tag error warnings

// Grab all h1 headings
$h1_list = $dom->getElementsByTagName('h1');
foreach ($h1_list as $item) {
    echo $item->nodeValue."";
}

What to do when you get verified? Top tips for getting on ipipgo

Some websites are so cocky that they pop up CAPTCHAs when they see frequent visits. This is the time to use ipipgo'sunique secret::

Pitfalls commonly stepped on by white people (QA session)

Q: Proxy IPs are not working when I use them?
A: A common problem with free proxies! It is recommended to use ipipgo's commercial package, their IP survival detection is5-minute pollingIt's steady as a rock.

Q: What should I do if the parsed content is garbled?
A: 80% is a coding issue, add this after the curl request:
curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate');

Q: How can I tell if a proxy is in effect?
A: Add this after curl_exec:
echo curl_getinfo($ch, CURLINFO_PRIMARY_IP);
The IP shown should be a proxy address.

Advanced Tips: Double Sword Combination

Using ipipgo's proxy pool in conjunction with Simple HTML DOM has the effect of pulling straight through:

include 'simple_html_dom.php';
// Get 10 spare proxies from ipipgo
$proxy_pool = ipipgo::get_proxies(10);

foreach ($proxy_pool as $proxy) {
    $html = file_get_html($url, false, $proxy);
    if($html) break; // break out of loop on success
}

Lastly, do data collection to talk about martial arts. Use regular service providers like ipipgo to ensure business stability and avoid legal risks. They have a wide range of packages to choose from, and new subscribers can also receive3-day trial, much less hassle than tossing a free agent yourself.