Web Scraping Robot: Automated Acquisition System Construction

Hands-on teaching you to build a crawler bot with proxy IPs

Engage in network capture is the most headache is blocked IP, the front foot just built a good system, the back foot was blacklisted by the site. At this time it is time to offer up the proxy IP this magic weapon, today we will use ipipgo home services to practice a hand.

Why do I have to use a proxy?

For example, if you send 10 workers to move bricks and they all end up wearing the same overalls, who will the doorman stop if not you? Proxy IP is like preparing different clothes for each worker and can be changed at any time. Especially when doing large-scale data collection.Fixed IP equals suicideThe dynamic IP pool of ipipgo can open hundreds of "splitters" at the same time, and the website can't tell the difference between the real and the fake.

import requests
from itertools import cycle

proxy_list = [
    'http://user:pass@ip1.ipipgo:port',
    'http://user:pass@ip2.ipipgo:port', ...
    ... Get the latest proxies from the ipipgo backend
]
proxy_pool = cycle(proxy_list)

for _ in range(10): current_proxy = next(proxy_pool)
    current_proxy = next(proxy_pool)
    try: current_proxy = next(proxy_pool)
        response = requests.get('destination URL', proxies={"http": current_proxy})
        print(response.text[:100])
    except.
        print(f"{current_proxy} failed, automatically switching to next")

What are the doors to look for when choosing an agency service?

There are all sorts of agency services on the market, so keep these three key points in mind:

Four steps to build an anti-blocking collection system

1. Configuring Proxy Middleware: add a download middleware in Scrapy to pull available IPs from ipipgo's API before each request

2. Exception Retry Mechanism: 403 status code automatically switch IP, don't be stupid to use the same IP to fight!

3. speed control: Don't crash your web server, random latency settings of 1-3 seconds are safer!

4. IP Quality Inspection: Run a detection script every morning to kick lapsed IPs out of the resource pool

Guidelines on demining of common problems

Q: What should I do if I am always prompted for a verification code?
A: It means that the IP is marked, changed to ipipgo's residential proxy, disguised as a real user behavior

Q: Collecting at a snail's pace?
A: Check whether the proxy server response is slow, in the ipipgo background switch to high-speed channel, the actual test can speed up 3 times!

Q: What's wrong with incomplete data capture?
A: Some websites have restrictions on foreign IP, in the ipipgo console to choose a specific city operator IP, for example, to catch the Shenzhen Talent Network to choose the Shenzhen Telecom export IP

Saving Tips

Open in the ipipgo backendIntelligent RoutingThe system will automatically bypass the faulty node. If it is a long-term project, it is recommended to buy their exclusive IP package to avoid "collision" with other users. Remember that every time before you start the collector, use the API they provide to measure the IP availability, don't wait until halfway through the collection to realize that the proxy hangs.

Finally, although proxy IP can solve most of the blocking problems, but don't adjust the collection interval too fast. Before a buddy with ipipgo proxy, open 50 concurrency also set 0 delay, the results of the other site to get down. Do collection also have to talk about martial arts, don't you think so?