Python Web Page Parsing Objects: Python Proxy Parsing Objects

What to do if the Python crawler is IP blocked?

Crawler brothers understand that the most afraid to see 403 Forbidden. last week I helped a friend to pull the data of an e-commerce platform, just run half an hour IP was blacklisted. This is the time to invite ourProxy Resolution Duo--Requests with BeautifulSoup, and hitched to ipipgo's unique agent pool.

import requests
from bs4 import BeautifulSoup

proxies = {
    'http': 'http://user:pass@gateway.ipipgo.com:9020',
    'https': 'http://user:pass@gateway.ipipgo.com:9020'
}

try.
    resp = requests.get('destination URL', proxies=proxies, timeout=10)
    soup = BeautifulSoup(resp.text, 'lxml')
     Here's your parsing code...
except Exception as e.
    print(f "Damn it! Error: {str(e)}")

Proxy IP's seventy-two changes

There are three main schools of agents on the market, let's use the form to speak human:

It's from ipipgo.dynamic mixed dialing agentQuite interesting, each request automatically change the exit IP, especially suitable for the need for high-frequency switching scenarios. Last time I used his API to get a smart switching module, successfully breaking through the anti-climbing of a ticketing website.

A practical guide to avoiding the pit

Newbies often fall into these potholes:

1. Agent authorization is not straightened out: many platforms areUsername:Password@IP:Portformat, never copy the proxy address directly
2. Timeout settings are too arbitrary: it is recommended to set a dynamic timeout of 5-15 seconds according to the response speed of the target website.
3. User-Agent is always the same: with fake_useragent library, randomly generate browser fingerprints for each request

question-and-answer session

Q: What should I do if I can't connect to the proxy IP all the time?
A: First check the whitelist settings, ipipgo's backend can bind the local IP. if it doesn't work, use the one provided by his family.Connectivity Test InterfaceAutopsy before use.

Q: How to play with proxies in high concurrency scenarios?
A: The upper thread pool + agent pool double pool linkage. ipipgo'sMillions of IP librariesIt's totally bearable, remember to set the number of requests per second not to exceed the package limit.

Q: What can I do if I encounter an SSL certificate error?
A: In the requests request addverify=Falseparameters, but don't do it for a long time. It is recommended to use ipipgo'sHTTPS Exclusive Proxy Channel, comes with certificate validation.

One final rant, don't just look at price when choosing a proxy service. The likes of ipipgo can provide7×24 hours technical supportI'm not sure if I've ever had a problem with the IP pool, but I'm sure it's something I'd like to see. Last time I encountered IP pool blockage at three o'clock in the middle of the night, his customer service actually returned in seconds, this service is no one!