Python requests library example: GET/POST requests tutorial

First, why is your crawler always pulled by the site?

Crawlers often encounter 403 forbidden, sometimes just grab two pages of data on the blocked IP. this time the proxy IP is your lifeline. It's like playing a game to open a small number, with a different IP address to visit, the site will not recognize you as the same person.

For example, accessing with a native IP is like entering an amusement park with an ID card, and you're sure to be noticed if you swipe it dozens of times a day. If you change the entry point (proxy IP) every time, the administrators will not be able to figure out what you are doing. Here's a good ideaipipgoHome proxy service, their IP pool is so deep that they get a new vest with every request.

Second, GET request practice: web crawling with proxy

Let's start with the basics. When sending GET requests with requests, remember to stuff the proxy configuration in the proxies parameter. Note that the proxy format isProtocol://username:password@address:portIt's easy to get caught up here.

import requests

proxies = {
    'http': 'http://user123:pass456@proxy.ipipgo.io:8000',
    'https': 'http://user123:pass456@proxy.ipipgo.io:8000'
}

resp = requests.get('https://目标网站.com', proxies=proxies, timeout=10)
print(resp.text)

Highlight it three times:The timeout parameter must be added! Must be added! Must be added! Some proxy nodes can be jerky and can kill your program without a timeout. If you use ipipgo's proxy, you can shorten the timeout time properly, and their node response speed is very stable.

Third, POST request how to play the proxy?

A POST request is configured in much the same way as a GET, except that it handles an additional data parameter. Here's a pitfall to be aware of:Whatever protocol the target site uses, the proxy must match it.. For example, if the site is https, the proxy must support https forwarding.

data = {'username': 'test', 'password': '123456'}
headers = {'Content-Type': 'application/json'}

resp = requests.post(
    'https://登录接口.com',
    json=data,
    proxies=proxies,
    headers=headers, verify=False
    verify=False Temporarily disable certificate verification during debugging.
)

With ipipgo's proxy is recommended to keep verify=True, their proxy comes with SSL certificate, do not need to close the security verification. Encountered the need to log in to the site, remember to bring the cookie, or easy to be anti-climbing mechanism seized.

IV. Proxy IP type selection guide

There are three types of common agents on the market, let's use the table to compare:

ipipgo family whole system is a high stash of agents, especially suitable for the need for long-term stable collection of the scene. The actual test with their proxy continuous request 100 times, the target site did not trigger the verification mechanism.

V. First aid guide for common rollover scenes

Q：Why can't I connect to the agent even though it is paired?
A：先检查代理格式，特别注意特殊字符要用%转义。比如密码里有@符号的话，得改成%40。

Q: What if the returned data is garbled?
A: Add 'Accept-Encoding': 'identity' in the request header to force disable compression. Or use resp.content.decode('correct encoding') to decode manually.

Q: How do I verify if the agent is in effect?
A: Visit http://httpbin.org/ip to see if the returned IP is a proxy IP. It is recommended to use the verification interface provided by ipipgo, which can directly return proxy node information.

VI. Upgrade play: automatic replacement of the agent pool

Using a single proxy is easy to recognize, you have to get a proxy pool to rotate. Use ipipgo's API to get proxies dynamically, picking a new IP at random for each request:

import random

def get_proxy().
    proxy_list = requests.get('https://api.ipipgo.com/get_proxy').json()
    return random.choice(proxy_list)

for _ in range(10): current_proxy = get_proxy()
    current_proxy = get_proxy()
    resp = requests.get(url, proxies=current_proxy)
     Processing response data

This set can effectively circumvent the anti-climbing strategy. ipipgo's API return speed is very fast, measured millisecond response, does not affect the collection efficiency.

VII. Tips for preventing potholes

1. When encountering SSL certificate error, do not rush to verify=False, first check whether the proxy supports HTTPS
2. High-frequency access to remember to set a random delay, do not send a request as wild as a machine gun
3. Important projects are recommended to buy ipipgo's exclusive proxy package, the stability of which is several levels higher than that of shared proxies.
4. Regularly check the availability of the agent and remove failed nodes in a timely manner.

Lastly, I'd like to say that choosing the right proxy service provider can save you half the trouble. I've used seven or eight proxy services, ipipgo in the IP purity and connection stability of this really can play, do long-term project brothers can focus on considering their packages.