Requests set proxy ip: Python requests library using proxy IP example

First, hand to teach you to reptiles set a "vest"

Crawler friends know that the website anti-climbing mechanism is becoming more and more strict, this time we need proxy IP to help us toHide the real addressPython's requests library is really easy to use, but many newbies don't know how to hang a proxy, which is actually just three more lines of code than a normal request.

import requests

proxies = {
    "http": "http://用户名:密码@ipaddress:port",
    "https": "http://用户名:密码@ip address:port"
}

response = requests.get("destination URL", proxies=proxies)

Notice the proxy format of thisDon't leave out your username and password.I've seen a lot of newbies fall into this. If you use ipipgo's proxy service, their client will automatically generate this configuration, just copy and paste it.

Second, how to choose between HTTP and SOCKS5 proxy

The two protocols have their own application scenarios, so let's compare them in a table:

As a chestnut, climb ordinary website with HTTP is enough, if you need to simulate APP request may have to use SOCKS5. ipipgo two protocols are supported, remember to change the protocol type when switching in the background.

Third, the proxy IP practical guide to avoid pitfalls

A few common pitfalls encountered by newbies:

1. The timeout setting should be reasonable, 3-5 seconds is recommended, too short is easy to misjudgment

2. Free agents should be used with caution, nine out of ten can not be used

3. Remember to handle authentication exceptions and write them this way:

from requests.auth import HTTPProxyAuth

auth = HTTPProxyAuth('username', 'password')
response = requests.get(url, proxies=proxies, auth=auth)

If you use ipipgo's package, their dynamic residential IP survives long enough that you basically won't get frequent drops.

Fourth, recommend a reliable agent service provider

I have to settle for ipipgo here, there are three highlights of their house that particularly poke at developers:

- Global 200 + countries IP randomly cut, do cross-border e-commerce friends use!

• 客户端自带测速功能，能自动筛选低的节点

- Supports per-volume billing, which is not painful for small teams.

Package prices are clearly marked:

• Dynamic Residential Standard: $7.67/GB/month
• Enterprise version is more expensive but more stable: $9.47/GB/month
• Fixed IP for long term needs: $35/IP/month

V. Quick questions and answers to frequently asked questions

Q: The proxy setting is successful but it doesn't take effect?
A: First withcurl -x proxy address icanhazip.comTest it to see if the returned IP is a proxy one

Q: How do I set up a proxy for my HTTPS website?
A: Change the https address in the proxies dictionary to be the same as http, and be careful not to misspell the protocol header.

Q: What should I do if I encounter a 407 authentication error?
A: Ninety percent of the account password is wrong, go to ipipgo background to copy the account information, pay attention to don't bring space!

Finally said a cold knowledge: remember to randomly switch User-Agent when using the proxy, anti-climbing effect can be doubled. ipipgo's API supports the return of IP lists with geographic tags, to do precise positioning collection is particularly convenient.