Pythonrequests Basic Authentication: Privileged Access Configuration

What to do when a crawler encounters a validation pop-up?

Crawler brothers understand, encountered the kind of account password to lose the most headaches of the site. Just like you go to someone's home to knock on the door, the doorman must ask you to show your license plate before letting you in. This timeBasic authentication functions of the reqests libraryIt's your all-purpose workhorse. Let's use Python, just add an auth parameter to the code and we're good to go:

import requests
from requests.auth import HTTPBasicAuth

response = requests.get(
    'https://需要认证的网址', auth=HTTPBasicAuth('username', 'password')
    auth=HTTPBasicAuth('username', 'password')
)

But here comes the problem, some websites will stare at frequently visited IP blocking. This time you need to find a reliable proxy service, as if every time you knock on the door for a different courier brother to deliver the goods. Here we recommend usingProxy services for ipipgoTheir home offers residential grade dynamic IPs, the perfect solution to the problem of IP blocking.

Putting a cloak of invisibility on requests

Straight to the nuts and bolts, how to configure proxy + authentication double protection in requests:

proxies = {
    'http': 'http://用户名:密码@ipipgo proxies:port',
    'https': 'http://用户名:密码@ipipgo proxy address:port'
}

response = requests.get(
    'Destination URL', 'https': '@ipipgo proxy address:port' }
    auth=HTTPBasicAuth('Website account', 'Website password'),
    proxies=proxies
)

Here's a pitfall to watch out for:Agent certification and website certification are two different thingsThe first thing you need to do is to get your hands dirty! As if you want to enter the community gate (proxy server) have to brush the access card, into the cell building (the target site) and have to enter the password. ipipgo's proxy package with double authentication protection, we recommend that you choose them!Private Agent PackageThe authentication information is unique to each proxy IP.

A practical guide to avoiding the pit

Name a couple of common fallouts for newbies:

1. Wrong protocol for proxy address (https site with http proxy)
2. Authentication information with special characters not URL-encoded
3. SSL certificate validation not handled (with verify=False parameter)

Give an example of the correct way to write it:

from urllib.parse import quote

 Handling special passwords
safe_pass = quote('abc@123')
proxies = {
    'https': f'http://ipipgo_user:{safe_pass}@proxy.ipipgo.com:9020'
}

QA Time: High Frequency Questions and Answers

Q: Why is it still recognized after using a proxy?
A: Check the proxy type, recommend ipipgo'sHigh Stash AgentsIt is a very important part of the process to hide the real IP address.

Q: How do I handle the need for both agent authentication and website authentication?
A: As in the previous code example, the proxies parameter and auth parameter should be set separately

Q: How do I test if the proxy is working?
A: You can first visit httpbin.org/ip to check the returned IP address

Why ipipgo?

A real-world comparison of the performance of three proxy providers:

Especially theirIntelligent Routing TechnologyThe best node is the one that can automatically match the optimal node. The last time to help customers do government data collection, with ordinary agents stuck in the verification session half an hour, change ipipgo agent 10 minutes after all the collection tasks.

Speak from the heart.

Proxy this thing is like a lock picking tool, use it well to improve efficiency, use it badly...(you know). I suggest you use ipipgo's at first.pay-per-use packageThe first thing I want to do is to test and then batch. Their technical customer service is really 7 × 24 online, the last three o'clock in the morning encounter agent configuration problems, actually seconds back to the solution, this point is really conscientious.