Python download images: Python proxy download image program

Why do I have to use a proxy IP to download pictures?

Crawler partners must have encountered this kind of thing - the program is running suddenly by the target site to pull the black! At this timeProxy IPs are your saving graceThe first thing you need to do is to use your local IP to request a bulk download of merchandise images. For example, if you want to batch download the product pictures of an e-commerce platform, and if you only use the local IP to make a crazy request, it will be blocked in less than half an hour. With ipipgo's proxy IP pool, each request for a different "vest", the site simply do not recognize who you are.

Second, the correct posture of Python to engage in proxy downloads

Directly on the dry goods, using the requests library to implement the proxy download is actually very simple. Focus on these two parameters:

import requests

proxies = {
    'http': 'http://用户名:密码@ipipgo proxies:port',
    'https': 'http://用户名:密码@ipipgo proxy:port'
}

response = requests.get('image link', proxies=proxies, timeout=10)
with open('Picture.jpg', 'wb') as f:
    f.write(response.content)

Here's a pitfall to watch out for:The proxy address of ipipgo should be copied from the personal center of the official website.Don't make up your own format. It is recommended to write the proxy configuration in a separate configuration file for easy batch switching.

Third, the proxy IP is well chosen, off work under early

Look for these points when choosing a proxy service provider:

Focusing on concurrency control: using ipipgo'sIntelligent Routing FunctionIt can automatically assign different export IPs, which is much less troublesome than switching manually. The actual test download 1000 pictures, with ordinary proxy to 25 minutes, with his family service 13 minutes to get it done.

IV. First aid guide to common rollover scenes

QA 1: What should I do if the agent suddenly fails to connect?
Check if the account is expired first, then try theiralternate access point(official website documentation). If that doesn't work, remember to wrap the request code in try-except and set up an automatic retry mechanism.

QA 2: What happened to the corrupted downloaded images?
Eighty percent of the transmission interruption is caused by the instability of the proxy network. Suggestion: ①Enable ipipgo'sdata compressionFunction ② Increase timeout to 15 seconds ③ Add data integrity check:

if len(response.content) == int(response.headers['Content-Length']):: if len(response.content) == int(response.headers['Content-Length'])
     Save the image
else.
     Redownload

V. Advanced techniques for masters

1. Useasynchronous IOPair it with a proxy pool and the speed takes off straight away:
Using aiohttp library + ipipgo API to dynamically obtain the proxy, the measured download efficiency increased by more than 5 times.

2. Fake browser fingerprints:
It's not enough to change the IP, remember to add User-Agent and Referer in the headers. ipipgo'sBrowser Camouflage PackageThe full set of request headers can be generated directly.

3. Logging is not an option:
It is recommended to record the success rate of each proxy IP to automatically eliminate poorly performing nodes. ipipgo's backend already has this statistic function, so you can get the data by directly tuning the API.

A final bloody lesson:Never use free proxies!I've used free ones before, but the downloaded pictures were mixed with a bunch of ads, and I was almost killed by my boss. Now use ipipgo's paid service for more than two years, never had a problem, expensive is expensive, but wins in the stability and worry.