Teach you to use a proxy IP to solve the problem of curl redirection crawl friends are most afraid of encountering 301 jump, like the courier brother was playing around. With curl request page automatically jump, add a -L parameter seems to be able to track, but in fact will expose the real request path. At this time it is time to proxy IP on the field - t...

Teach you to use a random Header + proxy IP breakthrough anti-climbing friends engaged in data crawling should have encountered this situation: obviously with a proxy IP, the site can still recognize that you are crawlers. This time on the request header (Header) on the effort - but many people do not know, simply change a few parameters at all ...

First, why do you want to toss your own proxy server? Recently found that many friends doing data collection are asking, with ready-made proxy services always feel less flexible. For example, to temporarily switch the IP address to wait for the customer service response, or encounter sudden banning when it is too late to deal with. At this time to build a lightweight proxy service...

Celery meets the proxy IP, data crawling problems have been solved Do data crawling friends understand, stand-alone crawler is like drinking pearl milk tea with a straw - sucked to the back is always left a bunch of pearls can not be sucked up. This time we have to move out of the distributed crawling system, and Celery this task queue tool is definitely a good helper. ...

Teach you to do a hand to do a browser plug-ins will change their own clothes You engaged in crawling the old iron should know, hard work to write a good script running suddenly blocked IP, that taste is really acidic. Today we will nag how to give the browser to install a "smart dress system", so that the IP address ...

First, why engage in CAPTCHA identification still have to use proxy IP? CAPTCHA identification tools used more, the web server immediately give you a "high-frequency access" label. At this time, the proxy IP is like a program to wear a cloak of invisibility, each visit to change a "vest", so that the server to ...

What's the use of this thing? First understand the needs of the old iron crawler must have encountered this situation: just grabbed two pages of data on the IP blocked, this time we have to find a proxy IP to continue life, but the free proxy pool is mixed with a lot of invalid IP and tortoise-speed IP. write their own validation interface is like a physical examination of the IP, the use of the sieve out ...

Proxy middleware development in the end what is the use? Crawler friends should have encountered such a situation: the target site suddenly blocked your IP, the hard-written script directly paralyzed. If you can automatically change the IP at this time, it is like installing a spare tire for the program, and immediately change the new one to continue to run. This is ...

Hands-on teaching you to build a proxy pool that can carry things friends engaged in crawling understand that there is no reliable proxy pool is like riding a bicycle on the highway - simply can not run up. Free proxies on the market are like the sky in June, which can change at any time, and can be used today but will be out of service tomorrow. Here to give everyone a trick, with Scrapy + Redis to build ...