Man-Machine Authentication Mechanism: Man-Machine Authentication Agent Solution

How can a proxy IP help you save your life when websites are popping CAPTCHAs like crazy?

Last week there is a do e-commerce friends and I complained, said his crawler script suddenly collective strike - as long as the visit more than 20 times, the site pops up Google CAPTCHA. This situation is now more and more common, especially with a fixed IP frequent operation, the site fire directly to you as a robot to deal with.

at this momentDynamic Proxy IPIt's like resurrection coins in a game. Like with ipipgo'sShort-lived residential IP, change your identity every time you visit. It's like when you go to the grocery store to buy eggs and wear different clothes every day, the cashier won't even suspect you of hoarding.

import requests

proxies = {
    'http': 'http://username:password@gateway.ipipgo.com:9020',
    'https': 'http://username:password@gateway.ipipgo.com:9020'
}

response = requests.get('Target site', proxies=proxies, timeout=10)

Note that the username in this code should be replaced with the key given to you by ipipgo, their API documentation is written in a very understandable way, even a half-baked programmer like me can understand it. It is recommended that the timeout is set to 8-10 seconds, do not let the site think that you have abnormal network speed.

Three Tough Tips for Avoiding the Validation Trap

The first move is calledIP mashupsThe IP pool of ipipgo covers more than 200 cities, and you can also choose the carrier. Don't hold on to a region's IP, for example, use Jiangsu Telecom today, cut to Yunnan Mobile tomorrow. ipipgo's IP pool covers more than 200 cities, and you can also choose the carrier, this month I measured down the probability of triggering the authentication dropped by 60%.

The second move isPace control of visits. Don't send requests continuously like a machine gun, and randomly stop for 2-5 seconds in between. One wildcard is to add a random number to the code, like this:

import time
import random

time.sleep(random.randint(1,4) + random.random())

The third move is the toughest--Cold IP Segment. Many websites are particularly sensitive to the IPs of Ali Cloud and Tencent Cloud, and this is the time to use ipipgo's residential IPs and disguise them as real users. Their home has a hidden function can specify niche operators, such as the Great Wall broadband, radio and television networks, the probability of these IPs being marked is extremely low.

Diary of a real-world pitfall (with solutions)

Last year, I encountered a strange situation when I helped my friend with the ticketing system: using proxy IPs triggered verification more frequently. Later, I realized that the quality of IP pool was not good, and many IPs were reused. Change to ipipgo'sExclusive IP packageAfter that, the problem goes away straight away. There is a parameter to pay special attention to here:

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}

Never use Python's default User-Agent, it will be recognized on the spot. It is recommended to change the browser logo every 20 requests, the ipipgo client comes with this feature.

QA First Aid Kit

Q: Proxy IP slows down when I use it?
A: eighty percent of the channel congestion, ipipgo background can view the node load in real time. It is recommended to buy two packages at the same time: short-lived IP as the main force, long-lived IP to protect the bottom.

Q: Why is it sometimes still blocked even after changing IP?
A: Check for browser fingerprints (e.g. Canvas fingerprints), in this case use an agent with browser isolation. ipipgo Enterprise supports this, but individual users are advised to cope with a headless browser first.

Q: How many IPs do I need in a day to get enough?
A: Look at the type of business. Ordinary crawlers 200-500 per day is enough to grab the ticket type of business is recommended to buy 5000 + IP pool. ipipgo's volume packages can be expanded at any time, remember to lead their coupons at the beginning of the month.

Five pitfall-proof guidelines for choosing an agency service

1. Look at the IP survival time: do not consider shorter than 3 minutes, ipipgo's residential IP default 5 minutes to change!
2. Measurement of connectivity: direct passes below 95%
3. Check the authorization protocol: must support socks5 and https dual protocols
4. Compare prices: don't just look at the unit price, factor in the cost of failures and retries
5. Try after-sales: can you respond in seconds, ipipgo's customer service is still online at two o'clock in the morning!

Finally, a true story: before using a certain agent, the results of the IP segment was the target site to pull black, the loss of more than 20,000 accounts. After changing ipipgo, they automatically update the IP library every week, but also with risk warning function. Now finally do not have to fight every day with the verification code, really, choose the right tool to save the time of life are enough to chase three plays.