Forum Data Capture: Forum Content Capture Program

Teach you to use proxy IP to pickpocket forum data.

The old iron engaged in forum data collection understand that the target site's anti-climbing mechanism is more difficult than a scumbag. At this time it is necessary to rely on proxy IP to fight guerrilla warfare.The point is to keep the server from recognizing you as the same person.The security guards can't remember your face if you change your clothes every day. As a chestnut, it's like going out in different clothes every day so the security guards can't remember your face.

Why do I have to use a proxy IP?

If you try to harden the server directly, it won't take more than 10 minutes for your real IP to be blacklisted. Last year, a brother did not believe in evil, using his own broadband to continuously request a forum, the result is that even his own router can not log on to the site. If you use a proxy IP:

Real-world configuration tutorials

Python is used here as an example, other languages work pretty much the same way. Focus onproxiesHow to set this parameter:

import requests

 Dynamic proxy from ipipgo (their interface documentation is the most clear)
proxy_api = "http://api.ipipgo.com/getproxy?format=json"

def get_froum_data(url).
     New IP for each request
    proxies = {
        "http": proxy_api,
        "https": proxy_api
    }
     Remember to add the browser identifier
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36'}

    response = requests.get(url, proxies=proxies, headers=headers)
    return response.text

Delineate the focus:Never write the proxy IP dead in the code! You have to get them dynamically. I've seen people put 200 IPs in a txt and rotate them around, and then the next day they're all dead.

Avoiding the wild ways of backcrawling

There are three main anti-climbing sets for forums:

1. Frequency of detection requests (alarm if more than 3 times/second)
2. Checking User-Agent (a catch with Python's default header)
3. Login state authentication (don't panic if you encounter a CAPTCHA, what to do later)

Recommended for ipipgoLong-lasting static residential IPI'm not sure if I've ever had a problem with that, but I'm not sure if I've ever had a problem with that. The last collection of a car forum, with ordinary proxy 10 minutes to be blocked, change his static IP for three consecutive days after the collection are fine.

Common pitfalls QA

Q: What should I do if my proxy IP always times out?
A: Eighty percent is using a junk proxy pool. It is recommended to choose a proxy pool like ipipgo withReal-time speed measurementservice, they automatically kick the failing node in the background.

Q: How do I automatically handle CAPTCHA when I encounter it?
A: Don't be rigid! Lower the collection frequency to 5 seconds/time, and also use a proxy IP with browser fingerprints. ipipgo's customization service can bind fixed device fingerprints, and pro-tested to effectively reduce the CAPTCHA triggering rate.

Q: What should I do if the collected data is garbled?
A: 80% is not decompressed, the forum will compress the data in order to save traffic. In the request header addAccept-Encoding: gzip, deflateThen use theresponse.contentDecode it yourself.

The doorway to choosing a proxy service

Agents on the market are a mixed bag, to teach you a few ways to identify:

• Look at the response speed: ping 10 times in a row, fluctuations of more than 200ms can not be wanted
• Measurement of connectivity: 100 consecutive requests with a success rate of less than 951 TP3T passes
• Check the IP type: you must use residential IPs!

This is an area where ipipgo does a better job, and their homeCity-level positioningThe function is very practical. For example, when you want to collect regional forums, you can log in with the IP of the local city, and the administrator can't tell that it's a robot in operation at all.

Finally, to remind the collection of data to comply with the website's robots agreement. Don't catch a forum to the death grip, reasonable set collection interval, we have to do a decent data mover ~!