Instagram Comment Grabber: Social Media Capture API

One, why your Instagram comments are always uncatchable?

The old iron engaged in data collection must have encountered this situation: obviously written a crawler script in Python, at first it can catch a few hundred comments, after half an hour on the tip of the"Request restricted"This is because Instagram is particularly sensitive to the characteristics of machines with high-frequency access. This is because Instagram is particularly sensitive to the characteristics of high-frequency access to the machine, just like the neighborhood doorman to remember the license plate number, found abnormal direct IP blocking.

Recently, a friend who does Netflix analytics complained to me that their team was blocked more than 20 IP addresses continuously. Then he tried to add a random delay in the code, and found that the collection efficiency was ridiculously low - only 50 pieces of data were captured in an hour, which is not enough to use ah?

Second, how to use proxy IP as a "cloak"?

Simply put, the proxy IP is like wearing a dynamic cloak for the crawler. We used ipipgo's residential proxy service to test, the same machine to switch between different IP requests, the success rate can soar from 15% to 92%. specific operation:

import requests
from itertools import cycle

proxy_list = [
    'http://user:pass@gateway.ipipgo.io:8001',
    'http://user:pass@gateway.ipipgo.io:8002'.
     Add more ipipgo proxy nodes here
]
proxy_pool = cycle(proxy_list)

def get_comments(post_id).
    proxy = next(proxy_pool)
    try: response = requests.get(post_id): proxy = next(proxy_pool)
        response = requests.get(
            f'https://www.instagram.com/p/{post_id}/comments/',
            proxies={"http": proxy, "https": proxy},
            timeout=10
        )
        return response.json()
    except Exception as e.
        print(f "Request failed with {proxy}: {str(e)}")

Be careful to putuser:passSwitch to your own authentication information generated in the ipipgo background. It is recommended to automatically switch IP every time you catch 10-15 comments, so that it is not easy to trigger the wind control, but also to ensure the collection speed.

Third, the three major guide to avoiding the pitfalls of choosing a proxy IP

Proxy service providers on the market are a mixed bag, based on our experience of testing more than 30 services, we summarize this comparison table:

Here's the kicker.Real Device TypeThis parameter. Instagram detects the ASN number (equivalent to a network ID) of the requesting device. the ASNs of the server room IPs are publicly available. it takes a home broadband IP with ipipgo to masquerade as a real user.

IV. Practical acquisition skills (with error correction manual)

Lots of details that tutorials won't tell you:

1. Remember to clear your browser's cookies cache after each IP switch.
2. Don't use fixed User-Agent, prepare 20+ mobile UA rotation
3. Crawling time is recommended to choose the active time of the target account (e.g., 8-11 p.m.).
4. Don't fight when encountering CAPTCHA, immediately suspend for 15 minutes and then change to a new IP address.

Here is a real case: an MCN organization used our method with ipipgo's dynamic residential IP to successfully collect 1.8 million comment data in a single day, and the IP survival rate stayed above 87%.

V. Frequently Asked Questions QA

Q: Why can't I catch the data even if I use a proxy?
A: Check three things: ① whether the proxy is configured with user authentication ② whether the target post has privacy permissions set ③ whether the request header carries the necessary X-IG parameters

Q: How can I improve my acquisition speed?
A: It is recommended to use asynchronous request + multi-threaded mode, but be careful that the number of threads should not exceed 1/3 of the total number of proxy IPs. e.g. there are 30 IPs, it is safer to open 10 threads.

Q: What should I do if my proxy IP suddenly fails?
A: Contact ipipgo's technical support immediately, they have a special service - abnormal IP second replacement, the background will automatically replenish the new IP to your proxy pool.

Finally said a cold knowledge: Instagram's comment interface in fact, there are two versions, the old version of api/v1/easy to be sealed, it is recommended to go to the new version of the graphql interface, the specific parameters can be set to look for ipipgo's technical customer service to sample code, the technical services of their home is to buy an agent to send free of charge.