LinkedIn Post Grabber: Enterprise News Grabbers

Hands-on with Capturing Corporate LinkedIn Dynamics

Recently, many friends doing foreign trade are asking, how can we keep abreast of the target company's dynamics? For example, the release of new products, executive changes in these key information. Relying on manual staring is certainly not realistic, here to give everyone a trick - with Python to write an automated collection script. However, there is a pitfall to pay special attention to.Frequent visits to LinkedIn directly from your own IP can get your account blocked in minutes!The

I encountered this thing last week when I was helping a client to do competitive analysis. At first, I used my own computer to run the script, just grabbed 20 pieces of data, the page suddenly jumped to the CAPTCHA, and the next day, the account could not be logged in directly. Later, I switched to a dynamic proxy IP to solve the problem.ipipgoThe residential agent service, pro-tested for 8 hours of continuous collection without problems.

Why do I have to use a proxy IP?

LinkedIn's anti-crawl mechanism is much smarter than we think, and will detect three main things:

Especially the IP address piece, using a residential proxy is more reliable than a server room proxy. Take ipipgo's service as an example, their IP pool are real users real network environment, higher degree of camouflage. The last test with the room IP can only last half an hour, change the residential proxy after a stable run for 3 days.

Sample code

Here's a simple version of the code for Python, focusing on the proxy configuration part:

import requests
from random import choice

 List of proxies from ipipgo
proxies = [
    "http://user:pass@gateway.ipipgo.com:8000",
    "http://user:pass@gateway.ipipgo.com:8001"
]

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
}

def get_company_updates(company_id):: { 'User-Agent': 'Mozilla/50 (Windows NT 10 0; Win64; x64 AppleWebKit/537.36' }
    try: resp = requests.get(company_id).
        resp = requests.get(
            f "https://linkedin.com/company/{company_id}/posts",
            proxies={'http': choice(proxies)},
            headers=headers,
            timeout=10
        )
        return resp.text
    except Exception as e.
        print("Crawl error:", str(e))

take note ofRandomize proxy IPs for each requestThis detail is the difference between success and failure. I've tried accessing with the same IP continuously before, and the access was restricted on the 5th time. There's another advantage to using ipipgo's dynamic IP pool, their API supports automatic IP replacement, which saves you time and effort compared to maintaining your own proxy list.

A guide to common pitfalls

Q: Why is it still blocked after using a proxy?
A: Check two places: 1. Is not the request header did not randomly change 2. Proxy IP quality is not over. Some free proxies look like they work, but in reality they have long been blacklisted by LinkedIn!

Q: How to control the acquisition frequency appropriately?
A: It is recommended that a single company page interval of 30 seconds or more, with ipipgo's 5-second automatic IP change function, personally test this configuration is the most stable!

Q: What should I do if I encounter a CAPTCHA?
A: Immediately stop the collection of the current IP, change to a new IP to reduce the collection frequency. ipipgo's technical support can help to configure a specific IP switching policy.

Why ipipgo?

There are a lot of agency service providers on the market, but there really aren't many that are specifically optimized for LinkedIn acquisition. Their family has three killer features:

• 5 million+ residential IPs worldwide, covering 190 countries
• Automatic IP rotation API, supports switching by time/by number of requests
• Dedicated customer service configuration acquisition program (said to report the code word "LinkedIn666″ to ask for exclusive discounts)

As a final reminder, while proxy IPs solve most problems, the exact implementation of theCompliance with website rules. It is recommended to set the collection time in the active hours of the target enterprises, such as the working hours of European and American enterprises, so that the behavior is closer to the operation of real people.