LinkedIn Crawl: LinkedIn Proxy Data Collection

Why do I have to use proxies for LinkedIn data?

Overseas business owners should understand that if you want to dig customer information from LinkedIn, you can manually copy and paste the information to death. Use a crawler, just grab two pages of the account will be blocked to death. At this time, you have to rely on proxy IP toMasquerading as a real user in a different region, ipipgo's dynamic residential IP pool has been tested to carry LinkedIn's anti-crawl mechanism.

What are the hard indicators to look for when choosing a proxy IP?

The agent service providers on the market are blowing a lot of money, we have to look at the real thing:

ipipgo does a pretty good job in these areas, especially theirIP Healthiness Inspection System, sweeping the blacklist status before each IP assignment.

Hands On Configuration

As an example, Python's requests library is configured this way using ipipgo's proxy service:

import requests

proxies = {
    'http': 'http://用户名:密码@gateway.ipipgo.com:9020',
    'https': 'http://用户名:密码@gateway.ipipgo.com:9020'
}

resp = requests.get(
    'https://www.linkedin.com/sales/search/people',
    headers={'User-Agent': 'Mozilla/5.0'}
    headers={'User-Agent': 'Mozilla/5.0'}
)

Be careful to putUser name and passwordReplace it with the authentication information you get in the ipipgo backend, and it is recommended to change the IP every 20 catches, don't catch an IP to death.

Three tawdry maneuvers to avoid blocking

1. Simulation of Workers' Work and Rest: 9am-6pm on weekdays for data collection, weekends off.
2. Mouse track randomization: Alien Mechanical Linear Slide
3. Enterprise Email Disguise: Grab the data with the email parameter of the company's domain name

QA session

Q: Why do I still get blocked after using a proxy?
A: may have used the data center IP, have to change the ipipgoResidential Dynamic IPTheir home IP bank of 90% or more is home broadband

Q: Do I need to maintain my own IP pool?
A: Never! ipipgo's background automatically eliminates invalid IPs, which is much more reliable than manual maintenance.

Q: How many threads are appropriate to open at the same time?
A: It is recommended that novices control within 5 threads, the old driver up to 15 threads, remember to use ipipgo'sIntelligent Rate Adjustmentfunctionality

Don't step on these potholes.

I've seen some people buy shared IPs for cheap, and as a result, dozens of people use the same IP to capture data at the same time, and their accounts are directly blocked forever. There is also Iron Bean with a proxy open for 8 hours, LinkedIn is not a fool. Suggest using ipipgoAuto Sleep ModeThe catch is 1 hour and 15 minutes off, just like the real thing.

Lastly, to be honest, if you want to get LinkedIn data steadily, you have to spend money on the proxy. ipipgo is recently doing activities, new users to send 5G traffic, enough to test for half a month. Remember to use theirDynamic Residential AgentsDon't pick it as a static corporate IP, that shit is good for something else.