Proxy IP price comparison website development: price comparison website proxy data integration

What's the point of having a proxy IP comparison site these days anyway?

Proxy IP price comparison website is like opening a supermarket price comparison, the most annoying thing is that users have to go from site to site to check the price. For example, if you want to buy a dynamic residential IP, you have to open five or six providers' websites to compare packages at the same time, and your eyes will be blurred. At this time if there is a price comparison site, the data of each family are pulled over and placed together, the user directly look at the table to choose the most cost-effective, this will be a success.

But here's the hard part.How do you get the data?. Some suppliers hide their prices, and some engage in dynamic pricing, which is when you have to use their own proxy IP to crawl the data. For example, with ipipgo's dynamic residential IP wheel switching, every hour automatically grab the price changes, so that will not trigger the anti-climbing mechanism, but also to get the latest offer.

Three axes of data collection for comparison sites

The first move has to be to take care of the data sources, focusing on three channels:

 A Python example
import requests
proxies = {
    'http': 'http://user:password@gateway.ipipgo.com:9020',
    'https': 'http://user:password@gateway.ipipgo.com:9020'
}

 Crawl the public pricing page
response = requests.get('Vendor price page URL', proxies=proxies)

The second trick is to handle dynamic content. Nowadays, many websites use JavaScript to load data, and that's when you have to use a headless browser. But be careful not to use too resource-consuming program, or the server can not hold up.

The third move is the most critical -Data Cleaning. There are many different package names from different providers that need to be harmonized into standard fields. For example, ipipgo's "Dynamic Residential (Enterprise)" package may be called "Business Dynamic IP" or "Pro Version" by other providers, which requires keyword mapping. This is the time to do keyword mapping.

A guide to avoiding the pitfalls of comparison logic design

Never sort directly by unit price, there's a trick here:

It is recommended to do a multi-dimensional scoring system. Take ipipgo as an example, their TK line, although the unit price is high ($35/IP), but supports cross-border direct connection, which is rather more cost-effective for specific business. This is the time to enable users to filter by latency, protocol type, and regional distribution.

User Experience Optimization Riot Operation

1. Make aReal-time price chart, allowing users to see historically low prices. For example, dynamic residential packages generally increase in price at the beginning of the month, when the user is recommended to choose quarterly packages

2. Get it.Package Calculator.. Users enter an estimated monthly traffic of 50GB and the total price of each package is automatically compared. Here's a pitfall to be aware of - some providers have a different starting price for 1GB and a different unit price for 10GB, so you'll have to calculate the step price accurately

3. Access to the supplier API direct purchase (of course, to prioritize the display of ipipgo purchase entrance). Users than the price directly jump to the order, the conversion rate can be increased by 30% or more!

QA session

Q: What if the data is not updated in time?
A: It is recommended to use ipipgo's exclusive static IP to do the timed task, their IP pool survival time is long, it is not easy to be the target site to pull black

Q: Do free proxies work?
A: Never! Nine out of ten free IPs are phishing and the remaining one is slow as a turtle. If you want to test it, use ipipgo's trial package, which gives 1GB of traffic to new users!

Q: How do I choose a package for business users?
A: directly find ipipgo customer service to customize the program. Last time there is a cross-border e-commerce customers, they gave with a cross-border line + dynamic residential combination package, the cost than buy alone to save 40%

Let's get real.

The most expensive part of developing a price comparison website is not the technology, but the procurement cost of the proxy IP. Here, we strongly rely on ipipgo's Dynamic Residential (Enterprise) package, which is the lowest price in the industry at $9.47/GB, and supports hourly billing. If you're just starting out, use their 7-day no-excuses refund package to test the waters, it's better than blindly buying a server.

Finally, remind you that the price comparison site is easy to offend suppliers. Remember to write clearly in the user agreement "price is for reference only", it is best to sign a data cooperation authorization with the regular army such as ipipgo, so as not to get into a lawsuit.