HTTPX sending POST requests: HTTPX library usage

Proxy play for HTTPX sending POST requests

Engaged in crawling old iron know, now many sites are added to the anti-climbing mechanism. This time you have to use a proxy IP torotational identityto avoid being blackmailed by the target website. Let's use the HTTPX library + ipipgo proxy today, hand in hand to teach you how to play the POST request.

First of all, let's understand what HTTPX is.

HTTPX than requests library is more fierce , support for HTTP / 2 and asynchronous requests . For example, when you want to register accounts in bulk or submit forms automatically, you can use this library. First install the library:

pip install httpx

Basic POST request writing

The regular operation without proxies looks like this:

import httpx

resp = httpx.post(
    url="https://example.com/login",
    json={"username": "test", "password": "123456"}
)
print(resp.text)

But this is easy to get blocked IP, this time we have to ask ouripipgo proxy serviceUp.

Put a "vest" on the request.

Proxy IP is like a new vest for the request, here we recommend using ipipgo'sTunnel AgentThe configuration method is simple for thieves:

proxies = {
    "http://": "http://用户名:密码@TunnelAddress:Port",
    "https://": "http://用户名:密码@tunneladdress:port"
}

with httpx.Client(proxies=proxies) as client.
    resp = client.post("https://example.com/api", json={"data": "test"})

Be careful to change the username and password to the authentication information you got in the ipipgo backend, so that the IP will be changed automatically for each request.

How to choose the type of agent

There are several types of agents on the market, so let's compare them in a table:

ipipgo's high stash of proxies has been tested to be the most covert, and is especially suitable for projects that need to run stably over a long period of time.

Common pitfalls QA

Q: What can I do if the request keeps timing out?
A: Try adjusting the timeout parameter and checking the availability of the proxy IP. ipipgo's console has a real-time monitoring function that can quickly locate the problem node.

Q: What if I need to switch IP frequently?
A: It is recommended to use ipipgo's dynamic rotation service, their API supports automatic switching by the number of requests, and you don't have to maintain your own IP pool at all.

Q: Does the agent need account password authentication?
A: ipipgo's proxy services all use whitelisting + double authentication, which is more secure than simply using IP verification to prevent theft by others.

Advanced Tips: Asynchronous Bombing

When it's time to send wild POST requests, use asynchronous mode to directly double the efficiency:

import httpx
import asyncio

async def send_request():: async with httpx.AsyncClient(proxies=proxies) as client.
    async with httpx.AsyncClient(proxies=proxies) as client:
        tasks = [client.post(url, json=data) for _ in range(100)]
        return await asyncio.gather(tasks)

asyncio.run(send_request())

Together with ipipgo's multi-million IP pool, this writeup can be easily realizedThousands of times per second.of security requests.

One final rant, don't just look at price when choosing a proxy service. A service like ipipgo can provide7×24 hours technical supportThe company's website is a great source of information for developers, and it's a great way to find people to solve problems at any time. They also recently came out of the pay-per-volume package, especially friendly to small developers, interested in the old iron can go to the official website to take a look.