What does the number of threads open when the crawler does the ip agent have anything to do with ip?

for example, what is the relationship between my 1000 threads and the concurrency of proxy ip per second?

Mar.31,2021

is approximately equal to the relationship. Each thread requests the agent ip to visit the target web page. If the request happens to be made at the same time, the concurrency of the agent IP is 1000 per second.


each thread needs to occupy one IP, but it will be reused.
for example, if the proxy IP has 500 and requests 1000 times per second, the same IP will be used every 2 times in these 1000 requests (reasonable), but in fact, this is not the case. Some IP may be used three times, some only once. It depends on what your IP strategy is.

Menu