For a school project I am making around 10000000 requests (I know it's a lot but I need revenue data for all movies in the IMDB CSV dump). I have max 19 simultaneous open connections and have implemented a system that doesn't make more than 40 requests per second.
After about 8000 requests, I get a timeout from the server (an actual timeout, not a 503-error). I've tried this on 2 PC's and I have this issue on both pc's.
Is this some kind of rate-limit that I am reaching, or is there a different reason?
¿No encuentras una película o serie? Inicia sesión para crearla:
¿Quieres puntuar o añadir este elemento a una lista?
¿No eres miembro?
Contestado por Travis Bell
el 27 de febrero de 2024 a las 17:40
Hi @marnick,
There's no other block than the rate limiting. Probably best to just retry and continue on timeouts and or other errors you might come across.