Is there a library or framework in pyhton that can dynamically generate tasks, monitor their status, and remove tasks or continue to run in real time according to their status or properties?

look at the library such as celery aps, but there is too little Chinese information on the Internet.
talk about the demand:
1 for example, there are 100 sports games a day, and the URL that generates these 100 games is each thrown into the task queue for a task.
2 crawls data every 20 minutes before this time according to its status, such as the end time. When the time is up, delete the task and stop crawling.
3 the task will return data to be stored in the database
is there a solution like this? what kind of technology will be used to solve the problem?

Task-scheduling python-crawler

Feb.28,2021

you can use redis.

100th game, put score into zset, with match number value, time so that it will be sorted automatically by time.

then you fetch the first element of the zset each time, according to the value to get the specific match deadline. If the time is not exceeded, the crawler will be triggered, and then the time will be + 20 minutes. If the time exceeds, it will be deleted from the zset.

can also be replaced by an ordered collection without redis's zset,.

Previous: Consume kafka data and then check the database. How to deal with this scenario?

Next: How to implement showmount-e command with Python