How to reset or empty the data of scrapy-redis 's dupefilter?

after you have climbed with scrapy-redis spiders, you will not be able to climb again. If you change the name of the spider, you can crawl again, and if you change back to the original name, you will start the deduplication mechanism again. Although you need dupefilter, how to solve this problem if you generate some data during debugging and want to crawl again.

how to delete crawled url data previously recorded by dupefilter.

Feb.26,2021

solved
it's hard to wait several days for no one to reply


I also encountered the same problem, but what you said above gave me an idea. I changed the name of the spider and I can use it. Then it is found that the key, with the name of the crawler is saved in redis and the key can be deleted.


SCHEDULER_FLUSH_ON_START = True
you can add this sentence in the settings to automatically clean up the key in redis

MySQL Query : SELECT * FROM `codeshelper`.`v9_news` WHERE status=99 AND catid='6' ORDER BY rand() LIMIT 5
MySQL Error : Disk full (/tmp/#sql-temptable-64f5-7ada81-1e16a.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
MySQL Errno : 1021
Message : Disk full (/tmp/#sql-temptable-64f5-7ada81-1e16a.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
Need Help?