Exception occurred during distributed crawler: redis.exceptions.ResponseError: DISCARD without MULTI

  1. this exception occurs during a distributed crawler using scrapy-redis , not from the beginning, but from the crawler. Five machines are used to crawl at the same time.
  2. exception information
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/commands/crawl.py", line 58, in run
    self.crawler_process.start()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/crawler.py", line 291, in start
    reactor.run(installSignalHandlers=False)  -sharp blocking call
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1267, in run
    self.mainLoop()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 1276, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/twisted/internet/base.py", line 902, in runUntilCurrent
    call.func(*call.args, **call.kw)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/utils/reactor.py", line 41, in __call__
    return self._func(*self._a, **self._kw)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 122, in _next_request
    if not self._next_request_from_scheduler(spider):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy/core/engine.py", line 149, in _next_request_from_scheduler
    request = slot.scheduler.next_request()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy_redis_bloomfilter/scheduler.py", line 175, in next_request
    request = self.queue.pop(block_pop_timeout)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/scrapy_redis/queue.py", line 115, in pop
    results, count = pipe.execute()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3443, in execute
    return execute(conn, stack, raise_on_error)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3339, in _execute_transaction
    self.immediate_execute_command("DISCARD")
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3275, in immediate_execute_command
    return self.parse_response(conn, command_name, **options)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 3402, in parse_response
    self, connection, command_name, **options)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/client.py", line 768, in parse_response
    response = connection.read_response()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/redis/connection.py", line 638, in read_response
    raise response
redis.exceptions.ResponseError: DISCARD without MULTI
May.12,2022
Menu