What if the file name is too long when crawling?

Traceback (most recent call last):
  File "run.py", line 84, in <module>
    run().get_spider()
  File "run.py", line 73, in get_spider
    qimingpianReportDowload().start_request()
  File "/home/shenjianlin/my_project/spiders/qimingpianReportDowload/qimingpianReportDowload.py", line 31, in start_request
    with open(save_path, "wb") as code:
OSError: [Errno 36] File name too long: "/home/shenjianlin/pdf_file/:- :-181202"
Jan.18,2022

redefine the file name


produces a unique id as the physical file name (such as UUID or MD5), and then uses a catalog table to maintain the relationship between the ID and the logical file name.

Table of contents can be used with files, database tables, redis, etc., depending on your preferences.

Menu