in pipelines, the code is as follows:
import logging
from scrapy.utils.log import configure_logging
configure_logging(install_root_handler=False)
logging.basicConfig(
filename="log.txt",
format="%(levelname)s: %(message)s",
level=logging.INFO
)
logging.error("s") will output on the console SRAV 2018-11-12 19:19:53 [root] ERROR: s
will not be output to the set log.txt.
Why?
has encountered a similar problem, which I haven't solved completely, which may be due to the global setting problem of logging. Because logging is in singleton mode, and basicConfig will not work if root logging is already set up. In large projects, you should make global settings for logging before reading all modules, otherwise conflicts may occur.
if it has been set in other modules, it cannot be set again. I also encountered it today when I found that there was basicConfig in one of my modules, causing the subsequent settings to be invalid.
Previous: Does flutter have a good selector recommended by widget or third-party libraries?
Next: Wechat obtains user information through the official account token. The Times token expires.
system: Ubuntu 16.4 python3.6 twisted-15.2.1 Scrapy 1.5.0 is also installed in the virtual environment prompt the following message when creating a Scrapy: (pyvirSpider) root@ubuntu: myScrapy-sharp scrapy startproject test Traceback (most recent...
recently read Learning Scrapy, which mentions a crawler that automatically turns pages and crawls items on each page. The book says that Scrapy uses last-in, first-out queues. suppose there are 30 items on each page, and start_url is set to the first ...
after starting the framework to crawl the target web page start_url, you need to extract an eigenvalue from the string start_url as the collection name of the MongoDB database, and then store the item through pipeline. outline flow: spiderpipeline ...
when executing the scrapy shell xx URL, there is no response. The stdout in the log file returns , and does not respond to the URL address in quotation marks, and the interface does not respond. Solve used to run successfully, but suddenly failed t...
problem description there are 6000 url, to start the celery generation task at 12:00 and send the queue to two servers to crawl. I use middleware to get 10 proxy ip to carry up the request at a time. After 100, I proceed to process the next set of 100...
problem description there are 6000 url, to start the celery generation task at 12:00 and send the queue to two servers to crawl. I use middleware to get 10 proxy ip to carry up the request at a time. After 100, I proceed to process the next set of 100...
problem description there are 6000 url, to start the celery generation task at 12:00 and send the queue to two servers to crawl. I use middleware to get 10 proxy ip to carry up the request at a time. After 100, I proceed to process the next set of 100...
problem description there are 6000 url, to start the celery generation task at 12:00 and send the queue to two servers to crawl. I use middleware to get 10 proxy ip to carry up the request at a time. After 100, I proceed to process the next set of 100...
< H1 > attach the source code of the crawler file. < H1 > import scrapy from openhub.items import OpenhubItem from lxml import etree import json class ProjectSpider(scrapy.Spider): name = project -sharp allowed_domains = [] start_urls ...
files have been downloaded, the original files are all about 1m, but scrapy downloads are all 3k. As shown in the following picture. ...
I run the single file directly without import errors. In addition, it is normal for me to use mongodb in the py file alone, but when I run it in the scrapy project, I will say that the import failed. Why? import json import pymongo from scrapy.utils.pr...