How to run multiple crawlers in scrapy - Codes Helper - Programming Question Answer

How to run multiple crawlers in scrapy

scrapy crawl kr.py
2018-07-09 00:03:06 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: multispider)
2018-07-09 00:03:06 [scrapy.utils.log] INFO: Versions: lxml 3.2.1.0, libxml2 2.9.1, cssselect 1.0.1, parsel 1.4.0, w3lib 1.19.0, Twisted 16.2.0, Python 2.7.5 (default, Apr 11 2018, 07:36:10) - [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)], pyOpenSSL 16.2.0 (OpenSSL 1.0.1e-fips 11 Feb 2013), cryptography 1.7.1, Platform Linux-3.10.0-514.26.2.el7.x86_64-x86_64-with-centos-7.3.1611-Core
Traceback (most recent call last):
  File "/usr/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 150, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 90, in _run_print_help
    func(*a, **kw)
  File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 157, in _run_command
    cmd.run(args, opts)
  File "/usr/lib64/python2.7/site-packages/scrapy/commands/crawl.py", line 57, in run
    self.crawler_process.crawl(spname, **opts.spargs)
  File "/usr/lib64/python2.7/site-packages/scrapy/crawler.py", line 170, in crawl
    crawler = self.create_crawler(crawler_or_spidercls)
  File "/usr/lib64/python2.7/site-packages/scrapy/crawler.py", line 198, in create_crawler
    return self._create_crawler(crawler_or_spidercls)
  File "/usr/lib64/python2.7/site-packages/scrapy/crawler.py", line 202, in _create_crawler
    spidercls = self.spider_loader.load(spidercls)
  File "/usr/lib64/python2.7/site-packages/scrapy/spiderloader.py", line 71, in load
    raise KeyError("Spider not found: {}".format(spider_name))
KeyError: "Spider not found: kr.py"

scrapy running multiple crawlers reported an error
screenshot do not know if it is right to write this?

Mar.24,2021

Previous: Flattening of js multi-level nested objects

Next: How to use vsCode to edit a project that develops the go language?

Use python to convert dict1 to dict2, in the example. I tried for a long time but couldn't find a solution.
dict1 = { "system.cpu.user.pct": { "value": 12.83 }, "system.load.1": { "value": 0.33 } } dict2 = { "system": { "...

Python

Feb.26,2021
On the problem of regular matching
description: a regular match is given to the content of an input box, and the matching content is the product activation code. looks like this: "0C31-0B81-BB32-3094-0C31-0B81-BB32-3094 " Code: $( -sharplicenseCode ).keyup(function () { le...

Python html5 html php javascript

Feb.26,2021
Python two-dimensional list, each sublist (with different number of elements) takes one element to combine, listing all possible situations
for example, the known two-dimensional list [[aformab], [dpene], [f]] requires an element from each sublist to be added and lists all the combinations. The output of this question is adf,aef,bdf,bef,cdf,cef. There are many such lists, and the number of s...

Python

Feb.26,2021
What should I do if I want to randomly pick out five different colors and numbers of playing cards?
I want the program to randomly issue five cards with different colors and numbers (it doesn t matter if there are occasional cards with the same number), but I find that five cards are always the same. import random suites = [ "Hearts ", "Diamonds ...

Random-number novice-learning novice-problem python

Feb.26,2021
How to solve the problem of (Python) SyntaxError: Non-ASCII character once and for all
this problem works fine after I add code to the first line, but do I have to pay attention to this every time? Python novice asks ...

Python

Feb.26,2021
Forbidden (403) CSRF verification failed. Request aborted.
{% csrf_token%}, is added to the form form of html in the template. MIDDLEWARE = [ django.middleware.security.SecurityMiddleware , django.contrib.sessions.middleware.SessionMiddleware , django.middleware.common.CommonMiddleware , ...

Python

Feb.26,2021
How to use python to bulk distribute native ssh public keys?
when we use the ansible tool, the default is to use ssh for remote control. you need to generate key pairs in a way similar to ssh-keygen-t rsa , and then use the ssh-copy-id command to distribute the public key file to a remote host and enter the pas...

Operation-and-maintenance-automation python ansible

Feb.26,2021
Python has been unsuccessful in replacing the content in front of the vertical bar symbol in the title.
python replaces the content in front of the vertical bar symbol in the title, but it has been unsuccessful, for example: Chen Ning column (Chinese football fight to correct the name of Wales Uruguay for the championship, replace Chen Ning column for oth...

Python

Feb.26,2021
It is invalid to change the data-src of the video to src by crawling Wechat's official account article.
the article on Wechat s official account crawled through python at the backend found that the video in the article could not be played, and the video was nested in iframe, as shown in <iframe class="video_iframe" data-vidtype="2"...

Front-end Wechat-public-platform javascript python

Feb.26,2021
Python3 creates a new BeautifulSoup object in a child thread for a specific web page, but there is no exception in the encoding error, main thread.
as in the title, write a simple function test to generate a soup object from the URL using Python requests and BeautifulSoup, (see the example below). If you call this function directly in the main thread, everything will be fine, but if you call this f...

Web-crawler multithreaded beautifulsoup python

Feb.26,2021
Session problem of Django
blog s article like function, like once + 1, use session to record the current user problem point is: if you like article 1, article 2 will indicate that it has been supported. The reason is to judge that session how to realize that articles cannot b...

Django python javascript

Feb.26,2021
How pandas groups and merges by alias alias
1. Condition: according to whether the alias in the two lines intersect, and if so, Then merge separated by-sharp sign, where: alias 2 in alias separated by-sharp sign, data name alias 0 potato potato-sharp egg-sharp potato-sharp potato 1 potato p...

Python

Feb.26,2021
Centos7shell postgres login failed
Baidu said that there is a problem with the verification method in the file var lib pgsql 9.5 data pg_hba.conf, but the reason for changing the verification method is to use the command psql-U postgres-d mydjango-p 5432-h 127.0.0.1 to log in directl...

Nginx centos django python postgresql

Feb.26,2021
How to set callback for web direct pass of Tencent Cloud COS?
I ve been looking for it for a long time, but I can t find it. 1. After the direct upload of web is completed, the server cannot know whether the file was uploaded successfully, who uploaded it, and the information of the file 2. Using web to dire...

Java python node.js html5 javascript

Feb.26,2021
Pandas's dataframe condition Filter performance Optimization?
currently I have a piece of code that spends most of its time on the above two sentences of data filtering in dataframe. temp_df = df [df [ "data_date "] .isin (date_list)] temp = temp_df [rule [2]] [temp_df [ "data_date "] = = d] at present, it tak...

Python pandas

Feb.26,2021
In pycharm, how does the chart displayed by matplotlib.pyplot.show () pop up when output to sciview,?
< H2 > 1. Question: < H2 > like the title, how does the chart displayed by matplotlib.pyplot.show () in pycharm pop up to sciview,? < H2 > 2. Code: < H2 > import matplotlib.pyplot as plt import matplotlib.animation as animation from matplotlib impo...

Matplotlib pycharm python3.x

Feb.26,2021
There is a problem with pip when switching the version of python.
I need to install both python 2.6 and python 3.6 on my machine. Based on my query, I found that I could use pyenv to do this things, but according to an article I read, python 2 series didn t come with package management tools until python 2.7.9. The...

Linux package-Management python

Feb.26,2021
The thread pool of futuers in python3 is very efficient, but sometimes I wonder if it can be realized if it is a little slower, such as an interval of 0.5 between tasks.
The question is written on the title, is there any way to be spaced by BAN, when it is applied too fast in crawlers? Or is it that there is no interval in this kind of concurrency? ...

Thread-pool thread concurrent asynchronous-programming python3.x

Feb.26,2021
Selenium uses Chrome to fill in the form.
Open a chrome browser using selenium, and open a Baidu web page. Using the send_keys () method to fill in the content in the search box, there are the following problems -sharpcoding:utf-8 from selenium import webdriver from selenium.webdriver.chrom...

Python

Feb.26,2021
Python installation error, update machine required
the newly installed win7 system, Ann python is, the installation package setup comes out of this, which I have encountered before. I forgot how to solve it. ...

Python python3.x

Feb.27,2021

Menu