would like to ask, write a crawler, how to tell when the crawler should stop?
the initial state is a url; and then there is a
while(isNotEmpty(urlList)){
// do something
}my idea is this, but the speed of queuing url can not keep up with the speed of consumption, so that the urlList is empty, and then the crawler stops. I would like to ask which Daniel has written the framework of the crawler and what are the conditions under which the crawler stops running.
the idea is a little strange, urlList links are also put in, put a climb on the line. The crawler stops when you don't put a link to urlList.
depending on the specific circumstances to be crawled:
1:
2:Kafkatopic how the crawler stops depends on your own business.
as long as the reptile is done to remove the weight.
if the crawler is controllable, replace multithreading with a single process. By killing the process. Control the crawler.
screen deploy the crawler project.
as in the title, write a simple function test to generate a soup object from the URL using Python requests and BeautifulSoup, (see the example below). If you call this function directly in the main thread, everything will be fine, but if you call this f...
segmentfault dvaws.open().websocketwebsocketws ws.openfetchGETloginsendfetchCOOKIECOOKIEFETCHCOOKIECOOKIE ...
Why did spring fail to automatically match fields? Wechat s front end reads as follows: Java: ...
there are some conflicts between eslint rules and webstrom shortcut key formatting for example: <Route path=" list" component={TopicList} >, Under the airbnb specification, there must be a space in front of the of the self-closin...
study online tutorials learn to package projects using webpack, a simple code implementation. the entry file index.html of the interface is as follows: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&qu...
< H1 > github address < H1 > error will be reported when using npm run dev const path = require( path ); const merge = require( webpack-merge ); const common = require( . webpack.common.js ); dev-server let BundleAnalyzerPlugin = require( ...
want to use vue2.0 to develop a small web application for the company, running on the mobile side. How can the above functions be achieved? ...
the front and back end communicates with websocket. How does the front end convert long data into ArrayBuffer 64-bit binary data ...
the npm start command runs normally, but the error will be reported when executing with npm run build as follows: syntax error: Unexpected token (Unexpected token (26:6) my index.js: import React, { Component } from react ; import ReactDOM from...
how to set the global URL here, you can use both in the development environment and online. ...
for testing and production deployment, just throw the front-end page to nginx but at the time of development, how do you deploy the front-end page? ...
$(document).ready(function(){ $(".pageloading").show(); $(window).on("load", function() { $(".pageloading").fadeOut(); }); }); google chrome can, will $( ".pageloading "). FadeOut (); but the safari networ...
...
Today, the application receives the message in the production environment and reports an error in the parser, and then the weblogic console cannot log in, but the service of the background application is still there and can still receive the message. Th...
after vue-router lazy loading configuration, the page becomes blank this problem does not occur when lazy loading is not configured, when configuring lazy loading *const product = resolve => { require.ensure([ . pages product.vue ], () =>...
as shown in the figure above, use css to make a magnification effect when a mouse passes over a square, but as soon as you enlarge a square, the square next to it will be squeezed out or squeezed into the next row. Is there any way to fix the enlarge...
require the fs module in the front-end project, but keep reporting errors? fs.readFileSync is not a function introduce code var fs = require( fs ); fs.readFileSync( index.html , utf8 ); this is the webpack configuration file const path = re...
there is a bypass Filter in proxyTable. How do I use it? Is it the return path? Or return true false ...
webpack project upgrade version configuration is as follows: webpack.json "scripts": { "dev": "webpack --mode development", "build": "webpack --mode production", "watch": &quo...
according to the demo in ...