On the problem of monitoring page change and regular crawling increment

there is a project. I want to crawl the page at 19:00 every day, every 30 minutes, until I get to the incremental content, and then cycle again at 19:00 tomorrow. The configuration is as follows 

@every(minutes=30)
def on_start(self):
    ...


@config(age=24 * 60 * 60)
def index_page(self, response):
    ...
  1. this setting, every= every 30 minutes, age= every 24 hours, can play the effect of timing start?
    if you want to start a timing function that starts at 19:00 every day, is there a more appropriate way to start a run at 19: 00 for the first time?
  2. in addition, the URL of the web page of the project will change if the content is the same. In addition to manually comparing the local database, is there a more appropriate way to monitor and only crawl increments?
Pyspider
Jun.13,2021

the first problem is solved by itself:
calls the time and date interface of Python and uses if to judge.
second question, since URL can change, maybe I'm giving you the only way to do it right now.

Previous: The spring cloud eureka client project cannot be started, and the configuration file yml is not valid.

Next: Vue, v-if binds to an object in data, changes the properties of the object, and does not update dom

css mysql arrays josn react html typescript webpack npm sass R objective-c .net sql-server jquery python-3.x angularjs django angular excel regex iphone ajax linux xml pandas vba spring database wordpress string wpf xcode windows bash postgresql oracle multithreading eclipse list firebase algorithm macos forms image scala visual-studio azure bootstrap spring-boot react-native python-2.7 docker performance function winforms matlab powershell apache dataframe api sqlite numpy rest shell selenium flutter dart maven loops qt swing android-studio csv express file class tensorflow sorting codeigniter perl
© 2021 CodesHelper
Advertising Contact us About us
Menu