Why can pyspider implement url to crawl multiple pieces of data?

it is said that on_message can, but I still can"t test it. Is there any way to achieve it?

def detail_page(self, response):
    results = json.loads(response.text)
    for result in results:
        date = result["date"]
        number = response.url[6:]
        self.send_message(self.project_name, {
            "number": number,
            "date":date,
         }, url="%s-sharp%s" % (response.url, date))
 
def on_message(self, project, msg):
    print msg
    return msg

Pyspider python python-crawler

Mar.21,2022

detail_page can return a list, and then when on_result processes, when the list is processed.
there is nothing wrong with your code. This is not the problem. Also, the message sent by the send_message that does not call the on_message, test during the test is sent to the message on the left. On_message. is called only when the project is running

I didn't pay attention to the weight removal of url when I first came into contact, resulting in no result. If you want to grab the same url, you need to modify the url, such as adding the anchor point (- sharptag),) so that it will be crawled again. I hope those who learn later will pay attention to this problem.
yours could be the same reason

The

code does not see the problem. It is estimated that you can not get the result by manual debugging in webUI, and you can see it when run gets up.

Previous: Under what circumstances is cPP a [n] a right?

Next: What messages are stored in pyspider's redis message queue?