Meituan takeout data encountered problems.

I am downloading Meituan takeout data now, but there are three problems.
one is that the address of the restaurant always changes. As soon as I got the list of restaurant URLs at the first level, the second level showed that I did not have access to the web page.
the second question is that when a web page is crawled by a DS counting machine, it shows that the page does not exist, but I can display it when I copy it to the browser with the same URL. Does this mean that Meituan found out that I was under the data and did a reverse crawl? The third problem with
is that I set the URL to a subordinate clue, but should I run a subordinate clue automatically when I run a higher-level clue? Now it doesn"t run automatically, what"s wrong with me?

C-sharp java python

Feb.15,2022

1. The change of the URL is the setting of the target website, which cannot be solved by the crawler. The fact that you have no right to access the page should not be the change of the URL, but the same situation as the second problem.
2. I encountered anti-crawling.
3. Hierarchical collection is set, and the two-level rules should be run manually separately, not automatically.
4. Need to change the agent IP, look for the kind of high concealment or exclusive.

Previous: Why use webpack to start local dev-server?

Next: How do tasks that have been running be handled with a thread pool?

Is the currency decimal type often used in actual development?
I usually use int for storage. The unit of data storage is divided into points. For example, for goods worth one yuan, it is 100 in the database. I think it s simple and straightforward, and there s no need to think about compatibility or anything. ...

C-sharp java python javascript php

Mar.03,2021
Meituan takeout data encountered problems.
I am downloading Meituan takeout data now, but there are three problems. one is that the address of the restaurant always changes. As soon as I got the list of restaurant URLs at the first level, the second level showed that I did not have access to the...

C-sharp java python

Feb.15,2022
If the web page information is grabbed too fast, which leads to the blocking of IP, how to deal with the web crawler?
collect a website too frequently, fewer and fewer web pages can be collected, and even be blocked. Only by controlling the collection speed and frequency can IP, continuously obtain data, and the browser s cookies should be cleaned regularly. ...

C cPP c-sharp java python

Mar.04,2022
If the web page information is grabbed too fast, which leads to the blocking of IP, how to deal with the web crawler?
collect a website too frequently, fewer and fewer web pages can be collected, and even be blocked. Only by controlling the collection speed and frequency can IP, continuously obtain data, and the browser s cookies should be cleaned regularly. ...

C cPP c-sharp java python

Mar.04,2022
If the web page information is grabbed too fast, which leads to the blocking of IP, how to deal with the web crawler?
collect a website too frequently, fewer and fewer web pages can be collected, and even be blocked. Only by controlling the collection speed and frequency can IP, continuously obtain data, and the browser s cookies should be cleaned regularly. ...

C cPP c-sharp java python

Mar.04,2022
How do I get the VIEWSTATE of POST?
encountered a very contradictory problem, I have a website need to crawl it, using POST mode, the form contains VIEWSTATE, now the problem is that the browser query can see VIEWSTATE, but how do I know in advance with the code? ...

Search-engine c-sharp java python javascript

Jun.17,2022