About the anti-crawler problem of a website

the website I encounter now seems to use distil networks, an anti-crawler service. If you need to get the data, you must bring cookie, without cookie. All requests will be returned directly

.
<!DOCTYPE html>
<html>

<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
<meta http-equiv="cache-control" content="max-age=0" />
<meta http-equiv="cache-control" content="no-cache" />
<meta http-equiv="expires" content="0" />
<meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />
<meta http-equiv="pragma" content="no-cache" />
<meta http-equiv="refresh" content="10; url=/distil_r_captcha.html?requestId=953a747e-5387-423c-b87d-1d44f55e846b&httpReferrer=%2Fbook%2FFlight%2FSelect%3Fculture%3Dzh-CN%26type%3D1%26dst1%3DCAN%26ast1%3DSIN%26dd%3D2018-10-05%26adt%3D1%26chd%3D0%26inf%3D0" />
<script type="text/javascript">
    (function(window){
        try {
            if (typeof sessionStorage !== "undefined"){
                sessionStorage.setItem("distil_referrer", document.referrer);
            }
        } catch (e){}
    })(window);
</script>
<script type="text/javascript" src="/tgrairwaysdstl.js" defer></script><style type="text/css">-sharpd__fFH{position:absolute;top:-5000px;left:-5000px}-sharpd__fF{font-family:serif;font-size:200px;visibility:hidden}-sharpcfcxdbattebszfeeswyw{display:none!important}</style></head>
<body>
<div id="distilIdentificationBlock"></div>
</body>
</html>

and the failure time of this cookie is very fast, only about 5 minutes, and if you use it too many times, it will also fail for about four or five times, and I use fiddler tracking. After getting all the cookie parameters, I still can"t get the data using this cookie. I guess the reason is that the P parameter that I use many times, although the return value is different each time, is not available. The p parameter should only be used once. I hope this question can be answered

Pay attention to

. I have heard of distil, which seems to be very powerful. I suggest you try other crawler methods of Chrome Devtools Protocol,. I feel that there are some obvious features


I also encountered distil networks, when I was climbing a website. I successfully loaded it into the web page with a browser frame, but when I got the elements of the page, document.getElementById returned all empty. Need to find an organization to discuss how to break this thing


this problem, I encountered, however, mainly 301automatic page jump problem, automatic, no, will be detected, manual page jump bar. I solved the problem by jumping manually

Menu