The Python selenium chrome crawler seems to have been recognized. How else can it get around besides calling js?

here"s the thing. I was doing exercises on the food and drug safety website and found that there was such a situation on this website. As long as you use selenium to drive chrome to do any browser operation, you can"t get data

but after opening the website through selenium, there is no problem if you click manually.
after I also tried, I first filled in input data through selenium, and then manually clicked, but still prompted "the server did not return data".

from selenium import webdriver
import time

browser = webdriver.Chrome()
url = "http://app1.sfda.gov.cn/datasearch/face3/base.jsp?tableId=89&tableName=TABLE89&title=%CA%B3%C6%B7%CC%ED%BC%D3%BC%C1%C9%FA%B2%FA%D0%ED%BF%C9%BB%F1%D6%A4%C6%F3%D2%B5&bcId=137403916083811026153735196207"  
browser.get(url)
browser.find_element_by_id("goInt").send_keys("4")
-sharp
time.sleep(5)
browser.find_element_by_xpath("//*[@id="content"]/div/table[4]/tbody/tr/td[7]/input").click()

I hope you can click.

Javascript python

Apr.01,2021

Visualization is an anti-selenium method based on js, and you can't lose any browser.find_element.
for example: you call browser.find_element_by_id ("goInt"), and all subsequent requests return a value of 400.
the following is my personal understanding:
the find_element of selenium sends a request to the local server (chromediver, parses html and js) to get the data. I have seen the source code of selenium before and opened a service locally, which can be accessed directly through the URL.

provides another solution:

from PyQt5.QtWidgets import QApplication
from PyQt5.QtWebEngineWidgets import QWebEngineView
from PyQt5.QtCore import QUrl
import time

app = QApplication([])
view = QWebEngineView()
view.load(QUrl("http://app1.sfda.gov.cn/datasearch/face3/base.jsp?tableId=89&tableName=TABLE89&title=%CA%B3%C6%B7%CC%ED%BC%D3%BC%C1%C9%FA%B2%FA%D0%ED%BF%C9%BB%F1%D6%A4%C6%F3%D2%B5&bcId=137403916083811026153735196207"))
view.show()
page = view.page()

def test():
    page.runJavaScript("document.getElementById('goInt').value = 5")
    page.runJavaScript("document.getElementById('goInt').parentNode.parentNode.childNodes[7].childNodes[0].click()")
    -sharptodo 

view.loadFinished.connect(test)
app.exec_()

Supplementary content:
Asynchronous Writing method:

from PyQt5.QtWidgets import QApplication
from PyQt5.QtWebEngineWidgets import QWebEngineView
from PyQt5.QtCore import QUrl, QThread, pyqtSignal
import time

cnt = 2
app = QApplication([])
view = QWebEngineView()
view.load(QUrl("http://app1.sfda.gov.cn/datasearch/face3/base.jsp?tableId=89&tableName=TABLE89&title=%CA%B3%C6%B7%CC%ED%BC%D3%BC%C1%C9%FA%B2%FA%D0%ED%BF%C9%BB%F1%D6%A4%C6%F3%D2%B5&bcId=137403916083811026153735196207"))
view.show()

class WorkThread(QThread):
    -sharp 
    trigger = pyqtSignal(int)
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def run(self):
        time.sleep(2)
        self.trigger.emit(cnt)


def output(_):
    global cnt
    view.page().runJavaScript('_x("//*[@id=\\"content\\"]/table[2]/tbody/tr[1]/td/p/a")[0].text', lambda r: print(r))
    if cnt < 6:
        cnt += 2
        work.trigger.disconnect()
        work.trigger.connect(next)
        work.start()

def next(id):
    view.page().runJavaScript('_x("//*[@id=\\"goInt\\"]")[0].value = %d' % id)
    time.sleep(0.5)
    view.page().runJavaScript('_x("//*[@src=\\"images/dataanniu_11.gif\\"]")[0].click()')
    work.trigger.disconnect()
    work.trigger.connect(output)
    work.start()

work = WorkThread()
work.trigger.connect(next)

def main():
    view.page().runJavaScript("""
        function _x(STR_XPATH) {
            var xresult = document.evaluate(STR_XPATH, document, null, XPathResult.ANY_TYPE, null);
            var xnodes = [];
            var xres = xresult.iterateNext();
            while (xres) {
                xnodes.push(xres);
                xres = xresult.iterateNext();
            }
            return xnodes;
        }""")
    view.page().runJavaScript('_x("//*[@id=\\"content\\"]/div/table[2]/tbody/tr[1]/td/p/a")[0].text', lambda r: print(r))
    work.start();

view.loadFinished.connect(main)
app.exec_()

result:

have you solved the problem? I'm working on this website, too.

Previous: Help! Vue project packaged in Wechat browser and mobile browser white screen?

Next: Want to know what these two pieces of code mean? I don't understand

It is invalid to change the data-src of the video to src by crawling Wechat's official account article.
the article on Wechat s official account crawled through python at the backend found that the video in the article could not be played, and the video was nested in iframe, as shown in <iframe class="video_iframe" data-vidtype="2"...

Front-end Wechat-public-platform javascript python

Feb.26,2021
Form submission combined with ajax?
formajax sendcodephone_numNoneurl ...

Javascript python html django

Feb.28,2021
Django api returns 403 {"detail": "CSRF Failed: CSRF cookie not set."}
the API, I wrote by calling django through PUT is good on postman, but when I call API through axios, I return 403. response is { "detail ": "CSRF Failed: CSRF cookie not set. "}, in django, I have commented CSRF but it s still useless, ask for...

Javascript python

Mar.01,2021
How to implement top n problem with JavaScript? For example, most_common () of python
We know that the collections of the python built-in module has a lot of useful operations. for example: from collections import Counter -sharp -sharp top n user_counter = Counter("abbafafpskaag") print(user_counter.most_common(3)) -sharp[( ...

Javascript python python3.x algorithm

Mar.03,2021
Capitalization of the first letter of a string in the python list
The problem originates from the chapter in map reduce in Liao Xuefeng s python tutorial. Use functions and map, to capitalize the first letter of the string in the list , and then return the list . Link the initial code is, for in iteration, and...

Javascript python3.x

Mar.03,2021
Whether the data transmitted by socket needs to be encrypted
whether the data transmitted by socket needs to be encrypted ...

Java cPP javascript python golang

Mar.04,2021
On the problem of obtaining data by ajax
first of all, when the user logs in, I want to display the user s information and get the data in the profile_ajax view. This corresponds to html html formuid . Here is the problem. I click here to view the details. Will the ajax of html not enter th...

Html javascript python django typescript

Mar.06,2021
How does django receive json data from post?
how to use js post to django data in django to receive JS similar to this str = {"student":[{"number":"0","name":"a"}]} ; obj = JSON.parse(str); look at the data in chrome like this [student][0][...

Javascript python django

Mar.13,2021
Flask cannot get ajax send post data
flask cannot get ajax send post data var data1= { name : kikay , } var data = JSON.stringify (data1) ajax ( POST , http: localhost:5000 login ,null, data1, function (r) { console.log (data,r.status,r.response) }) send ajax @ app.route...

Javascript python

Mar.13,2021
There is a small problem encountered in data analysis and data extraction. The newcomer is looking for an answer.
Today, I was working on a machine learning project and encountered a small problem. I want to extract a column of data using the regular expression and then run it and find that there is no name1 data. This symbol is very puzzling for the Great God s ...

Github scala ruby javascript python

Mar.19,2021
Ionic project npm install reported an error gyp err! Configure error
how to solve it, gods! ...

CPP typescript javascript python node.js

Mar.22,2021
Can't React add click events to child components?
problem description render () { return ( <div onClick={()=>{alert(1)}}> <div onClick={()=>{alert(2)}}>Hello< div> <NavLinkBar onClick={()=>{alert(3)}} data={navList}> < NavLinkBar> < di...

Node.js javascript python react.js html5

Mar.25,2021
Does the browser use jQuery Ajax to get the server Datagram Provisional headers are shown error?
problem description try to do a hands-on project after learning the Flask framework. The function is to collect and use the hash decryption API exposed on the network to obtain plaintext when the server data is obtained through jQuery.get (), it is f...

Ajax jquery flask javascript python

Mar.28,2021
Python+selenium found that the password entry box display:none, of the login window caused an error when writing the script.
< H1 > Writing application automation scripts selenium.common.exceptions.ElementNotInteractableException: Message: Element < input id= "showPwd " class= "userinput " name= "Password1 " type= "text " > is not reachable by keyboard, online search, find...

Javascript python

Mar.30,2021
The incomprehension of the scope of python and js variables
I have just learned python and encountered some problems about the scope of variables, as follows: the following is not very clear var a = 3 const f = () => { a += 4 console.log(a) } a() 7 f() 7 in js , you can not only access gl...

Javascript python scope variable

Mar.31,2021
If mac shadowsock can't be connected to win, it can be used on me. Mac suddenly can't be used today?
attach a log 14 Console 55 0x002A0027 0x0006354A 0x11111100 0x00000000 15.582676 + 0800 default [0x7fc4989de600] Created session 14 Created session 55 Console 15.582917 + 0800 Console default [0x7fc4989de600] Decoding: C0 0x002A0027 0x0006354A 0x1111...

Golang ruby javascript python

Apr.07,2021
Logic topic, given 26! The letters are arranged (regularly), and the results of the arrangement are obtained for any number.
if there is an abc, there are 3! The order of is abc acb bac bca cab cba to any number, such as Numeric 4, output bca. now there are 26 English letters, a total of 26! Given any number N, how to find the alphabetic column? listen to others say tha...

Javascript python java cPP

Apr.11,2021
Parameter transfer problem of python decorator
problem description I wrote a decorator with parameters, and now I need the decorator to use different parameters according to different conditions. the environmental background of the problems and what methods you have tried I wrote a decorator w...

Php java ruby javascript python

Apr.11,2021
How to use runJavaScript to simulate an element of a mouse movement page in Python
question: I saw an answer before because the selenium framework was detected in the previous paragraph, and then I used PyQt5 to make a browser to implement the crawler. I now have the same problem, but the problem is that I don t know how to use Js s...

Javascript python

May.11,2021
On the realization of automatic Operation by Pyqt5
with the help of Li Yi s PyQt5 embedded browser injecting Javascript scripts to automate operations , I have now completed the automation of low-cost airlines, but now I have encountered several problems: after I complete the order, such as entering...

Javascript python qt5

May.24,2021