I try to use HtmlUnit to automate login process on my local TP-Link Router web-page. This is how the login page with the form on it is rendered in my Firefox browser:
And here is the code snippet to interact with the form:
package org.example;
import org.htmlunit.BrowserVersion;
import org.htmlunit.WebClient;
import org.htmlunit.html.HtmlInput;
import org.htmlunit.html.HtmlPage;
import java.io.IOException;
public class LoginFormScraper {
public static void main(String[] args) throws IOException {
try (WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getOptions().setCssEnabled(false);
HtmlPage page = webClient.getPage("http://tplinkwifi.net/webpages/index.html#/login");
webClient.waitForBackgroundJavaScript(30 * 1000);
HtmlInput passwordInput = page.getFirstByXPath("//input[@type='password']");
if (passwordInput != null) {
passwordInput.setValue("foo");
// continue form scraping
} else {
System.out.println("No input element found!");
System.out.println("///////////////////////");
System.out.println(page.asXml());
}
}
}
}
However, the output produced by that code is always like this, indicating that the page content is never given a chance to be fully loaded in the headless browser window:
No input element found!
///////////////////////
<?xml version="1.0" encoding="UTF-8"?>
<html lang="en">
<head>
<meta charset="UTF-8"/>
<meta name="version" content="AX55v1_1.11.0_2024-06-28T02:38:47.196Z"/>
<meta name="ui-type" content="svr"/>
<meta name="viewport" content="width=device-width,initial-scale=1,minimum-scale=1,maximum-scale=1,user-scalable=0"/>
<meta name="apple-touch-fullscreen" content="yes"/>
<meta name="apple-mobile-web-app-capable" content="yes"/>
<meta name="apple-mobile-web-app-status-bar-style" content="black"/>
<meta name="format-detection" content="telephone=no"/>
<link rel="shortcut icon" href="./assets/ico/favicon-ccbe82f2.ico"/>
<title>
Opening...
</title>
<script type="module" crossorigin="" src="./js/index-6a96bdab.js">
</script>
<link rel="modulepreload" crossorigin="" href="./js/index-ba20424f.js"/>
<link rel="modulepreload" crossorigin="" href="./js/vendor-d41cf34c.js"/>
<link rel="modulepreload" crossorigin="" href="./js/su-a307a2f8.js"/>
<link rel="modulepreload" crossorigin="" href="./js/update-store-e417b711.js"/>
<link rel="stylesheet" href="./assets/css/index-3aab50d1.css"/>
</head>
<body>
<div id="app">
</div>
<script type="text/javascript">
//<![CDATA[
(function handleUnsupportedBrowser() {
try {
// https://cn.vuejs.org/about/faq.html#what-browsers-does-vue-support
const isIE
= window.ActiveXObject
|| 'ActiveXObject' in window
|| /MSIE|Trident/.test(window.navigator.userAgent)
const isUnsupportedBrowser = !localStorage || !localStorage.setItem || isIE
if (isUnsupportedBrowser) {
throw new Error('unsupported browser')
}
} catch (error) {
location.href = './error.html'
}
})()
//]]>
</script>
</body>
</html>
Is there any way I can access HTML elements in a situation like this with HtmlUnit or should I use a different tool?