So, I am building an express.js application that uses puppeteer and pupeteer-cluster for accessing one ukrainian test platform. The thing that I want to do is to solve test using this tools.
So, the flow is like this:
- Enter user cabinet
- Search for the test
- Enter the test
- Solve it.
The problem appears when I already have entered the test, answered a few questions and, on some point I am getting an error like this: Attempted to use detached Frame 'D78184744F4CA70F50307B66DDF53250'. My error says that there is a problem getting an element named with a class “.v-test-questions-title” when it’s for sure must be there.
async solveSingleAnswerQuestion() {
const pageElement = new PageElement(this.page)
const question = await pageElement.getContent('.v-test-questions-title') -> error says the problem is here
const variants = await pageElement.getContents(
'.v-test-questions-radio-block',
'.v-test-questions-checkbox-block'
)
console.log({ question, variants })
const answerIndex = await generateResponseIndex(
question || '',
variants
)
console.log({ answerIndex })
const variantsElements = await pageElement.getElements(
'.v-test-questions-radio-block',
'.v-test-questions-checkbox-block'
)
await variantsElements[answerIndex - 1].click()
const submitButton = await pageElement.getElement('.v-blue-button-test')
await submitButton.click()
await waitFor(5000)
}
Below is the code for PageElement (class that makes all the code cleaner)
import { cleanString, waitFor } from '@/utils'
import type { ElementHandle, Page } from 'puppeteer'
export class PageElement {
constructor(private readonly page: Page) {}
async loadAllElements() {
const pageElement = new PageElement(this.page)
const button = await pageElement.getElement('.vo-btn-blue')
if (button) {
await button.click()
await waitFor(5000)
await this.loadAllElements()
}
}
async getElementsByUniqueSelector(selector: string, fallback?: string) {
const elements = await this.getElements(selector, fallback)
const filteredElements = (
await Promise.all(elements.map(this.isElementWithSingleClass))
).filter((element) => element !== null)
return filteredElements
}
async getContentsByUniqueSelector(selector: string, fallback?: string) {
const elements = await this.getElementsByUniqueSelector(
selector,
fallback
)
const contents = await Promise.all(
elements.map((element) => element.evaluate((el) => el.textContent))
)
return this.cleanContents(contents)
}
async getContents(selector: string, fallback?: string) {
const elements = await this.getElements(selector, fallback)
const contents = await Promise.all(
elements.map((element) => element.evaluate((el) => el.textContent))
)
return this.cleanContents(contents)
}
async getElements(selector: string, fallback?: string) {
const elements = await this.waitForAndGetElements(selector)
if (elements.length > 0) {
return elements
}
if (fallback) {
return await this.waitForAndGetElements(fallback)
}
return []
}
async getContent(selector: string, fallback?: string) {
const element = await this.getElement(selector, fallback)
const content = await element?.evaluate((el) => el.textContent)
const cleanedContent = cleanString(content || '')
return cleanedContent
}
async getElement(selector: string, fallback?: string) {
const elements = await this.getElements(selector, fallback)
return elements[0]
}
private async waitForAndGetElements(selector: string) {
const el = await this.page
.waitForSelector(selector, {
visible: true,
timeout: 5000,
})
.catch(() => null)
const elContent = await el?.evaluate((el) => el.textContent)
console.log({ elContent })
return await this.page.$$(selector) -> error says that the problem is here. Puppeteer can't get the ".v-test-questions-title" element
}
private async isElementWithSingleClass(element: ElementHandle) {
const className = await element.getProperty('className')
const classList = (await className.jsonValue()).toString().split(' ')
return classList.length > 1 ? null : element
}
private async cleanContents(contents: (string | null)[]) {
return contents.map((content) => cleanString(content || ''))
}
}
Also, my cluster initialization
import { Cluster } from 'puppeteer-cluster'
export const cluster = await Cluster.launch({
concurrency: Cluster.CONCURRENCY_CONTEXT,
maxConcurrency: 1000,
// monitor: true,
puppeteerOptions: {
headless: true,
dumpio: true,
args: [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-dev-shm-usage',
],
},
})
And this is the service code
async solveTestByTitle(testTitle: string) {
await this.enterCabinet()
const test = new Test(this.page)
const pageElement = new PageElement(this.page)
await test.startSolvingTestByTitle(testTitle)
const question = await pageElement.getContent('.v-test-questions-title')
console.log({ questionInService: question })
while (question) {
await test.solveSingleAnswerQuestion()
}
}
I have been trying a lot of things like changing puppeteer options, running code in the container, changing the browser, changing the code, using page.waitForNavigation(), page.waitForNetworkIdle in some places, nothing helped, but some of the approaches just changed the error name.
Worth noticing that when a bot answers the question, url does not change.
