The website I am working on is dynamic with javascript. When directly using requests, it gives me message <noscript>Javascript is required. Please enable javascript before you are allowed to see this page.</noscript>. I have used this link https://curlconverter.com/# copy the cURL (bash) to it to get the python code. the python code works fine for this approach. I began to remove unnecessary lines of headers, and finally let two of the lines of cookies sucuri_cloudproxy and cookiesession1. The code is working fine.
But I need to get the cookies each time I execute the code, so I used selenium for once to get the cookies. But this doesn’t work as expected and I got the same message posted at the beginning of the question
Here’s my try till now
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import requests
from bs4 import BeautifulSoup
url = 'https://www.moi.gov.kw/mfservices/immigration-fines/residency/295010900353'
opts = Options()
opts.headless = True
driver = webdriver.Chrome(executable_path='D:/Webdrivers/chromedriver.exe', options=opts)
driver.get(url)
cookies = driver.get_cookies()
for cookie in cookies:
if 'sucuri_cloudproxy' in cookie['name']: x1=cookie['name'] + '=' + cookie['value']
if cookie['name'] == 'cookiesession1': x2=cookie['name'] + '=' + cookie['value']
sessionlist = []
sessionlist = list([x1, x2])
sessionid = ';'.join(sessionlist)
print(sessionid, 'n')
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36',
'cookie': "'" + sessionid + "'",
}
mylist = ['236033100148', '295010900353']
for item in mylist:
response = requests.get(f'https://www.moi.gov.kw/mfservices/immigration-fines/residency/{item}', headers=headers)
print(response.text)