im a really newbie python programmer here,
in short i want to scrape data from this website (example link 1, link 2). I want to create a python code that could fetch me the data preferably in below format:
Category Value
Bid Price 101.413 (add 0.191)
Bid Yield to Maturity 4.596%
Ask Price 101.571 (add 0.192)
Ask Yield to Maturity 4.510%
any solution?
I already tried to use chatgpt and my limited knowledge to understand and write the proper command, something like below
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Set up Chrome options (optional)
chrome_options = Options()
chrome_options.add_argument("--headless") # Run in headless mode
# Path to your ChromeDriver (adjust the path accordingly)
driver_path = 'C:/Users/Admin/Desktop/python course/chromedriver-win64/chromedriver.exe'
service = Service(driver_path)
# Initialize the WebDriver
driver = webdriver.Chrome(service=service, options=chrome_options)
try:
# Open the website
url = 'https://www.bondsupermart.com/bsm/bond-factsheet/USY4907LAG78'
driver.get(url)
# Wait for the element to be present using a CSS selector derived from the JS path
wait = WebDriverWait(driver, 10) # Wait for up to 10 seconds
# Using a CSS Selector derived from the provided JS path
ask_yield_element = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "body > app-root > app-pre-login-layout > main > app-bond-factsheet > div.ng-star-inserted > div > div.container.ng-star-inserted > div > div > div.bg-indicative-tpl.rounded-lg.lg\:flex.lg\:h-full > div > div.col-span-12.lg\:col-span-8.\32 xl\:col-span-9 > div > div:nth-child(2) > div > span")))
# Extract and print the Ask Yield to Maturity value
ask_yield = ask_yield_element.text
print(f'Ask Yield to Maturity: {ask_yield}')
finally:
# Close the WebDriver
driver.quit()
while it could give me the right answer of
Ask Yield to Maturity: 4.510%
i only managed to do that when i put the JS path, which is not same for every link, already tried a hackyway such as search for another JS path, but it isnt accurate enough, most likely will show a wrong data.