I am trying to extract the real ranking of products from an e-commerce site, but my script currently assigns ranks based on DOM position, which does not match the actual ranking shown on the site.
For example, my code incorrectly displays Rank #7 as Rank #4 because it is reading products in the order they appear in the HTML structure, rather than their real-time ranking on the site.
What I Have Tried:
Checking HTML Attributes: I inspected product elements in DevTools but couldn’t find data-rank, bestseller-badge, or other useful attributes.
Extracting Rank from Visible Text: Some products have ranking labels, but not all of them.
Looking at Network Requests (XHR/Fetch API): I noticed that the ranking might be fetched dynamically from an API, but I am unsure how to extract this data.
document.querySelectorAll(".product-card").forEach((product) => {
let rankElement = product.querySelector(".bestseller-badge");
let realRank = rankElement ? rankElement.innerText : "Unknown";
console.log(`Real Rank: ${realRank}`);
});
What I Need Help With:
How does an e-commerce site typically store and update product rankings?
How can I retrieve the real ranking (e.g., sorted by best sellers) instead of relying on DOM order?
Should I look at JavaScript-rendered elements or API responses for this data?
How can I fetch ranking data dynamically if the site loads it asynchronously?
I am a computer science student but still learning JavaScript and web scraping techniques. Any guidance would be greatly appreciated!