So I’ve been recently using twitter’s own backend API that it uses to display tweet/user data on a webpage to scrape and try and find a user’s latest tweet. The main issue I’m having with this, is the data I’m getting back is all jumbled and all over the place, and sometimes not even containing the latest tweet. It feels like it’s just grabbing random tweets and sending it back. I’ve even added a function to sort through the tweets and only give me the data for the tweet id, the date it was created, and the text inside of it, but still even with that sorting, it doesn’t seem like it’s getting the latest tweets of users.
What I’m doing is using the network API call in the “UserTweets” API and using that and the headers to make a call to the API myself and get the data response that it normally would with the web browser. I’m using the user’s ID of the user I want to scrape and creating a guest token to use for the x-guest-token header. I’m getting back over 30k lines in a json file of tweets, but I feel like maybe it’s not giving me everything? I just don’t understand why the response I’m getting back is different than the response you see in the “Response” tab of the “Network” tab when looking at UserTweets.
If anyone has something like this functional, can you assit me with what’s wrong and what I’m getting wrong? This is the fetch I’m using to get tweets:
const res = await fetch(
`https://api.x.com/graphql/V7H0Ap3_Hh2FyS75OCDO3Q/UserTweets?variables=%7B%22userId%22%3A%22${userId}%22%2C%22count%22%3A20%2C%22includePromotedContent%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22rweb_tipjar_consumption_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22communities_web_enable_tweet_community_results_fetch%22%3Atrue%2C%22c9s_tweet_anatomy_moderator_badge_enabled%22%3Atrue%2C%22articles_preview_enabled%22%3Atrue%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22responsive_web_twitter_article_tweet_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22creator_subscriptions_quote_tweet_preview_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Atrue%2C%22rweb_video_timestamps_enabled%22%3Atrue%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Atrue%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D&fieldToggles=%7B%22withArticlePlainText%22%3Afalse%7D`,
{
headers: {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/128.0.0.0",
accept: "*/*",
"accept-language": "en-US,en;q=0.9",
authorization: "Bearer AAAAAAAAAAAAAAAAAAAAANRILgAAAAAAnNwIzUejRCOuH5E6I8xnZz4puTs%3D1Zv7ttfk8LF81IUq16cHjhLTvJu4FA33AGWWjCpTnA",
"content-type": "application/json",
priority: "u=1, i",
"sec-ch-ua": ""Chromium";v="128", "Not;A=Brand";v="24", "Microsoft Edge";v="128"",
"sec-ch-ua-mobile": "?0",
"sec-ch-ua-platform": ""Windows"",
"sec-fetch-dest": "empty",
"sec-fetch-mode": "cors",
"sec-fetch-site": "same-origin",
"x-guest-token": token,
"x-twitter-active-user": "yes",
"x-twitter-client-language": "en",
Referer: "https://x.com/",
"Referrer-Policy": "strict-origin-when-cross-origin"
},
body: null,
method: "GET"
});