I have a twitter bot running with PM2 in my Heroku dyno. My bot creates a long-live HTTP request to filter tweets. My goal is to keep this bot running 24/7 in order to fetch every tweet that matches the hashtags that I’m looking for.
My Twitter bot works fine but there’s one problem that is giving me a hard time to debug: The bot is sleeping and not fetching tweets after a certain period of time.
Some times the bot is working for few hours, sometimes for few days. It is not clear what’s wrong since I don’t have logs helping me here. Whatever is happening, it is happening outside of the parts of the code with console.log, nothing is being printed in the Heroku logs.
Here is the code of the stream used in my bot:
const stream = needle.get(streamURL, {
headers: {
"User-Agent": "v2FilterStreamJS",
"Authorization": `Bearer ${token}`
},
open_timeout: 0
});
stream.on('data', data => {
try {
const tweet = JSON.parse(data)
// Handles the Tweet
handleTweet(tweet)
// A successful connection resets retry count.
retryAttempt = 0;
} catch (e) {
if (data.detail === "This stream is currently at the maximum allowed connection limit.") {
console.log(data.detail)
process.exit(1)
} else {
// Keep alive signal received. Do nothing.
}
}
}).on('err', error => {
console.log(error)
if (error.code !== 'ECONNRESET') {
console.log('Error: ' + error)
process.exit(1);
} else {
// This reconnection logic will attempt to reconnect when a disconnection is detected.
// To avoid rate limits, this logic implements exponential backoff, so the wait time
// will increase if the client cannot reconnect to the stream.
setTimeout(() => {
console.warn("A connection error occurred. Reconnecting...")
streamConnect(++retryAttempt);
}, 2 ** retryAttempt)
}
});
return stream;
Most of the code was taken from Twitter v2 Examples so I posting only the part that I updated. I set the open_timeout to 0 in order to avoid Needle timing out the connection. Reading Twitter docs, they have mentioned they send a keep-alive signal every 20 secs to keep the connection alive.
This is my ecosystem.config.cjs config file for PM2:
module.exports = {
apps : [
{
name: 'discord',
script: './app.js',
instances: '1',
exec_mode: 'cluster',
env: {
NODE_ENV: "development",
},
env_production: {
NODE_ENV: "production",
}
},
{
name: 'twitter',
script: './twitter.js',
instances: '1',
exec_mode: 'cluster',
env: {
NODE_ENV: "development",
},
env_production: {
NODE_ENV: "production",
}
}],
};
One weird thing is that running pm2 list doesn’t show any of these apps, even though both of them are running. One of my attempts was to use Heroku Scheduler to restart the twitter app every hour.
Thanks in advance