Diving into the Twitter API

Twitter’s astonishing growth is rivaled only by its intuitive, developer friendly API. In this second part of the series, we are going to learn more about Twitter’s API and how to work with it.

Too much abstraction is never a good thing.

In this Web 2.0 era, web applications which have an easy to use, intuitive API have a distinct advantage as it lets developers exploit and build for the platform and thus capture more users. As we move towards the social web and mashups, a good API is not a nice addition anymore: it is downright necessary. And remember, too much abstraction is never a good thing. While there are a number of API kits out there to simplify working with the API in question, wouldn’t it be cool to know what is actually going on under the hood? Wouldn’t it be exciting to deconstruct the actual voodoo going on between the kit and the API? Yeah, I thought so!

Before we Begin

For each functionality, I am going to show you how to achieve it in two ways: the first using a server side language, PHP in this case, and the second using only a client side language, JavaScript. Each implementation will function separately and achieve the required functionality. Feel free to mix and match these solutions to create a hybrid. The JavaScript examples will use JSON as the format for the payload while for the PHP example I’ve chosen XML.

Twitter API Basics

The first thing you need to know about the Twitter API is that it is RESTful. To cut the jargon, it means you access appropriate URLs using GET or POST requests to modify, request or manipulate the data exposed by the API.

There are three separate Twitter APIs actually.

The normal REST based API
The Search API
The Stream API

Each of these APIs have their own distinct set of functionality, quirks, advantages and disadvantages.

REST API

The REST methods constitute the core of the Twitter API, and are written by the devs at Twitter itself. It allows other developers to access and manipulate all of Twitter’s main data. You’d use this API to do all the usual stuff you’d want to do with Twitter including retrieving statuses, updating statuses, showing a user’s timeline, sending direct messages and so on.

Search API

The search API is actually the brainchild of the guys over at Summize Inc, a company which Twitter acquired for its API. The search API lets you look beyond you and your followers. You need this API if you are looking to view trending topics and so on.

Stream API

Finally we have the stream API. This API lets developers sample huge amounts of real time data. Since this API is only available to approved users, we aren’t going to go over this today.

Authentication and all that Boohockey

Public data can be freely accessed without an API key. When requesting private data and/or user specific data, Twitter requires authentication. You can authenticate with Twitter using either of two methods.

Basic Authentication

This is the default method of authentication Twitter initially launched with and still uses. In this method, you pass the user name and password as Bse64 encoded strings in the header of the HTTP request. A sample GET request would look like so.

GET /somefolder/index.html HTTP/1.1
Host: net.tutsplus.com
Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

Looks a tiny bit complicated doesn’t it? Let me explain. The first two lines are part of a standard GET request. The third line is where all of magic happens. We make sure the server knows we are using basic auth to authenticate with it, and then pass in the base 64 encoded string of username:password as the value. The server can then decode the credentials, check them and provide access if everything matches.

Twitter supporting basic auth is probably one of the reasons it got so big since it let a multitude of developers develop for the platform since the barrier of entry is relatively low. Basic auth is sufficient to retrieve all parts of the API.

OAuth

Twitter started supporting OAuth in the second quarter of 2009. Users using OAuth enabled application need not divulge their passwords to the program and can also delegate control to the program with multiple levels of access. However, we’ll not look at Twitter’s OAuth implementation today. OAuth and implementing its workflow is a rather complex topic and requires an article of its own. I’d rather skip over OAuth for this article rather than throw out some half baked explanations and writing code based on those explanations.

Caution

“Twitter only lets you make a predefined number of calls to its API.”

Before we delve in to the coding part, I need to make one thing absolutely clear: Twitter only lets you make a predefined number of calls to its API. The REST API’s limit is 150 for an hour, while the search API’s limit is undisclosed as of now. For authenticated calls, the call is deducted from the authenticating user’s limit, while for unauthenticated calls it is deducted from the calling IP’s quota.

Remember, when developing an application, make sure you either cache the information or respect the limit and stay within it. If you think the preset limit is insufficient for your application, you could always apply for white listing.

The Most Common Tasks

Instead of looking at a number of methods you probably won’t ever use, I’ve decided to show you how to do the three most common tasks.

Retrieving your status
Updating your status
Searching Twitter

I think this is a proper subset of the API which directly covers what the API is most used for without spreading itself too thin by covering a ton of methods. With that out of the way, let’s get started!

Retrieving your Status

The first functionality we are going to look at is displaying a user’s status. This is the most popular use of the API if a web developer wants to display a given user’s status without resorting to a Twitter badge.

PHP Method #1: Direct Retrieval

<?php
$response = new SimpleXMLElement('http://twitter.com/users/show/userid.xml',NULL,TRUE);
echo $response->status->text.' - '.date("g:i: A D, F jS Y",strtotime($response->status->created_at));
?>

The first method utilizes PHP. It’s a relatively simple method where we just grab the user specific XML file, and then parse it to display the current status.

We first convert the XML file into an object so we can easily traverse through it using the SimpleXML extension which comes standard with PHP. Now that the object has been created, we just find the status node, and print out the status text and creation time.

Remember, this only displays the current status of the user. If you want a list of recent updates, see below.

PHP Method #2: cURL

cURL is a command line tool and comes enabled on most hosted servers. To quote Wikipedia:

In computing, cURL functions as a command-line tool for transferring files with URL syntax.

It is a library designed to allow users to connect to different types of servers using different protocols. Using cURL is the method of choice for all the Twitter libraries out there and we’ll be using the same.

<?php

     $username = 'xxx';
	 $password = 'yyy';
     $curlhandle = curl_init();
	 curl_setopt($curlhandle, CURLOPT_URL, "http://twitter.com/statuses/user_timeline.xml");
	 curl_setopt($curlhandle, CURLOPT_USERPWD, $username.':'.$password);
     curl_setopt($curlhandle, CURLOPT_RETURNTRANSFER, 1);
     $response = curl_exec($curlhandle);
     curl_close($curlhandle);
     $xmlobj = new SimpleXMLElement($response); 

	 foreach ($xmlobj->status as $status)
	 {
    echo $status->text.'<br /> by'.$status->user->screen_name.' at '
    .date("g:i: A D, F jS Y",strtotime($status->created_at)).'<br /> <br /> ' ;
    }
?>

Let me explain. First, we assign our username and password to separate variables to be used later. I then set the URL to the value mentioned above. Since the method requires authentication, we send in our username and password.

Next, we execute our call and store the response in a seperate variable. We then convert it into a valid XML object and then parse the data, printing out the status and when it was posted.

I’ve formatted the data and time to look the way I want. If you want fancy dates and times like 3 hours ago or 12 days ago, you either need to look for a snippet or code one yourselves. Twitter only returns normally formatted data and time.

You’ll receive an output, similar to this:

I am stupid. Didn't use an IDE and missed a comma wasting 110 minutes.
Remind me not to try debugging when I am sleepy or tired.
at 6:01: PM Fri, February 14th 2010

Flood of traffic. :O I must have budgeted for this many visitors. Argh! Hope nothing breaks.
at 8:51: PM Thu, February 13th 2010

JavaScript

Using JavaScript to display your status is the most elegant way to go forward since you can load it asynchronously after the page has loaded, which means, even if Twitter is downm or being tardy, your site functions the same.

We are going to use jQuery’s ajax method to do all our dirty work today.

 $.ajax({
    		url : "http://twitter.com/statuses/user_timeline/userid.json?callback=?",
			dataType : "json",
			timeout:15000,

			success : function(data)
			{
				  $("#data").html("Data successfully obtained! <br />");
  for (i=0; i<data.length; i++)
				{
  $("#data").append("<p>" + data[i].text) +"</p>";
  $("#data").append("<p>" + data[i].created_at +"</p>");
				}
			},

			error : function()
			{
				alert("Failure!");
			},

		});

Again, a big chunk of code but, broken down, it’s simple really. We use jQuery’s lowest level AJAX function instead of the getJSON method, since the low level call seems to be more versatile.

First up, we define the URL and the datatype. We also add a callback function to the URL to circumvent the cross domain restriction on most browsers. Without this callback, our script wouldn’t run; it’d just return an error and quit.

I’ve chosen not to authenticate, because we are specifying an ID in the URL, and hence don’t need authentication – that and because basic auth is not really safe for sensitive information. You don’t want to send out your password over an insecure line.

Finally, the success function which is called when no errors are encountered. We just parse the returned JSON object and print out the text and the creation time. #data is just a container where we put all our data.

A Quick Note

This code is the template for all your methods which access data. Very minimal change is required to modify it to work with other API methods.

For the PHP example, all you’d need to do is change URL value to point to a new method, and you should be mostly done. If the method requires parameters, you just add them directly to the URL itself. Simple as that.

Same with the JavaScript example. All you’d need to do is change the URL that the methods request, and you should be done. Be sure to fetch only public data with the JavaScript method.

Remember, this code is the base for all your other methods which retrieve data. This includes methods getting your direct messages, timelines, friends, followers and mentioned tweets. Just change the url, add in a parameter as needed, and you’re all set! Easy, no?

Updating your Status

With any Twitter application you are creating, letting users update their status through it is a no-brainer. Remember, previously applications using basic auth were able to use a custom source string for all tweets sent from their application. Now, implementing OAuth is the only way you get a custom string. In short, if you want all tweets sent from your application to have a link back to your app, use OAuth. With that out of the way, let’s see the code.

PHP

<?php

    $username = 'xxx';
	 $password = 'yyy';
    $status= 'Testing out cURL with the Twitter API';
    $curlhandle = curl_init();
	 curl_setopt($curlhandle, CURLOPT_URL, "http://twitter.com/statuses/update.xml");
	 curl_setopt($curlhandle, CURLOPT_USERPWD, $username.':'.$password);
    curl_setopt($curlhandle, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($curlhandle, CURLOPT_POST, 1);
	 curl_setopt($curlhandle, CURLOPT_POSTFIELDS, "status=$status");
    $response = curl_exec($curlhandle);
    curl_close($curlhandle);

?>

We can use cURL just as easily to post to Twitter. The code is almost the same as before except we change the url to the appropriate one. Also, we make sure posting is enabled, and set the fields to be posted. The API method requires a status parameters, and so we set the value of the status variable to this parameter.

Do remember to make sure the text to be posted is UTF-8 encoded. Otherwise, we’d run into needless errors.

We now save the response to be used for later. In this example, I’ve chosen to do nothing. But in a practical application you’d definitely want to display a success/error message and/or display the response.

JavaScript

Posting a status update to Twitter using only JavaScript seems to be impossible right now since there is no way to pass the user’s id and password. With Flickr, those credentials are passed on as part of the POST request itself. With Twitter, this data needs to be sent in the HTTP headers, and there doesn’t seem to be a way of doing that.

Even disregarding the fact that you can’t send credentials over to the service, there is still the problem of not being able to make cross domain POST requests with the XMLHttp object. These two points make API methods which require a POST request a strict no-no with JavaScript.

In case you are interested, a successful post nets you this response.

<?xml version="1.0" encoding="UTF-8"?>
<?xml version="1.0" encoding="UTF-8"?>
<status>
  <created_at>Fri Aug 14 21:31:53 +0000 2009</created_at>
  <id>3316091255</id>
  <text>Testing out cURL with the Twitter API</text>
  <source><a href="http://apiwiki.twitter.com/" rel="nofollow">API</a></source>
  <truncated>false</truncated>

  <in_reply_to_status_id></in_reply_to_status_id>
  <in_reply_to_user_id></in_reply_to_user_id>
  <favorited>false</favorited>
  <in_reply_to_screen_name></in_reply_to_screen_name>
  <user>
    <id>18118645</id>
    <name>Tony / Siddharth</name>

    <screen_name>lordtottuu</screen_name>
    <location>India</location>
    <description>Gamer, developer, web designer, writer, geek. </description>
    <profile_image_url>http://s3.amazonaws.com/twitter_production/profile_images/300532536/NVAGSoNIpS_o5XxbmB4pl-boGheY6JBASaiLiieGIAO6JTitHnRMNJa6ktCPRMQW_normal.jpg</profile_image_url>
    <url>http://ssiddharth.com</url>
    <protected>false</protected>

  </user>

</status>

A Quick Note

The code techniques discussed above consist of the base for all your data which sends data to Twitter. This includes methods which lets you follow/unfollow someone, send direct messages, creating favorites, blocking people, and so on and so forth.

The only thing you’d need to do is change out these URLs, see whether they require extra parameters, and add them as needed. Nothing else required.

Searching Twitter

Letting users search through Twitter for information is potentially an important need of an application. For this purpose, we can use the search API to query Twitter.

PHP

<?php

    $searchstring = "Envato";
    $curlhandle = curl_init();
	curl_setopt($curlhandle, CURLOPT_URL, "http://search.twitter.com/search.json?q=$searchstring");
    curl_setopt($curlhandle, CURLOPT_RETURNTRANSFER, 1);

    $response = curl_exec($curlhandle);
    curl_close($curlhandle);

    $json = json_decode($response);
	foreach ($json->results as $result)
	{
	echo $result->text;
	}

?>

The above code lets you search Twitter for tweets which reference Envato. As usual, we change the URL to point to the correct API method and proceed. Since this method is only available in either JSON or ATOM formats, I’ve chosen to go with JSON.

I’ve used PHP’s built-in json_decode function to convert it into an object so we could parse the response easily. In this example, I’ve only printed out the tweets themselves. In your applications, you’d probably want to display more. A sample response is below.

[text] => @nsethi check out http://www.envato.com/ if you want some sick tuts...I'm gonna blog about it later.
[to_user_id] => 1273919
[to_user] => nsethi
[from_user] => thinklime
[id] => 3315720513
[from_user_id] => 33020944
[iso_language_code] => en
source] => Tweetie
[profile_image_url] => http://s3.amazonaws.com/twitter_production/profile_images/201032569/idTwitter_normal.jpg
[created_at] => Fri, 14 Aug 2009 21:10:42 +0000

As you can see, a lot of info about the user and the tweet itself is available. Feel free to mix and match.

JavaScript

 $.ajax({

    		url : "http://search.twitter.com/search.json?q=somestring&callback=?",
			dataType : "json",
			timeout:15000,

			success : function(data)
			{
				  // parse data here
			},

			error : function()
			{
				alert("Failure!");
			},

		});

As usual, we use the ajax method to ping Twitter’s search service. Just as in the previous example, we’ve included a callback function to get over the cross domain restriction.

A sample response, plucked directly from Twitter, looks like so.

{"results":[

     {"text":"@twitterapi  http:\/\/tinyurl.com\/ctrefg",

     "to_user_id":396524,

     "to_user":"TwitterAPI",

     "from_user":"jkoum",

     "id":1478555574,   

     "from_user_id":1833773,

     ... truncated ...],

     "since_id":0,

     "max_id":1480307926,

     "refresh_url":"?since_id=1480307926&q=%40twitterapi",

     "results_per_page":15,

     "next_page":"?page=2&max_id=1480307926&q=%40twitterapi",

     "completed_in":0.031704,

     "page":1,

     "query":"%40twitterapi"}
}

As a side note, the search method can be used in a variety of ways really.

?q=Envato – Searches for tweets containing the string Envato
?q=#Envato – Searches for hashtags of Envato
?phrase=Envato+marketplace – Searches for tweets containing the phrase Envato marketplace
?q=from:NETTUTS – Returns all tweets from NETTUTS
?q=to:NETTUTS – Returns for all tweets to NETTUTS

Why Other Methods Aren’t Covered

I’ve decided to limit myself to showing how to read from and write to a service using its API. This way, I can focus only on those specific methods. And also because most methods build on these basics. If you want to return a friend’s timeline, you’d just change the URL and parse through the response with minimal fuss.

The same occurs if you want to send a direct message to another user or mention another user. The basics of posting is still there. You just need to add in a parameter or two to make it all happen. It doesn’t make sense to cover all that when I can just explain the basics a lot more clearly.