How To Implement a RESTful API

Today we will discuss a REST API. We’ll briefly go over what it is and then we’ll implement one from both the server and client perspective.

What is a REST API

A lot of public APIs are REST APIs and knowing how to use them, interact with them and understanding what goes on in the backend can help you leverage a lot of services available. REST stands for “Representational State Transfer” which is a fancy way of saying when you contact the server, you can expect a response. REST APIs are used everywhere, from Google, to Twitter, from Facebook to Amazon. In fact, Amazon’s REST API is used in the saAlbumCoverArt download which downloads music album cover images.

Providing a REST API to your application allows other programmers or developers to hook into your web site. For Twitter, you contact the REST API to get anything from a follow list to recent messages (direct and public). It sounds really cool, ‘REST API’, really programmy, but it really is nothing special. We’ll go through a full example of a REST api from the client request to the server outputting the response.

An Example of a REST API request

We’ll pretend that Simply Amused provided an API allowing you to delete users. For now we’ll ignore the fact that this is completely insecure and dangerous. A REST API has a number of steps involved and has some specific requirements and assumptions.

  1. The client – our fictional program that will contact the server
  2. The server – in this case, it’s simplyamused.com
  3. Assumption: The API will return it’s result in XML.
    The structure will look like this:
    <result>
      <code>1 | 0</code>
      <message>Message</message>
    </result>
    

Let’s take a look at a full example, complete with controllers, models, exceptions and object orientation. We will use Perl on the client script, and PHP for the backend.

1. The Client – Make the request

The client script will make a HTTP request of the server.
http://simplyamused.com/api.php?action=deleteUser&userId=2

The request looks like any other URL that you would see on the Internet. Looking at the GET variables, it’s a rather simple API to use. While our example URL is simple, in practice, you will find the URLs will look quite long as there is usually a token or hash to authenticate the request. For our example though, we will simply the API.

The client script that utilizes the API is written in Perl. The reason it is in Perl is to show you that the REST API allows two different environments, languages or servers to talk to each other in a pre-defined way. This is the true power of a REST API. The client script doesn’t care that the server used PHP and that it uses Perl.

#!/usr/bin/perl

use strict;
use XML::Simple;
use LWP::Simple;
use Data::Dumper;

my $result = get('http://simplyamused.com/api.php?action=deleteUser&userId=2');

my $xmlResult = XMLin($result);

if ($xmlResult->{'result'}->{'code'} == 1)
{
  print $xmlResult->{'result'}->{'message'} . "\n";
}
print Dumper($xmlResult);
1;

The client script makes the request using Perl’s LWP module and then parses the XML for parsing to the XML::Simple module. The client script could be part of an automated provision/de-provision system, or it could simply be a handy script used to delete people.

2. The Server – Delegate the request

In our example above, the server used a file named api.php to handle the request, and had a required argument passed to it; action = deleteUser. The action variable determines what class api.php will call. The file api.php will delegate the actual work of deleting a user to the appropriate class based on the action variable. Consider this code:

' . "\n
    %d\n
    %s\n
  
";

try {
  // remove any unexpected characters from the action variable
  $action = preg_replace('/[^a-z0-9_]+/i', '', $_GET['action']);
  if (!isset($action) || empty($action))
  {
    throw Exception('Invalid API request');
  }

  $className = $action . 'API';
  if (class_exists($className)
  {
    $ApiCall = new $className();
    $ApiCall->run();
    printf($xmlFormat, 1, $ApiCall->getSuccessMsg());
  }
  else
  {
    throw new Exception('Invalid API request');
  }
}
catch (Exception $e)
{
  printf($xmlFormat, 0, $e->getMessage());
  // output an XML response
}

Our file api.php has delegated the request to the appropriate class.

Every [action]Api class will implement a run() method and that method will attempt to complete the request. If at any time, an exception is thrown, as a way to kill the script due to an error in input or processing, we can output our error XML response. Just like the run() method, every [action]Api class will implement a getSuccessMsg() method which returns a string used for the success message.

While I have chosen not to show it, all [action]Api classes should either implement an API Interface or extend an abstract API class. This will help define and ensure all API classes implement our two methods.

The use of exceptions allows our API to return a result even if something failed or has gone horribly wrong on the backend. Using exceptions in this method allows us to still output the expected API response so that client scripts or applications do not hang if an invalid response is sent. This is very important when building an API.

The output structure is defined in the $xmlFormat declaration at the top of api.php. Since our API is simple, and for brevity, I have chosen this method to output XML. It’s a quick hack to show how it works, not necessarily to be used as-is.

3. The Server – Handle the request

Since api.php has instantiated the deleteUserApi, let’s take a look at this object.

// Class deleteUserAPI.class.php
class deleteUserApi extends AbstractAPI
{
  protected $this->userId;

  public function __construct()
  {
    $this->userId = intval($_GET['userId']);
  }

  public function getSuccessMsg()
  {
    return 'Successfully deleted the user ' . $this->userId;
  }

  public function run()
  {
    // UserModel::deleteUser executes the following SQL and returns bool:
    //   'DELETE FROM users where userId= ' . $this->getOption('userId');
    if (!UserModel::deleteUser($this->userId))
    {
      throw new Exception('Could not delete the user $this->userId);
    }
    return true;
  }
}

It’s a simple class, where the run() method calls the model and deletes a user. If it fails, for whatever reason, it will throw an Exception which will be caught in api.php. If we wanted to add a new API command we would just add another class ensuring we implement the methods run() and getSuccessMsg() while ensuring all errors checking we do throws an Exception.

In Conclusion…

Rest APIs are very powerful. They allow different environments to communicate with each other and interact with one another. In today’s “connected world” Rest APIs are becoming ever more popular and knowing how to interact with and create them will be a benefit.

While the code I presented above is not robust, it does show what it takes to get an API running from both ends. The concepts are simple once broken down, and using objects to handle responses means your api.php file can be very generic by exploiting the fact that classes can have the same method in each

Posted in Articles, Programming | Leave a comment

Perl Exception Handling

What Is Exception Handling?

In it’s simplest form, Exception Handling allow you to write a block of code and wrap it in your own form of error handling. Typically a program will attempt to accomplish a number of tasks and then report to you the status of those tasks or produce an error. However, often when writing a program you do not want your program to die immediately when an error occurs as you may be able to handle those errors. This is what Exception Handling does for you. It is a programming construct designed to handle an error caused at runtime.

For those of you that understand how transactions work in a RDBMS, it’s the same principle. You can attempt to execute a chunk or block of code (or SQL statements) and if an error occurs somewhere, you can catch it (or in the SQL-world, rollback).

Here’s an example of a typical program that initiates a database connection, tries to do something and then inserts some data.

#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect('DBI:mysql:dbname', 'user', 'pass')
  or die "cannot connect to db";

# With the db command, do stuff..
my $id = connectToWebService('twitter', 'send', '@bob, Hi')
  or die "connectToWebService() failed for twitter...";

# Insert the $id->{'twitid'}, $id->{'msg'} into the database
...

print "success!\n";

It’s a simple enough script to understand. The connectToWebService() function is just a fake function I made up for our example. If it fails, it will terminate the program.

Handling errors with an exception allows you the flexibility of letting the application continue running. This provides the programmer a common tool to use to handle errors. Most languages provide it’s own Exception Handling framework. Everything from C to PHP, Java to .NET, they all provide built-in methods to handle exceptions. For my use, all languages provide them except Perl.

How To Do Exception Handling In Perl

Perl5 is probably the most common language that does not provide exception handling. However, like all common programming problems, someone has already thought of the solution. In Perl’s case, you would use the eval() function. Let’s look at our example from above but this time use an Exception Handling method for Perl.

#!/usr/bin/perl -w

use strict;
use DBI;

my $dbh = DBI->connect('DBI:mysql:dbname', 'user', 'pass')
  or die "cannot connect to db";

eval {
  # With the db command, do stuff..
  my $id = connectToWebService('twitter', 'send', '@bob, Hi')
    or die "connectToWebService() failed for twitter...";

  # Insert the $id->{'twitid'}, $id->{'msg'} into the database
  ... run code here ...
  1;
} or do {
  print "error!! - use the $@ variable for info\n";
  ... run more code to handle the error...
  $rc = $dbh->disconnect or warn $dbh->errstr;
}

print "success!\n";

Because the eval block ends with a perl-style return true “1;”, we know that the eval succeeded. If for example an error occurred and connectToWebService didn’t work because it couldn’t connect, eval would not be true and the “or do” block would be run. The die statement in the eval block “throws an exception” and is “caught” by the or do block below.

The “or do” block has access to the $@ global variable. The most common use is to provide a string to the die statemenet thus making it available in the $@ variable. If you are already familiar with exception handling for other languages, exec maps to try while “or do” is the catch code block. The $@ global variable is where you access the error message, if you have thrown one.

Posted in Articles | Leave a comment

Diff two source controlled directories

I constantly need to diff two directories just to check what files have changed between revisions.

When you do a CVS checkout of two separate branches whether they’re incremental or not, sometimes you need to check which files differ as a pre-check before doing something else. Here’s a little code snipped that will allow you to do just that:

diff -rbB --brief $1 $2  | egrep -v '(CVS|.svn|~|\.#)'

Throw that in a file named ‘diffdir’ or whatever you like in your ~/bin directory, make it executable and you’re good to go.

Posted in Snippets | Leave a comment

Why Database Abstraction Is Useless 90% of The Time

I’m sure you’ve read a lot of posts, articles and forums where one of the “best practices” is to use a database abstraction layer. It really isn’t, and experience has taught me that the amount of times you change database server is far too minimal to matter. When you do change, you’re re-writing a lot of SQL again to either leverage your change, or remove vendor-specific commands.

Portability Is Impractacle

Many proponents of database abstraction layers state that if a good abstraction layer is used, when you change your database it’s as simple as changing $db to $newdb and all of your code will continue to work; it’s just that easy. Unfortunately it’s not and statements like those couldn’t be farther from the truth.

Unless the only thing your application does is insert/update/delete one row in one table ever, you will run into problems. If this is the case, why are you changing to a new database server to begin with? You should buy yourself a lottery ticket if the only thing you must worry about is changing mysql_query() to pg_query(). The vast majority of times you will also need to redesign the database schema, re-create views/triggers/functions and re-write your code and SQL to accomodate new or different data types, indexes or syntax.

In some code I’ve seen, you will see a database abstraction layer which wraps mysql_query() but then will make use of auto_increment values and mysql_insert_id() calls. The abstraction layer is useless in this instance because should you switch your DB to PostgreSQL you will find auto_increment as a flag does not exist, and getting the last inserted primary key requires a sequence to be created. This is the simplest of conversion issues you will run into, nevermind triggers and functions.

Any good programmer will alway want to use the best tool for the job, versatility just isn’t viable. For databases, even moreso as each RDBMS implements it’s own unique features, functions and methodologies that must be considered in any database application (like the last inserted primary key feature that MySQL provides). While limiting yourself to the non-vendor specific features will make your SQL more portable, it will do nothing to improve your application both from a performance point of view as well as architecturally. Your SQL will actually become more complex as you attempt to make it more generic.

Changing a live application’s database just doesn’t happen often enough as it is cost prohibitive. Your database schema and SQL are not portable across different types of servers and your database abstraction layer doesn’t actually solve this problem.

Database Abstraction Vs. Database API

The other major argument for a database abstraction layer is that your code is not littered with mysql_query() statements everywhere. Instead, they are $db->query() or some other similar construct which will allow you to change the $db variable from one class to another. I hope you can see the error in this way of thinking. As discussed earlier, your SQL code can’t be ported from server to server in any non-trivial application.

The solution to this problem is rather simple. Move your abstraction further up the stack. Create an API for your application to interface with. This produces a level of abstraction from your application to your data source while still utilizing your server’s specific features.

Your code will not have SQL littered everywhere, it will be in one self-contained location so that updating, changing or even depreciating various tasks can be done easily. The code below shows you how the API approach differs from an abstraction layer.

class DataAccess()
{
  public function __construct($host, $user, $pass)
  {
    $this->link = mysql_connect($host, $user, $pass) or die 'errr';
    mysql_select_db('my_db_name', $this->link);
  }
}

class Users extends DataAccess
{

  public function varDumpUsers()
  {
    $q = mysql_query('SELECT * FROM user', $this->link);
    while (( $row = mysql_fetch_array($q) ))
    {
      var_dump($row);
    }
  }
}
// -- separate file with the appropriate require statement
$_user = new Users();
$_user->getAllUsers();

And now look at our database abstraction layer :

abstract class baseDb
{
  abstract public function __construct($host, $user, $pass);
  abstract public function query($sql);
  abstract public function getRow();
}

class MySQLDb extends baseDb
{
  public function __construct($host, $user, $pass)
  {
    $this->link = mysql_connect($host, $user, $pass) or die 'errr';
    mysql_select_db('my_db_name', $this->link);
  }

  public function query($sql)
  {
    return mysql_query($sql, $this->link);
  }

  public function getRow()
  {
    return mysql_fetch_array();
  }
}
// -- separate file with the appropriate require statement
$_db = new MySQLDb($host, $user, $pass);
$result = $_db->query('SELECT * FROM user');
while (( $row = $db->getRow() ))
{
  var_dump($row);
}

What is the difference? The difference lies in where the abstraction occurs. It’s not at the database connection level. All SQL is contained in one location (my API classes) and this allows me to decrease code-duplication, and let my application become a little dumber (or decouple it). In the first example, my code doesn’t care what database I’m using. It calls varDumpUsers() and expects somehow, all users get dumped to the screen. It’s an API call with a defined input of nothing and a return value of void. In the abstraction layer example, the application knows it’s talking to a database, provides the SQL to use and then executes it.

One of the benefits of the API method is that if I am constantly var dump’ing the values from the user table, it’s one call to the API. Via the abstraction method, it’s SQL, a loop and a var dump duplicated anywhere I need it. One of the golden rules of coding is never duplicate code.

What About Combining The Two?

Combining the two patterns can be beneficial as long as the abstraction layer is not simply a wrapper to the various mysql_ functions. To expand on our API example, consider the following:

class DataAccess()
{
  private $result = null, $link = null;
  public function __construct($host='localhost', $user='user', $pass='pass')
  {
    $this->link = mysql_connect($host, $user, $pass) or die 'errr';
    mysql_select_db('my_db_name', $this->link);
  }

  public function query(sql)
  {
    $this->result = mysql_query($sql, $this->link);
    if (!$this->result)
    {
      error_log('MySQL error ' . mysql_errno() . ' - ' . mysql_error());
      throw new Exception('MySQL error ' . mysql_errno() . ' - ' . mysql_error());
    }
    return $this->result;
  }

  public function getRow($offset=0)
  {
    if ($offset != 0) { mysql_data_seek($this->result, $offset); }
    return mysql_fetch_array($this->result);
  }
}

class Users
{
  private $db;

  public function __construct($db)
  {
    $this->db = $db;
  }

  public function varDumpUsers()
  {
    $this->db->query('SELECT * FROM user');
    while (( $row = $this->db->getRow() ))
    {
      var_dump($row);
    }
  }
}
// -- separate file with the appropriate require statement
$_da = new DataAccess(); // put in var so it can be passed to other API classes
$_user = new Users($_da);
$_user->varDumpUsers();

We’ve added to our DataAccess class and provided more functionality. Our query() method will throw an exception if it encounters an error and our getRow() method allows for offsets. It’s more than just an abstraction now, it’s layer that provides actual functionality on top of the standard mysql_() functions.

With that said you will also notice that the DataAccess class doesn’t add much value to the API. The only reason it’s used is to ensure that each additional API class is able to re-use the same database connection in this example. It’s still a database abstraction layer at it’s heart.

An Alternative?

Combining the two works, however the abstraction layer’s only justifiable purpose is to separate the database connection out of the API so that the API is not making new connections all the time. There are ways around this of course; you could make DataAccess a singleton, but consider this instead:

class ApiLoader()
{
  private $loadedApi = array(), $link = null;
  private static $instance;
  private function __construct()
  {
    $this->link = mysql_connect($host, $user, $pass) or die 'errr';
    mysql_select_db('my_db_name', $this->link);
  }

  public static function getInstance($apiName)
  {
    if (!self::$instance)
    {
      self::$instance = new ApiLoader();
    }
    return self::$instance;
  }

  public function loadApi($apiName)
  {
    if (!isset($this->loadedApi[$apiName]))
    {
      $this->loadedApi[$apiName] = new $apiName($this->link);
    }
    return $this->loadedApi[$apiName];
  }
}

class Users
{
  protected $link;

  public function __construct($link)
  {
    $this->link = $link;
  }

  public function varDumpUsers()
  {
    $q = mysql_query('SELECT * FROM user', $this->link);
    while (( $row = mysql_fetch_array($q) ))
    {
      var_dump($row);
    }
  }
}
// -- separate file with the appropriate require statement
$_apiLoader = new ApiLoader::getInstance();
$_apiLoader->loadApi('Users')->getAllUsers();

The ApiLoader->loadApi() method will load any API. Ideally it would throw an exception if it can’t include the file, or the class does not exist and the API should throw an exception if the SQL failing is fatal, but in it’s most basic form, the above code should demonstrate another alternative to database abstraction. Yes, it could use a lot of work, but the goal of the class is to show an alternative to wrapping a bunch of mysql_ functions in a class.

The ApiLoader class has one job, that is to load the API I wish to use. It creates the database connection and then instantiates the API. While this may not be the best way to handle API calls, it does remove the the database abstraction layer and actually provide an API of it’s own.

What Is The Other 10%?

The only time I’ve ever seen a plain database abstraction layer benefit anyone is for open source projects for popular applications. If I wrote an application, let’s say Drupal, I’d want as many people to be able to use it as possible. By adding that layer to abstract the database, I can ensure my target audience is broader. Unless you’re writing an application where supporting multiple database servers would help, you’re better off focusing on one database and utilizing all tools / optimizations / features available to you.

Final Thoughts

There is a difference between database abstraction and a database API. One is useful, the other is not.

Posted in Articles | Leave a comment

saAlbumCoverArt: Download album covers from Amazon

Abstract:

These set of classes allow you to use Amazon to download your music album cover images.  Your script will need to pass it the search string, which for amazon works best as “[album name] [artist]“.

Given these set of base classes you are able to add new search providers.  I’ve only written it for Amazon, but if there are others, extend the abstract class saAlbumCoverArtSearch and implement the execute() method.

Requirements:

  • PHP5 compiled with XML, file_get_contents open URL
  • An Amazon Web Services account and private keys

Download and terms:

You may download the archive free of charge.  The classes are fully documented in phpdoc style. The terms for this code is simple. It’s released under the Free Beer license. I will do my best to support the code, fix bugs, but it is provided as is with no warranty.

Download saAlbumCoverArt.tar.bz2

Example:

In it’s simplest form, here is an example:

$album = new saAlbumCoverArt(
  'Album Name',
  'Artist ',
  'path-to-save-cover-image'
)->execute();

Below is an example script showing how one could get all album covers from a music collection stored in a database, save the file to the directory of that music album, and then update the database.  This example runs from the CLI.  Changing this to run in web should be pretty trivial as lines 43 to 53 are the meat of the implementation.

#!/usr/bin/env php
<?php

$conn = mysql_connect('localhost', 'dbuser', 'dbpass')
    or die('cannot connect');
mysql_select_db('dbname');

$sql = "
  SELECT
     a.name              AS albumname,
     a.id                AS albumid,
     art.name            AS artistname,
     m.music_artist_id   AS music_artist_id,
     m.file_name         AS file_name,
     m.full_path         AS full_path
  FROM
    music as m,
    music_album AS a,
    music_artist AS art
  WHERE
    m.music_album_id = a.id      AND
    m.music_artist_id=art.id     AND
    a.cover IS NULL
";

$result = mysql_query($sql, $conn);
if (!$result || mysql_num_rows($result) == 0)
{
  echo mysql_error(); echo 'errrrrrrrrrr'; exit;
}
echo "number of albums: " . mysql_num_rows($result) . "\n";

$found = array();

while (($row = mysql_fetch_assoc($result)))
{
  // do not do duplicates
  if (isset($found[$row['albumname']])) { continue; }

  echo 'Running: ' . $row['artistname']
     . ' ' . $row['albumname'] . "\n";

  $cover = new saAlbumCoverArt(
    $row['albumname'],
    $row['artistname'],
    str_replace($row['file_name'],
      '',
      $row['full_path']
    ),
    array(
      'search' => array('basedir' => 0, 'amazon' => 1)
    )
  );
  $coverFile = $cover->execute();
  if ($coverFile === false)
  {
    $coverFile = 'none';
    echo "   None\n";
  }
  else
  {
    echo "    FOUND " . $coverFile . "\n";
  }

  // Update the database table setting the cover file name
  $sql = "
    UPDATE music_album
    SET cover='" . addslashes($coverFile) . "'
    WHERE id=" . $row['albumid']
  ;
  $result2 = mysql_query($sql, $conn);

  $found[$row['albumname']] = true;

}
echo "done \n\n";
Posted in Code | Leave a comment