Monday 20 September 2010

Database Migrations with the Zend Framework

If you're a user of the Zend Framework then you're no doubt aware that it was developed by Zend Technologies in response to the growing popularity of other frameworks (most notably Ruby on Rails and Spring) that don't use PHP. Worried that PHP would lose developer mind share to Ruby and Java, Zend developed the Zend Framework to give PHP developers a well designed foundation to build their PHP applications on.

Over the last 5 years, the Zend Framework has steadily grown into a fine framework to rival its competitors but for me there are a few areas where it is still lacking. If you're familiar with the Rails framework, then you'll know that one tool that Rails provides but that ZF doesn't is managed database migrations.

For these of you who don't know what a database migration in Rails is, consider your approach to designing your application database. Think about how you change a database table design and how you communicate those changes to your colleagues. You might use a WYSIWYG designer like MySQL workbench (and there's nothing wrong with that) to design and modify individual tables but how do you make sure that your team all update their local database schemas? How do you know which version of the database design should the staging server or the live server have?

This is the problem that the database migration paradigm aims to solve. It asserts that the design for the database should sit in the same code base as the whole application, and be versioned by your Revision Control System. Unfortunately, the Zend Framework doesn't help solve this problem and Rails has ZF beat here. In Rails, you drop to your command line, and using Rake (Ruby's make tool) you issue a command to create a migration. This command creates a file that you open in your editor and enter the changes you want to make to the database (in a rather lovely Domain Specific Language). Then you pop back to your command line, issue another command, and Rake applies these changes to your local database. Should you need to roll the changes back, you can do that too.

There have been attempts to bring database migrations to ZF before, notably Rob Allen's Akrabat project. In fact, the only issue I have with Rob Allen's implementation is that it passes the actual database adapter to the migration object, and the migration script interacts with the database directly. To my mind, this isn't quite right; I want the migration script to return a number of SQL statements and to have the migration system manage any actual interaction with the database. One benefit of this is that it makes the migration plugin more flexible; for example, it can then run in what Zend calls pretend mode (where it runs the migration but doesn't actually commit the changes).

The Mooduino Implementation


I've uploaded the first version of my migration plugin to my github (http://github.com/michaelhodgins/Mooduino). It doesn't do everything that I want it to do as yet but it is functional and I've started using it in my own work.

Installation is quite straight forward. First, download the Mooduino library from github. I put my libraries in my home directory on my development PC, so Mooduino would end up in the directory '/home/michael/programming/php/mooduino/'.

Open a terminal and issue these two commands:

zf --setup storage-directory
zf --setup config-file

These will create a file called .zf.ini in your home directory (or in a directory called .zf). Open the .zf.ini file and add the location of your local copy of the Mooduino library to the php.include_path value. This value should also include the path to your local copy of the Zend Framework.

You also need a new line telling the zf command line tool about the Mooduino migration provider. Add the following:

basicloader.classes.0 = "Mooduino_Db_Migrations_MigrationProvider"

If you already have a value for basicloader.classes.0, make it basicloader.classes.1. As an aside, if there is a value for php.includepath, this is a bug (related to Netbeans I think) and it should be php.include_path instead.

To check that this worked, return to your terminal and issue this command:

zf ? migration

This should print the following list, showing you the commands that you can use with the migration plugin.

zf generate migration name env[=development] base-class[=default]
zf redo migration step[=1] env[=development]
zf undo migration step[=1] env[=development]
zf current migration env[=development]
zf update migration to[=latest] env[=development]
zf show migration revision[=list] env[=development]
zf clear migration env[=development]

In operation


To actually use the plugin, you'll need a ZF project with the database configured in application/config/application.ini. The database will also need to have been created.

Use the first command in the above list to generate a migration file. For example:

zf generate migration Notes

creates a file called scripts/migrations/1283772577_Notes.php and this file contains a class called Migration_1283772577_Notes. The long number is a timestamp in seconds and obviously depends on when you issue the generate command. This timestamp is used to sequence the files. This class extends Mooduino_Db_Migrations_Migration_Abstract and contains two empty methods, up() and down(). These two methods are expected to return either a string or an array of strings; the strings are expected to be valid SQL statements for the application's RDBMS. The statements returned by the down() method are expected to reverse the effects of the statements returned by the up() method.

For example, if I wanted to create the notes table from my previous blog post, I'd fill in the migration as follows.

class Migration_1283772577_Notes extends Mooduino_Db_Migrations_Migration_Abstract {

  public function __construct() {
    parent::__construct('Notes', 12837725773
);
  }

  public function up() {
    return 'CREATE TABLE `notes` (
             `id` bigint(20) NOT NULL AUTO_INCREMENT,
             `text` longtext NOT NULL,
             `title` varchar(255) NOT NULL,
             `priority` int(11) NOT NULL,
             PRIMARY KEY (`id`),
             UNIQUE KEY `Search` (`title`),
             KEY `Priority` (`priority`)
            );
    ';
  }

  public function down() {
    return 'DROP TABLE `notes`;';
  }

}


If you return to your terminal, and issue the command zf show migration list, you'll see the migration details, showing that it hasn't been executed against your local database. To execute the up() method, issue the command zf update migration. This will execute all unapplied migrations unless you include the fourth parameter. If you issue this command and then look at your database, you'll see the notes table has been added, along with a table called schema_version, which the plugin uses to maintain its state.

Once you'll got a few migrations created, the plugin allows you to move your database back and forth between them. For example, the redo command rolls back a number or migrations and then reapplies them. The undo command just rolls them back. Both of these methods default to one step but for example the command zf redo migration 2 will rollback and reapply the last two migrations that were applied. The method clear rolls back all migrations. The current method shows you which migration was the last to be executed.

Lastly, notice that each method has an env parameter that defaults to development; you use this to set which database to execute the migrations on and should be an environment from your application.ini file (but that doesn't mean I recommend executing the migrations against your production database - this is a development tool, not a deployment tool).

Future developments


Though the database migration plugin is functional, it is far from feature complete. There are a number of improvements I'd like to make.

  • I've played around with using milliseconds for the timestamps (to reduce the chance of getting two migration files with the same timestamp) but this doesn't work reliably on 32bit operating systems because the integer overflows.
  • I'd like to add the pretend mode mentioned earlier.
  • I'd also like to make the database adapter object available to the migration object so that it can interact with the database directly if it needs to.
  • In a team environment, it is likely that more than one team member will be adding migration files and when it comes time to merge working copies, you might end up with missed migrations - migrations that have not been executed but that are older than your own migrations that have been executed. I'd like to add a new method to the plugin to automatically apply missed migrations.
  • Arguably the most impressive feature in the Rails migration system is the domain specific language used to script the changes to the database; rather than use SQL directly, in Rails you use a database agnostic language that builds at runtime SQL statements that are specific to the database that you are using. This means that you can use one RDBMS on your development machine (say, SQLite) and another in your staging environment (say MySQL). Creating this system would obviously be an enormous undertaking (perhaps 90% effort for 10% functionality) but it would be nice to have.

Feedback


If you find any bugs in the plugin, or have any suggestions for improvements, I'd love to hear from you.

Saturday 18 September 2010

Fat Models and Thin Controllers with the Zend Framework

In this article I'm going to discuss what I call Fat Models and how to create them using the Zend Framework. I'll also introduce an abstract base class that wraps up the idea in a reusable way. I'll finish up with a simple example showing the full MVC setup.

The idea of Fat Models is where, in the Model View and Controller (MVC) pattern, to put the code that validates whether the model object can be saved to the database. When a model object has been created or edited, usually via a form, the input is passed through one or more validation objects and only if these validations pass can the object be saved to the database.

Usually this validation code sits in the controller, either in the action method or in a private helper method. The drawback (for me anyway) is that this ties the model directly to the controller and makes testing the behaviour of the model more challenging. It also means that if a programming error is made in the controller, an invalid object might get saved to the database. We call this a fat controller, because it contains a lot of code.

In the Fat Model paradigm, the object maintains it's own validation code. The thin controller simply pumps whatever data it has into the model object and then asks the model object if it can be saved. If the validation fails, the model object populates a list of error messages, that the controller can use to show to the end user. This logic is held in an abstract class, shown below. I've also put this code into my github repository (http://github.com/michaelhodgins/Mooduino); it is under the GPLv2 licence.

abstract class Mooduino_Model_Abstract extends Zend_Db_Table_Row_Abstract { 
  private $_validation_errors = null; 
  /** 
   * Returns true if the model is valid (it passes the validation rules). 
   * @return boolean 
   */ 
  public function isValid() { 
    if (is_null($this->_validation_errors)) { 
      $errors = $this->validate(); 
      if (is_array($errors)) { 
        $this->_validation_errors = $errors; 
      } else { 
        throw new Exception('Validate method didn\'t return an array.'); 
      } 
    } 
    return count($this->_validation_errors) == 0; 
  } 
  /** 
   * Returns all validation errors if there are any or null if there are none. 
   * @return array[string]string|array[string]array[int]string 
   */ 
  public function getErrors() { 
    if (is_array($this->_validation_errors)) { 
      return $this->_validation_errors; 
    } else { 
      throw new Exception('Model has not been validated'); 
    } 
  } 
  /** 
   * Returns the validation error or array of errors for the given field name. 
   * Be sure that there is an error before calling this method by calling 
   * hasError() first. 
   * @param string $field 
   * @return string|array[int]string 
   */ 
  public function getError($field) { 
    $errors = $this->getErrors(); 
    if (array_key_exists($field, $errors)) { 
      return $this->_validation_errors[$field]; 
    } else { 
      throw new Exception('Field not found'); 
    } 
  } 
  /** 
   * Returns true if the given field name has a validation error or false if 
   * it hasn't. 
   * @param string $field 
   * @return boolean 
   */ 
  public function hasError($field) { 
    return is_array($this->_validation_errors) && array_key_exists($field, $this->_validation_errors); 
  } 
  /** 
   * An implementation of the method should validate the instance of the 
   * implementing class and return an array of error messages. If there are 
   * no errors, an empty array should be returned. 
   * @return array[string]string|array[string]array[int]string 
   */ 
  public abstract function validate(); 
  /** 
   * Overrides the save() method in Zend_Db_Table_Row_Abstract so that it is 
   * only called if $this->isValid() returns true. 
   * @return mixed 
   */ 
  public final function save() { 
    if ($this->isValid()) { 
      return parent::save(); 
    } else { 
      throw new Exception('Model can\'t be saved as it isn\'t valid'); 
    } 
  } 
  /** 
   * Given an Iterator such as a Zend_Form, this method will set any error 
   * messages to the form elements. 
   * @param Iterator $iterator 
   */ 
  public final function discoverErrors(Iterator $iterator) { 
    foreach ($iterator as $element) { 
      if ($element instanceof Iterator) { 
        $this->discoverErrors($element); 
      } elseif ($element instanceof Zend_Form_Element) { 
        $this->discoverError($element); 
      } 
    } 
  } 
  /** 
   * If there is a validation error for the given field, it will be set. 
   * @param Zend_Form_Element $element 
   */ 
  private final function discoverError(Zend_Form_Element $element) { 
    if ($this->hasError($element->getName())) { 
      $element->addErrors($this->getError($element->getName())); 
    } 
  } 
}

The last two methods (one public and one private) are used later by the controller to populate a Zend_Form object with the errors that occurred during validation. The public method is recursive so that it correctly handles fieldsets.

An example model


As you can see, the abstract class extends Zend_Db_Table_Row_Abstract and in order to make use of this class, we'll need two model classes; a Table class and a Row class (you can also set up a Rowset class but I don't tend to do that).

First the Table class; this is the class that describes the database table and it's relationships with other tables in the database. Before we can create a Table class, we need an actual database table. For this example, I'm going to use the following simple table.

CREATE TABLE `mooduino_notes`.`notes` (
 `id` bigint(20) NOT NULL AUTO_INCREMENT,
 `text` longtext NOT NULL,
 `title` varchar(255) NOT NULL,
 `priority` int(11) NOT NULL,
 PRIMARY KEY (`id`),
 UNIQUE KEY `Search` (`title`,`text`(767)),
 KEY `Priority` (`priority`)
);

The Table class is very simple in this example because we only have one table; it simply sets which table in the database to point to and which class to use as the Row class.

class Application_Model_DbTable_Notes extends Zend_Db_Table_Abstract {
  protected $_name = 'notes';
  protected $_rowClass = 'Application_Model_Note';
}

The Row class isn't much more complicated; it implements the abstract validate() method from the base class, and it encapsulates the database table columns names with getters and setters. I do this so that the code in the view doesn't need any knowledge of the database columns. Notice how the validate() method employs the Zend_Validate package to check the three fields in the model.

class Application_Model_Note extends Mooduino_Model_Abstract {
  /**
   * Returns an list of error messages, if there are any generated
   * while validating the note.
   * @return array
   */
  public function validate() {
    $errors = array();
    
    $titleValidator = new Zend_Validate_StringLength(array(
      'min'=>3,
      'max'=>255
    ));
    if (!$titleValidator->isValid($this->getTitle())) {
      $errors['title'] = $titleValidator->getMessages();
    }
    $textValidator = new Zend_Validate_StringLength(array('min'=>1));
    if (!$textValidator->isValid($this->getText())) {
      $errors['text'] = $textValidator->getMessages();
    }
    $priorityValidator = new Zend_Validate_Int();
    if (!$priorityValidator->isValid($this->getPriority())) {
      $errors['priority'] = $priorityValidator->getMessages();
    }
    return $errors;
  }
  /**
   * Returns the id.
   * @return int
   */
  public function getId() {
   return $this->id;
  }
  /**
   * Returns the title.
   * @return string
   */
  public function getTitle() {
    return $this->title;
  }
  /**
   * Sets the title.
   * @param string $title
   */
  public function setTitle($title) {
    $this->title = $title;
  }
  /**
   * Returns the text.
   * @return string
   */
  public function getText() {
    return $this->text;
  }
  /**
   * Sets the text.
   * @param string $text
   */
  public function setText($text) {
    $this->text = $text;
  }
  /**
   * Returns the priority.
   * @return int
   */
  public function getPriority() {
    return $this->priority;
  }
  /**
   * Sets the priority.
   * @param int $priority
   */
  public function setPriority($priority) {
    $this->priority = $priority;
  }
}

The views


We have our model, next our view. I'm going to have three; one that lists existing notes (the index action), one for adding a new note and one for editing an existing note. The index action's view, index.phtml, is as follows. Notice that I'm using Smarty syntax but you don't have to. The script checks that there are notes to display, and if there are, it lists them in a table.

<a href="/notes/new">New Note</a>
{if $notes->count() < 0}
<table>
 <thead>  
  <tr>
    <th>ID</th>
    <th>Title</th>
    <th>Priority</th>
   </tr>
 </thead>
 <tbody>
   {foreach from=$notes item=note}
   <tr>
    <td>{$note->getId()}</td>
    <td>{$note->getTitle()}</td>
    <td>{$note->getPriority()}</td>
    <td><a href="/notes/edit/id/{$note->getId()}">Edit</a></td>
   </tr>
   {/foreach}
 </tbody>
</table>
{else}
 <p>There are no notes at this time.</p>
{/if}

The new and edit action scripts both do the same thing; they render the form that they are passed by the controller. Save the following as new.phtml and edit.phtml in the scripts/notes directory.

{if isset($form)}
  {$form}
{/if}


The thin(ner) controller.


The last part of the example is of course the controller, now without any validation code. You'll notice though that it still sanitizes the user's input in the updateNoteFromRequest() method. I've used the Zend_Filter_StripTags class but you'd do whatever your application requires. In the controller you should notice that the action methods are rather short. Most of the functionality you might otherwise put in the action methods is handled elsewhere, either in private methods (like creating the Zend_Form object) or in the model classes.

class NotesController extends Zend_Controller_Action {

 public function init() {
  /* Initialize action controller here */
 }

 public function indexAction() {
  $table = new Application_Model_DbTable_Notes();
  $this->view->notes = $table->fetchAll();
 }

 public function newAction() {
  $table = new Application_Model_DbTable_Notes();
  $note = $table->fetchNew();

  if ($this->getRequest()->isPost()) {

   $this->updateNoteFromRequest($note, $this->getRequest());

   if ($note->isValid()) {
    $note->save();
    $this->_redirect('/notes/index');
   }
  }
  $this->view->note = $note;
  $this->view->form = $this->notesForm($note);
 }

 public function editAction() {
  $id = $this->getRequest()->getParam('id', 0);
  if ($id == 0) {
   $this->_redirect('/notes/index');
   exit;
  }
  $table = new Application_Model_DbTable_Notes();
  $select = $table->select()
    ->where('id = ?', $id);
  $note = $table->fetchRow($select);
  
  if ($this->getRequest()->isPost()) {

   $this->updateNoteFromRequest($note, $this->getRequest());

   if ($note->isValid()) {
    $note->save();
    $this->_redirect('/notes/index');
   }
  }
  $this->view->note = $note;
  $this->view->form = $this->notesForm($note);
 }

 /**
  * Takes the data from the request object and puts it into the note object.
  * @param Application_Model_Note $note
  * @param Zend_Controller_Request_Abstract $request
  */
 private function updateNoteFromRequest(Application_Model_Note $note, Zend_Controller_Request_Abstract $request) {
  $filter = new Zend_Filter_StripTags();

  $note->setTitle($filter->filter($request->getParam('title', '')));
  $note->setText($filter->filter($request->getParam('text', '')));
  $note->setPriority($filter->filter($request->getParam('priority', '')));
 }

 /**
  * Creates a form for the given note object.
  * @param Application_Model_Note $note
  * @return Zend_Form
  */
 private function notesForm(Application_Model_Note $note = null) {
  $form = new Zend_Form();

  $title = new Zend_Form_Element_Text('title', array('label' => 'Title'));
  $form->addElement($title);

  $text = new Zend_Form_Element_Textarea('text', array('label' => 'Text'));
  $form->addElement($text);

  $priority = new Zend_Form_Element_Text('priority', array('label' => 'Priority'));
  $form->addElement($priority);

  $submit = new Zend_Form_Element_Submit('submit', array('value' => 'Save'));
  $form->addElement($submit);

  if (!is_null($note)) {
   $title->setValue($note->getTitle());
   $text->setValue($note->getText());
   $priority->setValue($note->getPriority());

   $note->discoverErrors($form);
  }

  return $form;
 }

}

Saturday 31 July 2010

Using the PHP Smarty Template Engine with the Zend Framework

This post is about how to use the Smarty Templating Engine together with the Zend Framework. I'm not going to discuss why you'd want to do this; I'm only going to show how easy this is to do.

There are a number of articles already on the web showing how to use Smarty and Zend Framework (for example, this one by Andrea Belvedere and this one on Zend's devzone) but they usually have two problems

  1. They are usually quite old and out of date.
  2. They don't cover using both Zend_View and Zend_Layout.
In this article, I've going to correct these two points. This article uses Zend Framework 1.10 and shows how to use Smarty for both the view and the layout.

Setting up


I'm going to assume that you're already comfortable setting up a new Zend Framework project using the Zend Tool, zf. If you're not, here is a good video showing how to do this in Netbeans 6.9. I'm also going to assume that you already have a working PHP development environment.

First thing is to download the Smarty library and copy it into the library directory in your project. When I did this, I renamed the Smarty directory to remove the version number, so that the Smarty code is actually in 'library/Smarty/libs/'.

For Smarty to work, it needs two additional directories, 'cache' and 'templates_c', on the top level of your project (on the same level as the directories called application, library and so on). These directories need to be readable and writeable to by PHP.

Next, we need to put some configuration values into the application's config file. In Zend Framework 1.10, this is a file call application.ini in the 'application/config' directory. Paste the following under [production]
smarty.dir = APPLICATION_PATH "/../library/Smarty/libs/"
smarty.template_dir = APPLICATION_PATH "/views/scripts/"
smarty.compile_dir = APPLICATION_PATH "/../templates_c"
smarty.config_dir = APPLICATION_PATH "/configs"
smarty.cache_dir = APPLICATION_PATH "/../cache"
smarty.caching = 0
smarty.compile_check = true 
These lines will be used to tell the Smarty engine where to find things. In the .ini file, APPLICATION_PATH refers to the directory called application, where your controllers, views and models reside, so 'APPLICATION_PATH "/../templates_c"' means start at that directory, go up one level, and then find a directory called 'templates_c'.

Bootstrap and Zend_View


In older versions of the Zend Framework, it was normal to put bootstrapping code into the index.php file in the public directory but this is no longer the recommended approach. In Zend Framework 1.10, your bootstrapping code goes into the class Bootstrap in the application directory. We need to override the _initView() method in the otherwise empty Bootstrap class.
protected function _initView() {
 require_once 'Smarty_View.php';
 $view = new Smarty_View($this->getOption('smarty'));
 $viewRender = Zend_Controller_Action_HelperBroker::getStaticHelper(
  'ViewRenderer'
 );
 $viewRender->setView($view);
 $viewRender->setViewSuffix('phtml');
 Zend_Controller_Action_HelperBroker::addHelper($viewRender);
 return $view;
}
This method simply changes the class of object used to represent the application's view. Normally, it would be Zend_View, but we're saying use something called Smarty_View instead. The values that we've already put into the application.ini file are put into Smarty_View's constructor.

Smarty_View extends Zend_View_Abstract; there are many versions of this on the web (including in the official Zend Framework documentation) but this version is based on Andrea Belvedere's version.
<?php

class Smarty_View extends Zend_View_Abstract {

  private $_smarty;

  public function __construct($data) {
    parent::__construct($data);
    require_once $data['dir'] . "Smarty.class.php";

    $this->_smarty = new Smarty();
    $this->_smarty->template_dir = $data['template_dir'];
    $this->_smarty->compile_dir = $data['compile_dir'];
    $this->_smarty->config_dir = $data['config_dir'];
    $this->_smarty->cache_dir = $data['cache_dir'];
    $this->_smarty->caching = $data['caching'];
    $this->_smarty->compile_check = $data['compile_check'];
  }

  public function getEngine() {
    return $this->_smarty;
  }

  public function __set($key, $val) {
    $this->_smarty->assign($key, $val);
  }

  public function __get($key) {
    return $this->_smarty->get_template_vars($key);
  }

  public function __isset($key) {
    return $this->_smarty->get_template_vars($key) != null;
  }

  public function __unset($key) {
    $this->_smarty->clear_assign($key);
  }

  public function assign($spec, $value=null) {
    if (is_array($spec)) {
      $this->_smarty->assign($spec);
      return;
    }
    $this->_smarty->assign($spec, $value);
  }

  public function clearVars() {
    $this->_smarty->clear_all_assign();
  }

  public function render($name) {
    return $this->_smarty->fetch(strtolower($name));
  }

  public function _run() {
    
  }

}
This class simply maps the Zend_View_Abstract methods to the Smarty equivalents and sets up up the Smarty engine. We can test this set up with the following code. Create an action with the following code
$this->view->entries = array('moo', 'dave', 'fred', 'andy', 'jo');
$this->view->moo = 'Moo';
and a view template with the code
{$moo|strtoupper} says:

<ol>
 {foreach from=$entries item=entry}
 <li class="{cycle values="odd,even"}">{$entry} - {$entry|strlen}</li>
 {/foreach}
</ol>

Zend_Layout


The previous set up is all that is needed to start using Smarty with Zend_View.Unfortunately, if you're also using Zend_Layout, this will now be broken. If you're not using Zend_Layout, 'zf enable layout' is the Zend Tool command to switch this feature on.

There are two problems we need to overcome. Firstly, Smarty can't find the layout.phtml file. This is because we've told Smarty to look for template files in 'application/views/scripts/' but the layout template is in 'application/layouts/scripts/'. This is actually very easy to correct. A little documented feature of Smarty is that you can give it an array of directories to search, and it will search each in turn until it finds the correct file.

Add the following configuration to your application.ini file.
smarty.layout_dir = APPLICATION_PATH "/layouts/scripts"
Then change the following line in Smarty_View from
$this->_smarty->template_dir = $data['template_dir'];
to
$this->_smarty->template_dir = array($data['template_dir'], $data['layout_dir']);
The second issue to resolve is a small conflict between the way Zend_View and Smarty are implemented; They both expect to execute the view template inside their own scope. In vanilla layout template, the keyword $this would point to the Zend_View object, but now it must point to the Smarty engine. This means that we can't use the normal syntax to access the Zend_Layout object in the layout template.

The fix for this is, again, quite simple. We simply need to add the Zend_View and Zend_Layout objects to the Smarty object as template variables with the following two lines in the Smarty_View constructor.
$this->assign('_view', $this);
$this->assign('_layout', $this->layout());
Now, in the layout template, where we would normally use
<?php $this->layout()->content ?>
we instead use
{$_layout->content}
and where we might previously use
<?php $this->headLink() ?>
we use instead
{$_view->headLink()}

That's all there is to it. You now can use the Smarty template engine while still being able to use Zend_View, Zend_Layout and helpers.

Lastly, notice the line 'smarty.caching = 0' in the config file. This switches off Smarty's static caching system which saves a copy of the static HTML output into the cache directory; this is fine for a development environment but you might want to consider turning it on in production.

Saturday 24 July 2010

Using jQuery and JSONP to load Twitter statuses

In this post I wanted to quickly share how to use jQuery to use Twitter's API to load tweets into a HTML page. This uses AJAX but importantly, it doesn't use any server side code like PHP to load the tweets. All the processing is done on the client's browser which communicates directly with the twitter server. This is possible because Twitter's API supports JSONP, an extension to JSON.

Wikipedia has a good discussion of JSONP but basically, it allows a page to request data from a server other than the one that originally served the page; this isn't usually allowed because of the 'same origin policy' that states that a dynamic element within a browser, such as JavaScript or Flash, can only communicate back to the server that it was loaded from. JSONP gets around this by injecting the JavaScript request into a script tag.

The following code loads my latest tweets into the page. Notice the call to jQuery's getJSON() method, and in particular the first argument. This is the address of the Twitter API for loading my tweets. If you want to load your own tweets, obviously swap 'lampmichael' for your own twitter handle. Also notice 'callback=?' at the end. This is the magic sauce. jQuery will replace the question mark with a dynamically generated function name and when the Twitter service responds to this request, it will call that function which in turn calls the anonymous function that is passed as the second argument to getJSON().

In the anonymous function, the tweets are simply dumped into the body tag within an unordered list. Each tweet has quite a lot of data. In the sample code I've left a commented out debugger statement. Remove the two slashes and load the page in Firefox with Firebug installed to see exactly what Twitter sends back.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en-gb" lang="en-gb" dir="ltr" > 
<head> 
 <meta http-equiv="content-type" content="text/html; charset=utf-8" />
  <title>Twitter JSONP</title>
  <script src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js" type="text/javascript"></script>
  <script type="text/javascript"><!--
  $(document).ready(function () {
    $.getJSON(
      'http://twitter.com/status/user_timeline/lampmichael.json?count=10&callback=?',
      function(data) {
        if (data.length>0) {
          $('body').html('<ol id="tweets"></ol>');
    //debugger;
          data.forEach(function(datum) {
            $('#tweets').append('<li>'+datum.text+' ('+datum.source+')</li>');
          });
        }
      }
    );
  });
  //--></script>
  <style type="text/css">
  #tweets {
    max-width: 400px;
    list-style:none
  }
  #tweets li {
 margin-bottom: 10px;
 padding: 2px
  }
  #tweets li:nth-child(odd) {
   background-color:#EEE
  }
  </style>
</head>
<body>
</body>
</html>

As this is just HTML and JavaScript, save the code into a plain old HTML file and load it in your browser right off your desktop; there's no need to serve it from IIS or Apache.






Monday 26 April 2010

Wikipedia API User Agent String in PHP and cURL

We run a site that pulls data from wikipedia.org and recently the site stopped working. The site was using the following code to interact with the wikipedia.org API; this code queries the API to see if a page with the given title exists.
$url = sprintf('http://en.wikipedia.org/w/api.php?action=query&titles=%s&prop=info&format=json', urlencode($search));
$f = fopen($url, 'r');
$res = '';
while (!feof($f)) {
 $res .= fgets($f);
}
require_once 'Zend/Json.php';
$val = Zend_Json::decode($res);
Once this has executed, $val is an array with the response details.
The problem we started to encounter was that this code started to throw a 403 HTTP status error. The 403 status code means access is denied.
A quick investigation turned up the following page meta.wikimedia.org/wiki/User-Agent_policy which details how, in order to use the API, you now need to pass a User Agent string along with the request. Requests without the User Agent string are refused. User Agent strings are sent by requests from browsers and are used to describe the software that is making the request.
The problem was that fopen() doesn't send a User Agent string and can't be used to do so.
This is where cURL comes in (www.php.net/manual/en/intro.curl.php). cURL is a library for communicating over various internet protocols and allows you to set headers in requests. The same code above, rewritten to use cURL, is as follows:
$url = sprintf('http://en.wikipedia.org/w/api.php?action=query&titles=%s&prop=info&format=json', urlencode($search));
$ch=curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'your website address or app name'); 
$res = curl_exec($ch);
curl_close($ch);
require_once 'Zend/Json.php';
$val = Zend_Json::decode($res);
This small change was all that was needed to appease wikipedia's User Agent requirement.
Enhanced by Zemanta

Monday 29 March 2010

Processing in Netbeans

If you've ever browsed the Processing learning section you may have seen the article showing how to use Processing from within Eclipse.
How, I'm more of a Netbeans fan myself. This isn't a criticism of Eclipse, which I also use extensively. I just like Netbeans. Therefore, I'd like to share how to get Processing working from within the Netbeans IDE. I'll assume that you already have an installation of the Processing IDE. This howto is based almost directly on the Eclipse tutorial, so differences should be obvious.

Step 1. Download and install Netbeans

Netbeans can be downloaded from netbeans.org. Any of the bundles that contains Java SE will do. I use the 'All' bundle but you can get away with a smaller one.

Step 2. Create a new project

Ctrl-Shift-N creates a new project. For the project type, select Java >> Java Application. On the next page of the New Project dialog, give your project a name; I used 'ProcessingTest'. Also tick 'Create Main class' and enter a new like 'uk.co.mooduino.processingtest.Main'. You should use your own domain (this style of using a backward domain to create a unique namespace is common within java).

Step 3. Import the Processing library

To tell Netbeans about how Processing works, we need to import the Processing core library. Expand the 'ProcessingTest' project and then expend 'Libraries'. There should be one entry already, for the JDK. Right-click on the Libraries icon and select 'Add JAR/Folder...'. Browse wo where you installed the Processing IDE and then descend into the lib folder. Select 'core.jar'.

Step 4. Create a class and write your code!

Ctrl-N opens the New File dialog. Select 'Java Class' and give the class a name like 'MyProcessingSketch'. I'm going to use the same code as the Eclipse example;
package uk.co.mooduino.processingtest;

import processing.core.*;

public class MyProcessingSketch extends PApplet {

  public void setup() {
    size(200,200);
    background(0);
  }

  public void draw() {
    stroke(255);
    if (mousePressed) {
      line(mouseX,mouseY,pmouseX,pmouseY);
    }
  }
}

Step 5. Run!

Before we can run this code, we need to modify the Main class that Netbeans already created. Modify the code as follows:
package uk.co.mooduino.processingtest;

import processing.core.PApplet;

public class Main {

    public static void main(String[] args) {
        PApplet.main(new String[] {"--present", "uk.co.mooduino.processingtest.MyProcessingSketch"});
    }
}
Press F6 to execute the code. Your screen should switch to Processing running in fullscreen mode and the middle section should allow you to draw on it.



Monday 22 March 2010

Design patterns in Processing applications

This is a follow on to my previous two posts (here and here). In those post I constructed a simple circuit for measuring the ambient temperature and, with the help of an Arduino board, sending the temperature data down a USB cable. I also wrote a program in Processing to read that data and display it on my PC's screen. I program also stores the temperature data in a simple text file in the hopes that I'll find something interesting to do with it.
As I mentioned at the end of the last post, I was unhappy with the quality of the Processing code. It was somewhat spaghetti like, with three distinct functions all being handled by the same bundle of code. So, in this post, I'll discuss how I teased the code apart and in particularly how I used two design patterns.

Design Patterns are easy

Processing as a language makes it really easy to write quick and dirty code, as we've seen, but this doesn't mean you have to. To tidy my code up, I used two design pattterns; singleton and observer. These are two very easy design patterns to use in Java; the observer pattern is built in!

Singleton

The singleton pattern is simple; A class that is a singleton ensures that there is only on instance of that class and also provides a method to get a reference to the single instance. The Singleton pattern is somewhat notorious in Java but it needn't be as it only requires a couple of lines of code to implement. I've used Bill Pugh's solution which is the neatest.
First of all, in the Processing environment, click that right arrow in the tabbed section and select 'new tab' (or Ctl-Shift N). Give the tab the name 'TemperatureObservable' which is also the name of the class. The follow code implements the singleton pattern.
static class TemperatureObservable {
  private TemperatureObservable() {
    super();
  }
  
  private static class SingletonHolder { 
     private static final TemperatureObservable INSTANCE = new TemperatureObservable();
  }
  
  public static TemperatureObservable getInstance() {
    return SingletonHolder.INSTANCE;
  }
}
There are four things to notice.
  • The class is declared as static. Processing complains about the getInstance() method if it isn't.
  • The class's constructor is private. This ensures that only this class can create an instance of it.
  • There is a private static inner class, SingletonHolder, that creates an instance of the singleton class. The inner class is used so that the singleton instance is only constructed when it is needed. This is called lazy initialization and ensures that the singleton isn't constructed if it isn't going to be used.
  • The static method getInstance() returns the singleton; this is the only way that code outside of this class can gain access to the singleton.

Observer

This pattern is built into Java. It consists of two parts. Firstly, there is an object that we call the observable; this is an object that is expected to change as certain points. Other objects, called observers, register their interest in the observable; when the observable object's state changes, it notifies it's observers.
In this application, the TemperatureObservable objects, our singleton, is also the observable object. Java provides a Class called Observable and any classes that extend this base class automatically get the Observer pattern functionality. Of course, we need to pull in the code from TempDisplay that this object taking over; the TemperatureObservable's job will be to read the temperature data from the serial connection and notify it's observers that a new reading is available.
static class TemperatureObservable extends Observable {
  
  private static PApplet pApplet = null;
  private Serial port = null;
  private String temperature = "0";
  
  private TemperatureObservable() {
    super();
  }
  
  public static void setParent(PApplet pApplet) {
    TemperatureObservable.pApplet = pApplet;
  }
  
  public void addObserver(Observer o) {
    super.addObserver(o);
    if (this.countObservers() == 1) {
      this.setup();
    }
  }
  
  public void deleteObserver(Observer o) {
    super.deleteObserver(o);
    if (this.countObservers() == 0) {
      this.tearDown();
    }
  }
  
  public void deleteObservers() {
    super.deleteObservers();
    this.tearDown();
  }
  
  public void setup() {
    if (this.port == null) {
      String portName = Serial.list()[0];
      this.port = new Serial(TemperatureObservable.pApplet, portName, 9600);
      this.port.write("get temp");
      this.tick();
    }
  }
  
  public void tearDown() {
    if (this.port != null) {
      this.port.stop();
      this.port = null;
    }
  }
  
  public void tick() {
    if (this.port.available()>0) {
      TemperatureObservable.pApplet.delay(100);
      this.temperature = port.readString().trim();
      this.setChanged();
      this.notifyObservers(this.temperature);
    }
  }
  
  public String getTemperature() {
    return this.temperature;
  }
  
   private static class SingletonHolder { 
     private static final TemperatureObservable INSTANCE = new TemperatureObservable();
   }
  
  public static TemperatureObservable getInstance() {
    return SingletonHolder.INSTANCE;
  }
  
}
The new code allows a PApplet object to be registered with the TemperatureObservable class; this is only needed because the Serial object constructor expects a PApplet object as it's first argument. Further on, we also use the PApplet's delay() method. PApplet, by the way, is the superclass of the class you enter into the Processing sketch's first tab. How PApplet child class is called TempDisplay.
addObserver(), deleteObserver() and deleteObservers() are all methods in the Observable class that we override so that we can set up and tear down the serial connection on demand. Lastly, the tick() method is needed to tell the TemperatureObservable object to check the serial connection for a new temperature; remember that in the previous version, this check happened in the PApplet's draw() method.
In order to use the Observable singleton, we need an Observer. Java provides an Observer interface which has just one abstract method that we need to implement, update(). Objects that implement this interface can be passed to the Observable object's addObserver(). Our first observer is quite a simple one and is charged with maintaining the main TempDisplay object's copy of the current temperature (the one that it renders each time draw() is called).
static class TempDisplayObserver implements Observer {
  
  private TempDisplay display = null;
  
  public TempDisplayObserver(TempDisplay display) {
    this.display = display;
  }
  
  public void update(Observable o, Object arg) {
    this.display.setTemperature((String)arg);
  }
}
For this to work, the TempDisplay class needs a new method, setTemperature(). The TempDisplay code is as follows. Notice that the is much shorter than the version at the end last post, though I've also removed the code that saves the data to a file and we've yet to replace this.
import processing.serial.*;

String temperature = "0";
PFont font;


TemperatureObservable tempObs = null;

void setup() {
  TemperatureObservable.setParent(this);
  this.tempObs = TemperatureObservable.getInstance();
  this.tempObs.addObserver(new TempDisplayObserver(this));

  font = loadFont("Ziggurat-HTF-Black-32.vlw");
  textFont(font);
  textAlign(CENTER);
  size(200, 140);
  background(0);
  fill(0);
  smooth();
}

void draw() {
  background(255);
  this.tempObs.tick();
  text(temperature, width/2, height/2);
}

void setTemperature(String arg) {
  temperature = arg;
}
As you can see, at the start of the setup() method, we register the TempDisplay object with the TemperatureObservable class via it's setParent() method. we then get a reference to the singleton and add a new instance of TempDisplayObserver. Also notice in the draw() method, we call the singleton's tick() method.

File output again.

The second observer object is tasked with writing the data to the text file. Most of this code is simply copied from the previous version of TempDisplay.
static class TempFileObserver implements Observer {
  
  private String dataFolder = null;
  private PrintWriter output = null;
  private TimeZone tz = null;
  private DateFormat stamp = null;
  private Date lastSaveDate = null;
  
  public TempFileObserver(String dataFolder) {
    this.dataFolder = dataFolder;
    tz = TimeZone.getDefault();
    stamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
    stamp.setTimeZone(tz);
    lastSaveDate = new Date();
    String fileName = getFileName(lastSaveDate);
    output = getOutput(dataFolder, fileName);
  }
  
  public void update(Observable o, Object arg) {
    this.writeData((String)arg);
  }
  
  private void writeData(String temp) {
    if (this.output != null) {
      Date currentSaveDate = new Date();
      String message = stamp.format(currentSaveDate) + " " + temp;
      this.output.println(message);
      this.output.flush();
      if (this.isNextDay(this.lastSaveDate, currentSaveDate)) {
        this.output.flush();
        this.output.close();
        String fileName = this.getFileName(currentSaveDate);
        this.output = this.getOutput(dataFolder, fileName);
      }
      this.lastSaveDate = currentSaveDate;
    }
  }
  
  private String getFileName(Date date) {
      DateFormat dfm = new SimpleDateFormat("yyyyMMdd");
      dfm.setTimeZone(tz);
      return dfm.format(date) + ".data";
  }
  
  private PrintWriter getOutput(String folder, String file) {
    PrintWriter pw = null;
    File fileHandle = new File(folder, file);
    try {
      pw = new PrintWriter(new FileOutputStream(fileHandle, true));
    } catch (FileNotFoundException fnfe) {}
    return pw;
  }
  
  private boolean isNextDay(Date earlier, Date later) {
    boolean isNextDay = false;
    Calendar cEarlier = Calendar.getInstance();
    Calendar cLater = Calendar.getInstance();
    cEarlier.setTime(earlier);
    cLater.setTime(later);
    if (cLater.after(cEarlier)) {
      boolean dayIsAfter = cLater.get(Calendar.DAY_OF_YEAR) > cEarlier.get(Calendar.DAY_OF_YEAR);
      boolean yearIsAfter = cLater.get(Calendar.YEAR) > cEarlier.get(Calendar.YEAR);
      isNextDay = dayIsAfter || yearIsAfter;
    }
    return isNextDay;
  }   
}
To make use of this class, we just need to pass an instance of it to the singleton's addObserver() method. The TempDisplay's setup() method becomes:
void setup() {
  TemperatureObservable.setParent(this);
  this.tempObs = TemperatureObservable.getInstance();
  
  String dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
  if (dataFolder != null) {
    this.tempObs.addObserver(new TempFileObserver(dataFolder));
  }
  
  this.tempObs.addObserver(new TempDisplayObserver(this));
  //...
}

One last thing.

There is just one last thing that was bugging me; just occasionally, more than one temperature reading is available in the serial connection and this would cause the display to show more than one reading. This is easily fixed. We know that each reading should contain no white space and that individual reading are separated by a line break. Therefore, I changed the TemperatureObservable's tick method to split the value that it reads from the serial connection using a regular expression. The regular expression that I used is '[\\s]+'. This simply means 'one or more characters of white space.' The tick() method changes to the following:
public void tick() {
    if (this.port.available()>0) {
      TemperatureObservable.pApplet.delay(100);
      String outString = port.readString().trim();
      String[] temps = outString.split("[\\s]+");
      for (int i = 0; i < temps.length; i++) {
        this.temperature = temps[i];
        this.setChanged();
        this.notifyObservers(this.temperature);
      }
    }
  }

Conclusion

I was really pleased with the work I did here. Implementing these two design patterns was straight forward. It is just great that Processing allows me to simple programmes quickly while being able to leverage my Java knowledge. Maybe next time I'll do something more visually exciting.



Sunday 7 March 2010

File output with Processing

In my last post, I discussed how I put together a simple temperature probe, using an Arduino board, how the Arduino pushed the current temperature down a USB cable and finally how I consumed that data with a Processing application.
In this article I wanted to share my next steps. I want to be able to use the data from the sensor in a number of ways, not just in Processing, but in any language or platform. Therefore this week I investigated how to write the data to a permanent file on my hard drive. My initial plan had been to write an original program in Java that read the data from the serial port but I soon discovered that getting Java to talk to the serial port (at least on my Linux PC) wasn't exactly easy. Obviously it is possible because Processing is written in Java and this works find on my PC and so to save pulling my hair out, I decided at this time to continue with the Processing application that I started last time.

The task

I set my self the following tasks that I'd like the application to do.
  • The application must ask the user at runtime where on the hard drive the data files are to be stored.
  • The data file must be simple text that any file can read.
  • The data file mustn't individually get to large. Ideally, the application should generated one file per day, and automatically 'roll over' at the start of each day.
  • The data should be appended at the end of the data file if the file already exists.

The Code

First, I looked at how to choose a folder. Processing conveniently provides a function called selectFolder() that prompts the user to select a directory, so this was my starting point. There is also a function called createWriter() that is used to open a PrintWriter object pointing at a given file name.
String dataFolder;
PrintWriter output;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  String fileName = "/" + year() + month() + day() + ".data";
  output = createWriter(dataFolder + fileName);
 }
}
The only issue with this code is the use of month() and day(); These return an integer value, meaning that on the day that I write this, the newly created file would be called '201037.data' when of course it should be called 20100307.data'. We could easily fix this by checking whether each value is less than 10 and adding the 0 if necessary but there is a more flexible way.
Processing allows us to make use of the normal Java classes that are part of the JVM. In this instance, the class that I want to use is called DateFormat and this allows you to take a Date object and create a formated string based on it. In Java, a Date object records a moment in time and by default when you instantiate a new Date object, it records the moment that it was created.
String dataFolder;
PrintWriter output;
TimeZone tz;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  tz = TimeZone.getDefault();
  DateFormat dfm = new SimpleDateFormat("yyyyMMdd");
  dfm.setTimeZone(tz);
  String fileName = "/" + dfm.format(new Date()) + ".data";
  output = createWriter(dataFolder + fileName);
 }
}
This code creates the output file with the correctly formatted name.
Next, outputting the data to the file. I wanted each line in the data file to show the moment that the reading was received as well as the actual temperature. To time stamp each reading, I created another instance of DateFormat, this time with the hours, minutes, second and milliseconds at the end. I created a new function, writeData(), which is called from within draw() whenever a new temperature reading is received.
DateFormat stamp;

void setup() {
 stamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
 stamp.setTimeZone(tz);
}

void draw() {
 if (port.available()>0) {
  delay(100);
  temperature = port.readString().trim();
  writeData(temperature);
 }
 background(255);
 text(temperature, width/2, height/2);
}

void writeData(String temp) {
 if (output != null) {
  String message = stamp.format(new Date()) + " " + temp;
  output.println(message);
  output.flush();
 }
}
Notice also that in draw(), there is a call to trim() on the string returned from the serial port. This is to ensure that extra end of line characters are passed through to the data file. output.flush() is required to flush the data to the hard drive; if we didn't do this, the file wouldn't get written until some internal buffer was filled.
Next up, making sure that at the end of each day, the file name 'rolls over' and a new output file is created. In order to do this, we note the current time each time a temperature is recorded. if the current time is one calendar day after the previous time, then a new file has to be created. First, I wrote a function that would determine is one date is one calendar day later than another.
boolean isNextDay(Date earlier, Date later) {
 boolean isNextDay = false;
 Calendar cEarlier = Calendar.getInstance();
 Calendar cLater = Calendar.getInstance();
 cEarlier.setTime(earlier);
 cLater.setTime(later);
 if (cLater.after(cEarlier)) {
  boolean dayIsAfter = cLater.get(Calendar.DAY_OF_YEAR) > cEarlier.get(Calendar.DAY_OF_YEAR);
  boolean yearIsAfter = cLater.get(Calendar.YEAR) > cEarlier.get(Calendar.YEAR);
  isNextDay = dayIsAfter || yearIsAfter;
 }
 return isNextDay;
}
As you can see, this function makes use of another Java class, this time the Calendar class. The way Java handles dates, time and location specific information about date and time is quite confusing at first. Its important to remember that a Date object only records a moment in time. It isn't aware of what this moment means; whether it is a Tuesday or even which year it is. This is because different parts of the world have different names of Tuesday and even different names of the years (we're all used to the Gregorian calendar but this isn't the only calendar in use). Therefore Java gives us the Calendar class. With one of these, you can ask time specific questions and get answers that are correct for the PC that is running your software.
With this function in place, it was easy to update the writeData() function create a new file at the start of each day. I also created two other functions; These are simply refactored code from the setup() function:
Date lastSaveDate;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  tz = TimeZone.getDefault();
  stamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
  stamp.setTimeZone(tz);
  lastSaveDate = new Date();
  String fileName = getFileName(lastSaveDate);
  output = getOutput(dataFolder, fileName);
 }
}

void writeData(String temp) {
 if (output != null) {
  Date currentSaveDate = new Date();
  String message = stamp.format(currentSaveDate) + " " + temp;
  output.println(message);
  output.flush();
  if (isNextDay(lastSaveDate, currentSaveDate)) {
   output.flush();
   output.close();
   String fileName = getFileName(currentSaveDate);
   output = getOutput(dataFolder, fileName);
  }
  lastSaveDate = currentSaveDate;
 }
}

String getFileName(Date date) {
 DateFormat dfm = new SimpleDateFormat("yyyyMMdd");
 dfm.setTimeZone(tz);
 return "/" + dfm.format(date) + ".data";
}

PrintWriter getOutput(String folder, String file) {
 return createWriter(folder + file);
}
With this code in place this last item on my list is to append new data to existing data. At the moment, when the application is started, the call to createWriter() deletes any data already in the target file. It is much better to add the new data to the end of what is already there. Processing doesn't seem to give us any help in this area so I again dived into Java, this time for the File and FileOutputStream classes.
Whereas the createWriter() function takes the absolute file name and returns the PrintWriter object, the new code builds the PrintWriter up in stages. First, we pass the folder and filename to the File constructor. This creates a pointer to a file location but doesn't actually create the file on the hard drive. Then were create an instance of FileOutputStream(). To this we pass the file handle and, importantly, we also pass the boolean true; This tells the FileOutputStream to open in append mode. Lastly, we create the PrintWriter, passing in the FileOutputStream.
These changes are to the getOutput() function declared in the last refactoring. No other changes were required.
PrintWriter getOutput(String folder, String file) {
 PrintWriter pw = null;
 File fileHandle = new File(folder, file);
 try {
  pw = new PrintWriter(new FileOutputStream(fileHandle, true));
 } catch (FileNotFoundException fnfe) {}
 return pw;
}

All together now

Putting this all together, we end up with the following code, which includes the code from the previous article.
String temperature = "0";
PFont font;
Serial port;
String dataFolder;
PrintWriter output;
TimeZone tz;
DateFormat stamp;
Date lastSaveDate;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  tz = TimeZone.getDefault();
  stamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
  stamp.setTimeZone(tz);
  lastSaveDate = new Date();
  String fileName = getFileName(lastSaveDate);
  output = getOutput(dataFolder, fileName);
 }
 String portName = Serial.list()[0];
 port = new Serial(this, portName, 9600);
 port.write("get temp");
 font = loadFont("Ziggurat-HTF-Black-32.vlw");
 textFont(font);
 textAlign(CENTER);
 size(200, 140);
 background(0);
 fill(0);
 smooth();
}

void draw() {
 if (port.available()>0) {
  delay(100);
  temperature = port.readString().trim();
  writeData(temperature);
 }
 background(255);
 text(temperature, width/2, height/2);
}

void writeData(String temp) {
 if (output != null) {
  Date currentSaveDate = new Date();
  String message = stamp.format(currentSaveDate) + " " + temp;
  output.println(message);
  output.flush();
  if (isNextDay(lastSaveDate, currentSaveDate)) {
   output.flush();
   output.close();
   String fileName = getFileName(currentSaveDate);
   output = getOutput(dataFolder, fileName);
  }
  lastSaveDate = currentSaveDate;
 }
}

String getFileName(Date date) {
 DateFormat dfm = new SimpleDateFormat("yyyyMMdd");
 dfm.setTimeZone(tz);
 return dfm.format(date) + ".data";
}

PrintWriter getOutput(String folder, String file) {
 PrintWriter pw = null;
 File fileHandle = new File(folder, file);
 try {
  pw = new PrintWriter(new FileOutputStream(fileHandle, true));
 } catch (FileNotFoundException fnfe) {}
 return pw;
}

boolean isNextDay(Date earlier, Date later) {
 boolean isNextDay = false;
 Calendar cEarlier = Calendar.getInstance();
 Calendar cLater = Calendar.getInstance();
 cEarlier.setTime(earlier);
 cLater.setTime(later);
 if (cLater.after(cEarlier)) {
  boolean dayIsAfter = cLater.get(Calendar.DAY_OF_YEAR) > cEarlier.get(Calendar.DAY_OF_YEAR);
  boolean yearIsAfter = cLater.get(Calendar.YEAR) > cEarlier.get(Calendar.YEAR);
  isNextDay = dayIsAfter || yearIsAfter;
 }
 return isNextDay;
}

Next?

This code is all about saving the data from the temperature sensor to the hard drive so that I can use the data in other ways, and that is what I plan to do next. Two ideas have occurred to me; One suggestion that Hari made about the previous article was this it would be nice to graph the data, and so that's one thing I want to look at. As I'm a web developer to pay the bills, I also want to look at how I can consume the data in PHP to create a website showing the temperature data.
Before this, though, I think the code above desperately needs refactoring to separate out the three concerns of reading the data from the serial port, showing the data on the monitor and recording the data to the file.
So, until next time, stay happy.

Sunday 28 February 2010

Using Processing and Arduino to display room temperature.

I've recently been getting into the Arduino project. For those who don't know, this is an open source project for prototyping electronic hardware. The Arduino board itself contains a microprocessor, which you can program in a version of c/c++ called Wiring; the board also has a USB port making it really easy to connect to your computer and get started. It also makes it easy to connect to the Arduino from other programs or hardware.

I've been playing around with my Arduino board for a while (just making LEDs blink and such) but today I did my first serious application and I'd like to share how I did this. Please remember, I'm just getting started with my Arduino, so this isn't a master class, but the application does everything that I set out to do (just maybe not the best way). I guess I should also point out that this isn't 'getting started with Arduino tutorial' even though it isn't too advanced; I have only maybe 10 hours experience but I'm going to assume that you've already got your Arduino development environment set up. If you haven't, not to worry, there are tons of excellent tutorials on the web, such as this one.

The hardware measures the ambient room temperature, my Arduino board takes the output from the sensor, transforms it to Centigrade, and pushes it down the USB serial port. The final part is a desktop program written the the Processing language, and this reads the value from the USB serial port and displays it in a window.

Hardware

For this project, you'll need the following items:
  • An Arduino board
  • A solderless breadboard
  • A temperature sensor (I used a LM35DT)
  • Wires to connect the various components
  • A USB cable

I got all these part in my initial Arduino box that I purchased from EarthShine Design.
Setting up the hardware is really easy. Plug the temperature sensor into your breadboard so that the three legs are on separate columns. The sensor needs 5 volts to power it. The front of the sensor has the name on and a little dot above leg number one which takes five volts so should be connected to the 5V pin on the Arduino. The middle leg should be connected to the GND pin, next to the 5V pin. The third leg should then be connected to the pin marked ANALOG IN 0, just right of the 5V and GND pins.

Wiring (version 1)

This is the first version of the software that I pushed to the Arduino. Once installed, it pushes the current temperature reading down the serial port every half a second. There are a few parts that it is worth pointing out.
  1. The loop function doesn't use a delay in order to read every half second. Instead, it stores when it last checked the temperature in the variable 'previousMillis' and doesn't check again until the current time is greater than 'previousMillis' plus 'interval' (which is 500 millisecond). I did it this way so that the code can be used alongside other code that also needs to loop; if delay() had been used, that other code would be delayed too.
  2. Wiring's version of sprintf doesn't support floats, so the code has to work out the temperature times 10 (so 30 degrees is stored as 300) and then in printTemperature() we put the period in manually by using the modulus operator.
  3. The last line in the setup() function sets the Arduino's reference voltage to 1.1 volts. This is done because the sensor module's third leg produces up to 1 volt; the sensor component measures from 0 to 100 degrees. 0 degrees is indicated by 0 volts and 100 degrees is indicated by 1 volt. Arduino automatically converts this voltage to a number between 0 and 1023.
const byte potPin = 0;         //pin that the temperature sensor is attached to
long temperature = 0;         //the current temperature
long previousMillis = 0;       // will store last time temperature was updated
const long interval = 500;     // interval at which to check the temperature

void setup() {
 Serial.begin(9600);
 Serial.flush();
 analogReference(INTERNAL);
}

void loop() {
 unsigned long currentMillis = millis();
 if(currentMillis - previousMillis > interval) {
  doTemperatureMeasurement();
  previousMillis = currentMillis; 
 }
}

void printTemperature(int value) {
 byte len = 4;
 if (value == 100) len = 5;
 char buffer[len];
 sprintf(buffer, "%3i.%1i", value / 10, value % 10);
 Serial.println(buffer);
}

int readAverage() {
 int span = 20;
 int aRead = 0;
 for (int i = 0; i < span; i++) {
  aRead = aRead+analogRead(potPin);
 }
 return aRead / span;
}

void doTemperatureMeasurement() {
 int aRead = readAverage();
 temperature = ((100*1.1*aRead)/1024)*10;
 printTemperature(temperature);
}
Once you have this code installed on your Arduino, you can open the serial monitor and should see the current temperature rattled back at you.

Processing (version 1)

With the Arduino now communicating the temperature down the USB serial cable I wanted to display this value on my monitor.
A project related to Arduino and Wiring is Processing. Actually, the Processing environment is virtually the same as the Wiring environment and the language is very similar too. Once you have the Processing environment installed and running, start a new project and save it (I called mine 'TempDisplay').

In order to display the temperature on the screen, we need a special font file that Processing uses to render text; a suitable file ships with Processing. From within Processing, click 'Add File...' on the Sketch menu. Open the directory where you installed Processing and then descend into 'examples/Basics/Typography/Words/data/'. Here you'll find a file called 'Ziggurat-HTF-Black-32.vlw'. Select this file.
import processing.serial.*;

String temperature = "0";
PFont font;
Serial port;

void setup() {
 String portName = Serial.list()[0];
 port = new Serial(this, portName, 9600);
 font = loadFont("Ziggurat-HTF-Black-32.vlw");
 textFont(font);
 textAlign(CENTER);
 size(200, 140);
 background(0);
 fill(0);
 smooth();
}

void draw() {
 if (port.available()>0) {
  delay(100);
  temperature = port.readString();
 }
 background(255); // Set background to dark gray
 text(temperature, width/2, height/2);
}
Before launching the application, you'll need to quit out of the Arduino environment, as the serial communication can't be opened by both programs at the same time.
The only line of code here that might cause a problem is the first line of the function setup(). It assumes that the USB serial connection is the first returned by Serial.list(). If yours isn't, you may have to play around with the number in the square brackets.

Wiring (version 2)

There were two things that I wasn't happy with in this solution.
  1. The Arduino communication is too chatty, by which I mean, the Arduino pushes the temperature down the wire even when it hasn't changed.
  2. When the Processing application starts up, there is a notable delay before it gets an update from the serial port during which time the temperature is displayed as zero.
The first problem is easy to fix. I simply changed the code for 'doTemperatureMeasurement()' to only update the temperature if it has changed since last time it was determined.
void doTemperatureMeasurement() {
 int aRead = readAverage();
 long newTemperature = ((100*1.1*aRead)/1024)*10;
 if (newTemperature != temperature) {
  temperature = newTemperature;
  printTemperature(temperature);
 }
}
This code change has an unfortunate side effect; The Processing application now has an even more noticeable delay when it starts up before it shows the real temperature. This is obviously because it is now waiting not for the half second to pass but for the temperature sensor to read that the room temperature has changed.

Wiring (version 3)

This is the final version of the Arduino software, so I've copied the whole thing again. The only change though is that the software listens (in the loop() function) for a message on the serial port. The message it is listening for is 'get temp' and when it hears this, it pushing the current temperature down the wire regardless of whether it has changed since last time.
Most of the code is to do with handling the character buffer but I've also refactored doTemperatureMeasurement(), separating out the code that determines the actual temperature into a new function, readTemperature(). This is used in the loop() function if the new message is heard on the wire.
const byte potPin = 0;         //pin that the temperature sensor is attached to
long temperature = 0;         //the current temperature
long previousMillis = 0;       // will store last time temperature was updated
const long interval = 500;     // interval at which to check the temperature
const byte maxSerialChars = 8;
char buffer[maxSerialChars];

void setup() {
 Serial.begin(9600);
 Serial.flush();
 analogReference(INTERNAL);
}

void loop() {
 if (Serial.available()>0) {
  delay(100);
  int numChars = Serial.available();
  int index = 0;
  if (numChars > maxSerialChars) {
   numChars = maxSerialChars;
  }
  while (numChars--) {
   buffer[index++] = Serial.read();
  }
  if (buffer[0] == 'g' && buffer[4] == 't') {
   printTemperature(readTemperature());
  }
  for (byte x=0; x<maxSerialChars; x++) {
   buffer[x] = '\0';
  }
 }

 unsigned long currentMillis = millis();
 if(currentMillis - previousMillis > interval) {
  doTemperatureMeasurement();
  previousMillis = currentMillis;
 }
}

void printTemperature(int value) {
 byte len = 4;
 if (value == 100) len = 5;
 char buffer[len];
 sprintf(buffer, "%3i.%1i", value / 10, value % 10);
 Serial.println(buffer);
}

int readAverage() {
 int span = 20;
 int aRead = 0;
 for (int i = 0; i < span; i++) {
  aRead = aRead+analogRead(potPin);
 }
 return aRead / span;
}

long readTemperature() {
 int aRead = readAverage();
 return ((100*1.1*aRead)/1024)*10;
}

void doTemperatureMeasurement() {
 long newTemperature = readTemperature();
 if (newTemperature != temperature) {
  temperature = newTemperature;
  printTemperature(temperature);
 }
}

Processing(version 2)

This this new code in place you can test the special message by opening the serial monitor and typing in 'get temp' but lets update the Processing application to make use of this new feature. Actually, we only need to add one extra line of code to the Processing application! On the line after we open the serial port and before we load the font, add the line:
port.write("get temp");
That's it. The Processing application will now load the temperature almost straight away. Not only will it not have to wait for the thermo sensor to register a temperature change but it also won't have to wait for the half second to pass as it did in the first version.

Next?

So, what's next? I'm not sure where I'll take the this thermo sensor device but a few thoughts occur to me. It would be nice to make the device perminent, perhaps by popping the microprocessor off the Arduino board (you can replace the chip fairly easily) and getting rid of the breadboard too.
It also might be interesting to investigate replacing the USB serial connection with perhaps a bluetooth connection or even a WiFi connection.