Sunday 7 March 2010

File output with Processing

In my last post, I discussed how I put together a simple temperature probe, using an Arduino board, how the Arduino pushed the current temperature down a USB cable and finally how I consumed that data with a Processing application.
In this article I wanted to share my next steps. I want to be able to use the data from the sensor in a number of ways, not just in Processing, but in any language or platform. Therefore this week I investigated how to write the data to a permanent file on my hard drive. My initial plan had been to write an original program in Java that read the data from the serial port but I soon discovered that getting Java to talk to the serial port (at least on my Linux PC) wasn't exactly easy. Obviously it is possible because Processing is written in Java and this works find on my PC and so to save pulling my hair out, I decided at this time to continue with the Processing application that I started last time.

The task

I set my self the following tasks that I'd like the application to do.
  • The application must ask the user at runtime where on the hard drive the data files are to be stored.
  • The data file must be simple text that any file can read.
  • The data file mustn't individually get to large. Ideally, the application should generated one file per day, and automatically 'roll over' at the start of each day.
  • The data should be appended at the end of the data file if the file already exists.

The Code

First, I looked at how to choose a folder. Processing conveniently provides a function called selectFolder() that prompts the user to select a directory, so this was my starting point. There is also a function called createWriter() that is used to open a PrintWriter object pointing at a given file name.
String dataFolder;
PrintWriter output;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  String fileName = "/" + year() + month() + day() + ".data";
  output = createWriter(dataFolder + fileName);
 }
}
The only issue with this code is the use of month() and day(); These return an integer value, meaning that on the day that I write this, the newly created file would be called '201037.data' when of course it should be called 20100307.data'. We could easily fix this by checking whether each value is less than 10 and adding the 0 if necessary but there is a more flexible way.
Processing allows us to make use of the normal Java classes that are part of the JVM. In this instance, the class that I want to use is called DateFormat and this allows you to take a Date object and create a formated string based on it. In Java, a Date object records a moment in time and by default when you instantiate a new Date object, it records the moment that it was created.
String dataFolder;
PrintWriter output;
TimeZone tz;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  tz = TimeZone.getDefault();
  DateFormat dfm = new SimpleDateFormat("yyyyMMdd");
  dfm.setTimeZone(tz);
  String fileName = "/" + dfm.format(new Date()) + ".data";
  output = createWriter(dataFolder + fileName);
 }
}
This code creates the output file with the correctly formatted name.
Next, outputting the data to the file. I wanted each line in the data file to show the moment that the reading was received as well as the actual temperature. To time stamp each reading, I created another instance of DateFormat, this time with the hours, minutes, second and milliseconds at the end. I created a new function, writeData(), which is called from within draw() whenever a new temperature reading is received.
DateFormat stamp;

void setup() {
 stamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
 stamp.setTimeZone(tz);
}

void draw() {
 if (port.available()>0) {
  delay(100);
  temperature = port.readString().trim();
  writeData(temperature);
 }
 background(255);
 text(temperature, width/2, height/2);
}

void writeData(String temp) {
 if (output != null) {
  String message = stamp.format(new Date()) + " " + temp;
  output.println(message);
  output.flush();
 }
}
Notice also that in draw(), there is a call to trim() on the string returned from the serial port. This is to ensure that extra end of line characters are passed through to the data file. output.flush() is required to flush the data to the hard drive; if we didn't do this, the file wouldn't get written until some internal buffer was filled.
Next up, making sure that at the end of each day, the file name 'rolls over' and a new output file is created. In order to do this, we note the current time each time a temperature is recorded. if the current time is one calendar day after the previous time, then a new file has to be created. First, I wrote a function that would determine is one date is one calendar day later than another.
boolean isNextDay(Date earlier, Date later) {
 boolean isNextDay = false;
 Calendar cEarlier = Calendar.getInstance();
 Calendar cLater = Calendar.getInstance();
 cEarlier.setTime(earlier);
 cLater.setTime(later);
 if (cLater.after(cEarlier)) {
  boolean dayIsAfter = cLater.get(Calendar.DAY_OF_YEAR) > cEarlier.get(Calendar.DAY_OF_YEAR);
  boolean yearIsAfter = cLater.get(Calendar.YEAR) > cEarlier.get(Calendar.YEAR);
  isNextDay = dayIsAfter || yearIsAfter;
 }
 return isNextDay;
}
As you can see, this function makes use of another Java class, this time the Calendar class. The way Java handles dates, time and location specific information about date and time is quite confusing at first. Its important to remember that a Date object only records a moment in time. It isn't aware of what this moment means; whether it is a Tuesday or even which year it is. This is because different parts of the world have different names of Tuesday and even different names of the years (we're all used to the Gregorian calendar but this isn't the only calendar in use). Therefore Java gives us the Calendar class. With one of these, you can ask time specific questions and get answers that are correct for the PC that is running your software.
With this function in place, it was easy to update the writeData() function create a new file at the start of each day. I also created two other functions; These are simply refactored code from the setup() function:
Date lastSaveDate;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  tz = TimeZone.getDefault();
  stamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
  stamp.setTimeZone(tz);
  lastSaveDate = new Date();
  String fileName = getFileName(lastSaveDate);
  output = getOutput(dataFolder, fileName);
 }
}

void writeData(String temp) {
 if (output != null) {
  Date currentSaveDate = new Date();
  String message = stamp.format(currentSaveDate) + " " + temp;
  output.println(message);
  output.flush();
  if (isNextDay(lastSaveDate, currentSaveDate)) {
   output.flush();
   output.close();
   String fileName = getFileName(currentSaveDate);
   output = getOutput(dataFolder, fileName);
  }
  lastSaveDate = currentSaveDate;
 }
}

String getFileName(Date date) {
 DateFormat dfm = new SimpleDateFormat("yyyyMMdd");
 dfm.setTimeZone(tz);
 return "/" + dfm.format(date) + ".data";
}

PrintWriter getOutput(String folder, String file) {
 return createWriter(folder + file);
}
With this code in place this last item on my list is to append new data to existing data. At the moment, when the application is started, the call to createWriter() deletes any data already in the target file. It is much better to add the new data to the end of what is already there. Processing doesn't seem to give us any help in this area so I again dived into Java, this time for the File and FileOutputStream classes.
Whereas the createWriter() function takes the absolute file name and returns the PrintWriter object, the new code builds the PrintWriter up in stages. First, we pass the folder and filename to the File constructor. This creates a pointer to a file location but doesn't actually create the file on the hard drive. Then were create an instance of FileOutputStream(). To this we pass the file handle and, importantly, we also pass the boolean true; This tells the FileOutputStream to open in append mode. Lastly, we create the PrintWriter, passing in the FileOutputStream.
These changes are to the getOutput() function declared in the last refactoring. No other changes were required.
PrintWriter getOutput(String folder, String file) {
 PrintWriter pw = null;
 File fileHandle = new File(folder, file);
 try {
  pw = new PrintWriter(new FileOutputStream(fileHandle, true));
 } catch (FileNotFoundException fnfe) {}
 return pw;
}

All together now

Putting this all together, we end up with the following code, which includes the code from the previous article.
String temperature = "0";
PFont font;
Serial port;
String dataFolder;
PrintWriter output;
TimeZone tz;
DateFormat stamp;
Date lastSaveDate;

void setup() {
 dataFolder = selectFolder("Choose the folder where you want the temperature data to be recorded");
 if (dataFolder != null) {
  tz = TimeZone.getDefault();
  stamp = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss:SSS");
  stamp.setTimeZone(tz);
  lastSaveDate = new Date();
  String fileName = getFileName(lastSaveDate);
  output = getOutput(dataFolder, fileName);
 }
 String portName = Serial.list()[0];
 port = new Serial(this, portName, 9600);
 port.write("get temp");
 font = loadFont("Ziggurat-HTF-Black-32.vlw");
 textFont(font);
 textAlign(CENTER);
 size(200, 140);
 background(0);
 fill(0);
 smooth();
}

void draw() {
 if (port.available()>0) {
  delay(100);
  temperature = port.readString().trim();
  writeData(temperature);
 }
 background(255);
 text(temperature, width/2, height/2);
}

void writeData(String temp) {
 if (output != null) {
  Date currentSaveDate = new Date();
  String message = stamp.format(currentSaveDate) + " " + temp;
  output.println(message);
  output.flush();
  if (isNextDay(lastSaveDate, currentSaveDate)) {
   output.flush();
   output.close();
   String fileName = getFileName(currentSaveDate);
   output = getOutput(dataFolder, fileName);
  }
  lastSaveDate = currentSaveDate;
 }
}

String getFileName(Date date) {
 DateFormat dfm = new SimpleDateFormat("yyyyMMdd");
 dfm.setTimeZone(tz);
 return dfm.format(date) + ".data";
}

PrintWriter getOutput(String folder, String file) {
 PrintWriter pw = null;
 File fileHandle = new File(folder, file);
 try {
  pw = new PrintWriter(new FileOutputStream(fileHandle, true));
 } catch (FileNotFoundException fnfe) {}
 return pw;
}

boolean isNextDay(Date earlier, Date later) {
 boolean isNextDay = false;
 Calendar cEarlier = Calendar.getInstance();
 Calendar cLater = Calendar.getInstance();
 cEarlier.setTime(earlier);
 cLater.setTime(later);
 if (cLater.after(cEarlier)) {
  boolean dayIsAfter = cLater.get(Calendar.DAY_OF_YEAR) > cEarlier.get(Calendar.DAY_OF_YEAR);
  boolean yearIsAfter = cLater.get(Calendar.YEAR) > cEarlier.get(Calendar.YEAR);
  isNextDay = dayIsAfter || yearIsAfter;
 }
 return isNextDay;
}

Next?

This code is all about saving the data from the temperature sensor to the hard drive so that I can use the data in other ways, and that is what I plan to do next. Two ideas have occurred to me; One suggestion that Hari made about the previous article was this it would be nice to graph the data, and so that's one thing I want to look at. As I'm a web developer to pay the bills, I also want to look at how I can consume the data in PHP to create a website showing the temperature data.
Before this, though, I think the code above desperately needs refactoring to separate out the three concerns of reading the data from the serial port, showing the data on the monitor and recording the data to the file.
So, until next time, stay happy.

4 comments:

  1. hy

    i also did a little project and want to statically (hope i spelled that wright - im no native)
    save it on an a other place

    creaWriter("eg_my_file.txt") saves it just to the sketch folder

    and

    createWriter("C:\blabla\blabla....\myfile.txt");

    gets me an error with the \

    do u have an idea or can u help me ?

    ReplyDelete
    Replies
    1. Hi

      I think you need to escape the backslashes, as follows:

      createWriter("C:\\blabla\\blabla...\\myfile.txt");

      See http://en.wikipedia.org/wiki/Escape_character

      Alternatively, you could try using Unix style file paths, which should work on Windows too, as follows:

      createWriter("C:/blabla/blabla.../myfile.txt");

      I hope this helps.

      Delete
    2. hy again,

      thank you !! - yes it worked

      both styles works fine :)

      Delete