Sensor Data To The Cloud: Part 2

OK, data in the cloud (at least if you saw the first post in this series), but for the moment the cloud is being used as WOM (Write Only Memory). What we want now is the ability to get the data out so that we can do something with it.

I plan eventually to be able to access the raw data points, probably using some sort of JSON format served by a web server interface, but for now let's have some fun and look at graphs in a web browser.

You can see the code on github at https://github.com/kmhughes/robotbrains-examples/tree/master/data/cloud/org.robotbrains.data.cloud.timeseries.server.

The current interface is very quick and dirty, but it at least let me start looking at the data. Here is an example of a couple of graphs from my test installation.

I first set up a small web server. I used the one from the Interactive Spaces open source project, though I made some changes to it that I have wanted to make for a while and moved it to using Netty 4 rather than Netty 3, cleaned out some of the deprecated methods, and did some general refactoring. These web server modifications will continue for some time as I get more familiar with the Netty 4 API, and decide on the new internal architecture for the web server. It was written in a rather demand driven fashion and now with 3 years of usage experience, it is definitely time for a cleanup.

The main class for the data web server is StandardDataWebServer. This class creates an instance of the Interactive Spaces web server and attaches a dynamic content GET handler. You can see this in the startup() method.

webServer = new NettyWebServer(webServerPort, log);
webServer.startup();

webServer.addDynamicContentHandler("graph", true, new HttpDynamicRequestHandler() {
  @Override
  public void handle(HttpRequest request, HttpResponse response) {
    handleGraphRequest(request, response);
  }
});

Notice that the URL prefix has been set to graph. Graph URL paths, minus the host and port for the webserver, will be of the form

/graph/source/sensing_unit/sensor?start=start_date&end=end_date

Here is the URL path I used for the temperature graph above.

/graph/keith.test/pi2/temperature?start=2015/11/23@00:00:00&end=2015/11/25@08:00:00

Here you can see that source is keith.test, the sensing_unit is pi2, and the sensor on the sensing unit is temperature.  The values for these fields are set in the Python script running on the Pi and your values will be different.

The date formats are easy enough. Year first, then month, then day. After the @ is the time in a 24 hour clock, so 00:00:00 is midnight.

The method handleGraphRequest() performs the following operations through various methods it calls:

1. It parses the URL and creates the sensor data query with the method getDataQueryFromRequest().
   
   • The URL path components specify the exact sensor the data is wanted for.
   • The URL query parameters specify the start and stop dates for the query range.
2. The sensor data query is handed to the Database Relay (class KairosDbDatabaseRelay) that then queries KairosDB to get the requested data. The query method called is 
   querySensorData().
3. The results of the sensor data query are transformed into the correct components for a JFreeChart data graph and the chart is generated in renderChart().
   1. An XYDataset is created from the sensor data.
   2. A chart is then created from the XYDataset.
4. Finally the JFreeChart chart is rendered as a PNG graphic and sent back to the web browser in writeChartResponse().

The JFreeChart dataset is easily created from the sensor data samples.

private XYDataset createDataset(SensorData data) {
  XYSeries series = new XYSeries("Fun Data");
  for (SensorDataSample sample : data.getSamples()) {
    try {
      series.add(sample.getTimestamp(), sample.getValue());
    } catch (SeriesException e) {
      log.error("Error adding to series graph", e);
    }
  }

  return new XYSeriesCollection(series);
}

The chart is then created.

private JFreeChart createChart(SensorDataQuery query, XYDataset dataset) {
  return ChartFactory.createTimeSeriesChart
      String.format("Sample Data: %s - %s - %s",
          query.getSource(), query.getSensingUnit(), query.getSensor()),
      "Time", "Value", dataset,
      false, false, false);
}

The chart is then finally written to the web browser.

private void writeChartResponse(HttpResponse response, JFreeChart chart) throws IOException {
  BufferedImage chartImage = chart.createBufferedImage(560, 370, null);
  ImageIO.write(chartImage, "png", response.getOutputStream());
  response.setContentType(CommonMimeTypes.MIME_TYPE_IMAGE_PNG);
}

As you can see, the dimensions of the PNG graphic are currently fixed, eventually the height and width will be set from query parameters.

It was very easy to get JFreeChart to visualize my data for me, I look forward to learning more about the API.

The modifications to the class KairosDbDatabaseRelay were pretty simple as the KairosDB Java API is really easy to use. The query code is given below.

String metricName = createEventKey(query.getSource(), 
    query.getSensingUnit(), sensor);

QueryBuilder builder = QueryBuilder.getInstance();
builder.setStart(query.getStartDate().toDate())
    .setEnd(query.getEndDate().toDate());

builder.addMetric(metricName);

QueryResponse response = kairosdbClient.query(builder);
for (Queries queries : response.getQueries()) {
  for (Results results : queries.getResults()) {
    for (DataPoint dataPoint : results.getDataPoints()) {
      SensorDataSample sample =
          new SensorDataSample(sensor, dataPoint.doubleValue(),
              dataPoint.getTimestamp());

      data.addSample(sample);
    }
  }
}

KairosDB requires a metric name for each item it is storing time series data for. The Data Relay uses the source, sensing unit and sensor concatenated together with a spacer character to create its metric names. Eventually the Relay will contain a small database giving the sources, sensing units, and sensors to provide mappings for data types for the data. The metric name generated here will be in metricName.

Next a KairosDB QueryBuilder is created. The start and end dates for the query are then set. The Relay uses the Joda time library (if you are working with time in Java, you must use Joda), but the KairosDB wants Java dates, so the toDate() method is called to convert from Joda DateTime instances to the Java Date class. finally the metric name is added to the query.

The next bit of code has 3 nested for loops. This may seem overkill, but the KairosDB API allows for multiple queries per call to KairosDB, and for the moment we only have the one metric, so have to dig through several data structures to get the data returned from the backend database.

And there you have it. A quick and easy web interface to graph the time series data.

I plan on continuing to expand the Relay to give new functionality. Not every change I make will become a blog post, so you might want to keep an eye on the Github project if you find this interesting and want to be kept aware of the changes. At some point I want to be able to pull the data into the data analysis package R and eventually do machine learning from the data, I will probably end up writing posts about those things.