Pages

Sunday, December 13, 2015

Greenhouse Phase 4: Data To The House

The first several articles in the Greenhouse Series were about


Now it would be nice to get the data from the sensors in the Greenhouse into the main house for capture, processing, etc.

To do this, I decided to use Series 2 XBee radios. These powerful little radios are very easy to interface to and have a very nice feature, they support mesh networking. A lot of people look for cheaper radios, XBees are not the cheapest in the world, but these little radios have a whole lot of features and, if you need these features, spending some extra money rather than implementing those features yourself can be the right decision to make.

XBees


Let's start off by talking a little bit about Series 2 XBee radios.

What Is Mesh Networking?


So what is mesh networking? Mesh networking is useful in situations where you have a bunch of radios that need to talk to each other, some are too far away from each other to communicate directly, or perhaps there is a wall in the way that doesn't let the signal thorough. Also, sometimes noise prevents a signal from getting through, or something breaks. So how can we have a reliable network that makes sure that all data is received?

Suppose you have radios A and B, and A wants to talk to B, but B is too far away from A, or there is a wall in the way or something that means that the connection won't happen. One way of handing this is to have a 3rd radio C that is within range of both A and B. A could send its message for B to C and C could then forward the message to B, A -> C -> B.

One issue here is if C stops working for some reason, suddenly A can no longer talk to B.

So suppose we have a radio D that is also reachable by A and by B. It doesn't matter if C can talk to D or not, just that A and B can talk to D. But say C is working. Now there are 2 different routes from A to B, either A -> C -> B, or A -> D -> B. If radio A finds that C isn't working, it can route its message for B through D. Of course, if both C and D are not working for some reason, once again there is no way for A to talk to B.

It is possible for these chains to be longer. Each additional radio in the chain is called a hop. Say radio A can't reach B, but can reach C. C can't reach B, but can reach E. And suppose E can reach B. Now A can send a packet to B via the path A  -> C -> E -> B. This path has 3 hops.

So radios don't necessarily have to be able to talk directly to each other to transmit information to each other, they can use intermediate radios. Not only that, but they needn't use the same path each time. Some radios in a path that used to work may not be available at any given time, but if there is a path of radios who can talk to each other, the packet can eventually get from A to B. This is the basis of mesh networking and is similar to how the Internet works.

Other XBee Concepts


There are a few more concepts useful for understanding XBee radios. Every mesh has to have 1 radio that is the Coordinator. A mesh can have only one Coordinator. This radio controls the network. It determines the ultimate local addresses that each radio in the mesh gets. Radios have 2 addresses, one short one local to the mesh the radio is part of, and a much longer one that is unique to the radio no matter what mesh it is in.

Another radio type is a Router. Router radios can act as intermediaries and radios who cannot talk directly to each other must go through one or more Routers.

Finally there can be End Devices. These End Devices can be battery powered and lower powered. They can go to sleep and turn themselves back on. They can talk to Routers or Coordinators, but cannot talk to each other. This limited functionality means they can be lower cost or can work on battery power for very long periods of time because they can sleep between transmissions.

An example mesh network might look like below.




One more important concept is that of the PAN (Personal Area Network) ID. All XBees must have a PAN ID, and all radios that are going to communicate with each other must have the same PAN ID. The PAN ID is determined by the Coordinator, and can be manually set or automatically created by the Coordinator. Finally, you have have multiple independent XBee networks in the same physical location, each with its own PAN ID. These networks will not be able to communicate with each other, only radios with the same PAN ID can communicate with each other.

In this post we will set the PAN ID manually.

The XBee Hardware


You can use the XBee hardware in several ways.

You can talk to it over a serial connection and control it from a host computer or microcontroller. This style of usage gives you the most flexibility, but as it requires some programming, it is also the most complex usage. The radios have standard RX and TX pins for communication.

It is also possible to wire digital and analog sensors directly to an XBee, eliminating the need for a host computer or microcontroller. In this style of usage, circuitry and programming are minimized. It is also possible then to make the radio sleep for the intervals between sensor samples, which means that a battery powering the XBee could last a very long time The number of sensor pins is limited, but if this is all you need, it is a great option for the simplicity and long battery life.

Configuring the XBees


XBee radios are configured with software from Digi, the manufacturer of XBee radios, called XCTU. When I first started using XBees, XCTU only ran on Windows machines, which meant I had to use virtual machines or Wine on my Linux box. But Digi have since rewritten the program in Java and it is now possible to run it on all major OSes.

The program is really nice now. Not only does it work on all OSes, but it also has a much improved user interface. It also has new functionality that is very useful for debugging, you can visualize your entire XBee network and see which radios can talk to which other radios, transmit data to any radio by typing the data into the UI. All in all, XCTU has become a very nice tool.

So jump into your favorite search engine and search for "digi xctu download" and pick the version appropriate for your computer and follow the directions for installation.

My Initial Network


The wireless network I build will eventually cover both the greenhouse and the house itself, but this post is about getting data from the greenhouse into the house. To do this I will use 3 XBees. Two will be XBee Pros. These radios have a lot more output power and I figured they would be good for getting through the outside walls of the greenhouse and the house and cover the distance between the 2 buildings. They might also be overkill, but I thought I would give them a try. The Coordinator Radio will be a standard XBee in the house and doesn't have to be as high powered.

Hardware for Configuring


It is necessary to have a serial connection to the radios to program them. To help with this, I bought several Sparkfun XBee Explorers. These little boards are awesome. You plug the radio into the board, hook up a USB cable and you are working with them in no time. I often hook the Coordinator Radio up to a host computer using an Explorer in production.

The Radios are blue and the Explorers are red in the picture below. You may also notice tape on the antenna for the radios. I usually use tape to mark the PAN ID and whether the radio is a Coordinator, Router, or End Device when building a network so I know which is which.






Running XCTU


Running XCTU is easy. If you follow the directions on the Digi site on Linux, it will install the software in /opt/Digi/XCTU-NG. I have not yet puzzled through how to set permissions so that I can use XCTU without running as root, for now I use the command

sudo /opt/Digi/XCTU-NG/app &

This will run the program as root and run it in the background so that I can do other things in the same terminal window.

The initial window you should see is the following.




Now we need to scan for the radios. First I want to program the Coordinator radio, so I labeled the Coordinator Radio with tape that had C written on it, plugged it carefully into the Sparkfun USB Explorer, and then plugged the Explorer into my computer.

If you look at the initial XCTU window, you will see an image kinda shaped like an XBee with a magnifying glass next to it. It is the second large icon from the left just under the menu bar. Click this and you will see the following dialog box.



On my machine the Explorer got assigned /dev/ttyUSB0, so I click the checkbox and hit Next >.

The next dialog box will ask me how I want to talk to the XBee, and will let me pick serial properties like baud rate, number of stop bits, etc.


The default baud rate for XBee radios is 9600 baud, in fact all the values seen are the defaults, so just clock Finish.

Another dialog box will pop up and give messages about its search for radios on the port given. If everything is wired property and you left the connection values  alone you should eventually see the following dialog box.


Select the checkbox next to the radio and click Add selected devices.

You should then see a window like below. To be honest, this window is from after I made the radio a Coordinator, so your picture will be slightly different. But that won't matter at all.



Now double click on the radio entry in the left hand pane. This will bring up data in the right hand pane about the radio.



Configuring the Coordinator


The radio plugged in now is to become our Coordinator. If you look over the above window, you will see that the radio is currently a Router. I have found that most brand new radios I buy have the Router AP firmware in them. That's fine, XCTU makes it pretty easy to change. There are a series of 5 large icons in the right pane labeled things like Read, Write, Update and others, and we will use Update.

Before we continue, let's discuss what is going on. XBee Series 2 radios are basically small computers and each radio can either be a Coordinator, a Router, or an End Device. What determines the type of radio is the firmware uploaded to it. Right now the radio seen in that last window shown has the Router AT firmware and so is a Router, and we want it to contain the Coordinator API firmware so that it will be a Coordinator. We won't worry about the different between AT and API in this post, we need API. So click the Update button to see the following dialog.




Make sure the product family is selected as XB24-ZB. If this isn't chosen, you have the wrong kind of radio and should go make sure you order a Series 2 radio. Scroll through the middle list until you find ZigBee Coordinator API. As for the Firmware version, I always make sure I chose the latest firmware. I have never tried to find out if my Coordinator, Router, and End Device radios are running the latest firmware, but it probably is safer to make sure they are always running the latest at the same time.

Then click Update and wait.

Once the update is done, XCTU will read the values from the radio again and you should see the following.



Notice the function set for the radio is now labeled as Coordinator, though if your window is like the picture I have, you will see a ... and some of the letters of Coordinator. Rolling the mouse over this area will give a tooltip giving the function set name without the ... in the middle of the text. If it is not ZigBee Coordinator API, trying doing the update again, you didn't select the correct firmware.

Now we need to set the PAN ID. Remember every radio that needs to talk to each other needs to have the same Pan ID. For this post, we will make the Pan ID have the value 2000, though you can pick any value you want as long as all the radios get the same value. Scroll through the radio parameters until you see PAN ID. You probably won't need to scroll at all, it is the top value. Type 2000 into the text box.



Next, the software libraries that will be talking with the XBee are using Escaped Mode. Without going into too much detail, this mode makes it easier to detect XBee radio frames, so many software libraries use it. To use Escaped Mode, we need to tell the radio to set its configuration parameter AP to the value 2.



Now we need to write these values into the permanent memory in the radio so that they are kept when the radio is turned off and turned back on. If you look at those large icons again in the right hand pane, you will see Write. Click that button.

If you want to confirm that your values are set correctly, click the Read button and scroll through the window to make sure PAN ID and AP have the proper values.

Cool, now we have a Coordinator Radio to rule our network!

Configuring the Greenhouse Radio


Now it is time to configure the XBee that will go in the greenhouse. I decided to make this radio a Router for no particularly good reason. My general rule of thumb is that if the radio is not going to go to sleep between sensor measurements and will not be battery powered, I usually make it a Router to have the most flexibility.

Since my greenhouse is across the lawn, I chose an XBee Series 2 Pro, which has a higher power output than a standard XBee Series 2.

I labeled the greenhouse XBee with a piece of tape with the letter R to label it as a Router, carefully seated it on a second Sparkfun USB Explorer, and plugged it into my computer. I then clicked the Search Radio icon on the left side of XCTU and got the following dialog.




Notice the additional serial port /dev/ttyUSB1. Click Next >. You will again see the Port Values dialog where you can chose baud rate, stop bits, etc. Just click Finish. After a little while you will see the following dialog.



Make sure both radios are checked and click the Add selected devices button. You will now see two radios in the left panel of the XCTU window.




If you are looking at these images carefully, the above picture is not exact. This is actually from after the radios were configured, so the radio already has the Router API firmware. But ignore this for now, and imagine it claims the radio is Router AT, which is how the radios are usually set up when bought brand new.

Now double click on the new radio, which is below the Coordinator radio in the above picture, and get to its main screen.




First off is to update its firmware to be XBee Router API by clicking the Update button.




After the firmware update finishes, set its PAN ID to 2000.



Also set its API Mode to Escaped Mode, which is done the same way as the Coordinator Radio, setting the AP configuration parameter to 2. Then click Write.

Confirm the values by clicking Read and scrolling to check PAN ID and AP.

Are The Radios Communicating?


One of the features in XCTU that I really like that the previous version didn't have is the ability to look at the network of radios and see who talks to whom and exactly what packets they are sending and the contents of those packets. You can also construct packets and have the radio send them. All of this really helps with debugging.

To get into this mode, look at the toolbar immediately under the menu bar and click the rightmost icon. As you mouse over it the tooltip will read Switch To Networking Mode. This will show the following window.






Now click on the leftmost icon in the righthand pane, it is labeled Scan. If the radios are properly configured and powered up, you will see a window like the following.




In this window, the Coordinator radio is on the right and the Router is on the left. You can see that they talk directly to each other by the line between them.

If your network were more complicated, you would see more radios and who is talking to whom by the line between the various pairs of radios.

The mode you are in now shows the network as a visual graph. You can also see a table of radios by clicking the Mode button next to the Scan button and selecting Table. You will then see the following.




By clicking on the Connections dropdown on the right hand side of each radio row you can see a list of the radios that radio is connected to.

Consoles Mode


As stated earlier, you can also connect to a radio and tell it packets to send. Though we won't go into much detail here, I will at least show the window. The Consoles Mode is entered by clicking the button just to the left of the Networking Mode button. If you mouse over the button it will show Switch to Consoles working mode. Clicking on this will show the following.




Some day perhaps I will talk about how to use this window.


Conclusion


So now we have a couple of XBee radios configured to get data from the greenhouse into the house network for storage and processing. We have discussed some of the basics of XBee Series 2 radios and mesh networking. You have seen how to use Digi's XCTU tool for configuring radios and for looking at who is talking to whom. I also set up another Pro radio to sit between the greenhouse and the Coordinator, but have not shown that radio as it is configured the same way as the greenhouse Router.

In the next installment of this series we will wire the Greenhouse XBee Router to the sensor platform built before and write the software to capture the sensor data, get it into the house, and then upload it to a time series database in the cloud.

Be seeing you!



Sunday, November 29, 2015

Using PlatformIO for Embedded Projects

I recently started using the PlatformIO system. It is a command line based development environment for many of the embedded platforms out there, such as the Arduino, Teensy, and Amtel, and I thought it would give me an easy way of automating my software builds, so decided to try it for one of my current projects. You can learn more at http://www.platformio.org. As a quick overview, it is a Python-based system that connects to a variety of toolchains, such as avrdude and gcc-avr that also has a very nice system for downloading board descriptions and libraries and all of their dependencies and making them available for your projects.

To give you an idea of how much time and effort PlatformIO saves you, the following is the set of commands to compile a project of mine for a room sensor that I will be talking more about in a future post.

$ avr-gcc -Os -g -std=gnu99 -Wall -funsigned-char -funsigned-bitfields \
    -fpack-struct -fshort-enums -ffunction-sections -fdata-sections \
    -DF_CPU=20000000UL -DBAUD=9600UL -I. -mmcu=atmega328p \
    -c -o roomsensor.o roomsensor.c

$ avr-gcc -Wl,-Map,blinkLED.map  -Wl,--gc-sections  -mmcu=atmega328p \
    roomsensor.o -o roomsensor.elf

$ avrdude -c usbtiny -p m328p -e -U flash:w:roomsensor.elf

That is a lot! Many people will wrap all this up into a Makefile so they don't have to do type in each command every time, but if something changes about your program and its needs or you decide to target it another board, you may find yourself having to modify various parts of build process. Also, personally I hate creating Makefiles, so if I can find a process that eliminates the need to write them, I am happy.

With PlatformIO, the commands to compile and upload the software is

$ platformio run -t upload

Let's get PlatformIO installed. I am using a Linux-based system, so my installation instructions are for that. The documentation at PlatformIO is quite good, so you should find it easy to find instructions for your particular platform.

I tried just installing PlatformIO on my Linux laptop but found it wouldn't properly install due to the fact that my Linux distro didn't have the latest version of pip, so the instructions below makes sure you have the latest before installing PlatformIO

$ sudo pip install -U pip setuptools
$ sudo pip install -U platformio

Once PlatformIO is installed, you should create a directory for your new project and initialize it for a PlatformIO project with the following command.

$ platformio init

You can also specify things in the init command, like what board you are developing against. However, I am working with a bare ATMega328p chip and a USBTiny programmer and I could find nothing in the supported boards that would work with what I am using. It may be there, but I didn't find it.

The initialization process will create a src/ and lib/ directory for your project, as well as a platformio.ini file that will contain all information about your project and how it is to be build and uploaded. Since a bare init command was used without specifying a board or anything, I had to fill in the values myself using my favorite editor.

[env:mybaseenv]
platform = atmelavr
board_mcu = atmega328p
board_f_cpu = 20000000UL
upload_protocol = usbtiny -e

If you specify the board to be used, say an Arduino UNO, you will find very different contents in platformio.ini. The following command will initialize the project.

$ platformio init --board uno

The contents of platformio.ini will be filled in for you.

[env:uno]
platform = atmelavr
framework = arduino
board = uno

But on with the current project.

I then created the file roomsensor.c in the src/ directory. This program is for reading a PIR sensor and lighting an LED when the sensor detects motion. The PIR is wired to pin PB1 and the LED is wired to pin PB0.

#include <avr/io.h>
#include <util/delay.h>

int main(void) {

  // Set the direction of the LED port to output.
  DDRB |= 0b00000001;

  while (1) {
    if (PINB & 0b00000010) {
      // Light the LED.
      PORTB = 0b00000001;
    } else {
      // Turn the LED off.
      PORTB = 0b00000000;
    }

    _delay_ms(50);
  }

  return (0);
}

I will show the circuit for the room sensor in the post that will cover the sensor, for now I am just wanting to discuss PlatformIO. If you actually want to just blink an LED, you could eliminate the if statement and just use a couple of delays for the amounts of time you want the LED to remain on and remain off.

As stated above, you can now compile the program and upload it to the 328p with the following command.

$ platformio run -t upload

All in all, I have found PlatformIO to be easy to use. It was easiest to use when I used an Arduino. I had to do a bunch of experimentation to figure out how to handle the bare atmega328p and the USBTiny programmer, but now that I have that figured out, future projects will be as easy as a board supported by PlatformIO. Actually, I should look how to define my own board description that I can then make available for others to use.

Friday, November 27, 2015

Sensor Data To The Cloud: Part 2

OK, data in the cloud (at least if you saw the first post in this series), but for the moment the cloud is being used as WOM (Write Only Memory). What we want now is the ability to get the data out so that we can do something with it.

I plan eventually to be able to access the raw data points, probably using some sort of JSON format served by a web server interface, but for now let's have some fun and look at graphs in a web browser.

You can see the code on github at https://github.com/kmhughes/robotbrains-examples/tree/master/data/cloud/org.robotbrains.data.cloud.timeseries.server.

The current interface is very quick and dirty, but it at least let me start looking at the data. Here is an example of a couple of graphs from my test installation.






I first set up a small web server. I used the one from the Interactive Spaces open source project, though I made some changes to it that I have wanted to make for a while and moved it to using Netty 4 rather than Netty 3, cleaned out some of the deprecated methods, and did some general refactoring. These web server modifications will continue for some time as I get more familiar with the Netty 4 API, and decide on the new internal architecture for the web server. It was written in a rather demand driven fashion and now with 3 years of usage experience, it is definitely time for a cleanup.

The main class for the data web server is StandardDataWebServer. This class creates an instance of the Interactive Spaces web server and attaches a dynamic content GET handler. You can see this in the startup() method.

webServer = new NettyWebServer(webServerPort, log);
webServer.startup();

webServer.addDynamicContentHandler("graph", true, new HttpDynamicRequestHandler() {
  @Override
  public void handle(HttpRequest request, HttpResponse response) {
    handleGraphRequest(request, response);
  }
});

Notice that the URL prefix has been set to graph. Graph URL paths, minus the host and port for the webserver, will be of the form

/graph/source/sensing_unit/sensor?start=start_date&end=end_date

Here is the URL path I used for the temperature graph above.

/graph/keith.test/pi2/temperature?start=2015/11/23@00:00:00&end=2015/11/25@08:00:00

Here you can see that source is keith.test, the sensing_unit is pi2, and the sensor on the sensing unit is temperature.  The values for these fields are set in the Python script running on the Pi and your values will be different.

The date formats are easy enough. Year first, then month, then day. After the @ is the time in a 24 hour clock, so 00:00:00 is midnight.

The method handleGraphRequest() performs the following operations through various methods it calls:

  1. It parses the URL and creates the sensor data query with the method getDataQueryFromRequest().
    • The URL path components specify the exact sensor the data is wanted for.
    • The URL query parameters specify the start and stop dates for the query range.
  2. The sensor data query is handed to the Database Relay (class KairosDbDatabaseRelay) that then queries KairosDB to get the requested data. The query method called is 
    querySensorData().
  3. The results of the sensor data query are transformed into the correct components for a JFreeChart data graph and the chart is generated in renderChart().
    1. An XYDataset is created from the sensor data.
    2. A chart is then created from the XYDataset.
  4. Finally the JFreeChart chart is rendered as a PNG graphic and sent back to the web browser in writeChartResponse().
The JFreeChart dataset is easily created from the sensor data samples.

private XYDataset createDataset(SensorData data) {
  XYSeries series = new XYSeries("Fun Data");
  for (SensorDataSample sample : data.getSamples()) {
    try {
      series.add(sample.getTimestamp(), sample.getValue());
    } catch (SeriesException e) {
      log.error("Error adding to series graph", e);
    }
  }

  return new XYSeriesCollection(series);
}

The chart is then created.

private JFreeChart createChart(SensorDataQuery query, XYDataset dataset) {
  return ChartFactory.createTimeSeriesChart
      String.format("Sample Data: %s - %s - %s",
          query.getSource(), query.getSensingUnit(), query.getSensor()),
      "Time", "Value", dataset,
      false, false, false);
}

The chart is then finally written to the web browser.

private void writeChartResponse(HttpResponse response, JFreeChart chart) throws IOException {
  BufferedImage chartImage = chart.createBufferedImage(560, 370, null);
  ImageIO.write(chartImage, "png", response.getOutputStream());
  response.setContentType(CommonMimeTypes.MIME_TYPE_IMAGE_PNG);
}

As you can see, the dimensions of the PNG graphic are currently fixed, eventually the height and width will be set from query parameters.

It was very easy to get JFreeChart to visualize my data for me, I look forward to learning more about the API.

The modifications to the class KairosDbDatabaseRelay were pretty simple as the KairosDB Java API is really easy to use. The query code is given below.

String metricName = createEventKey(query.getSource(), 
    query.getSensingUnit(), sensor);

QueryBuilder builder = QueryBuilder.getInstance();
builder.setStart(query.getStartDate().toDate())
    .setEnd(query.getEndDate().toDate());

builder.addMetric(metricName);

QueryResponse response = kairosdbClient.query(builder);
for (Queries queries : response.getQueries()) {
  for (Results results : queries.getResults()) {
    for (DataPoint dataPoint : results.getDataPoints()) {
      SensorDataSample sample =
          new SensorDataSample(sensor, dataPoint.doubleValue(),
              dataPoint.getTimestamp());

      data.addSample(sample);
    }
  }
}

KairosDB requires a metric name for each item it is storing time series data for. The Data Relay uses the source, sensing unit and sensor concatenated together with a spacer character to create its metric names. Eventually the Relay will contain a small database giving the sources, sensing units, and sensors to provide mappings for data types for the data. The metric name generated here will be in metricName.

Next a KairosDB QueryBuilder is created. The start and end dates for the query are then set. The Relay uses the Joda time library (if you are working with time in Java, you must use Joda), but the KairosDB wants Java dates, so the toDate() method is called to convert from Joda DateTime instances to the Java Date class. finally the metric name is added to the query.

The next bit of code has 3 nested for loops. This may seem overkill, but the KairosDB API allows for multiple queries per call to KairosDB, and for the moment we only have the one metric, so have to dig through several data structures to get the data returned from the backend database.

And there you have it. A quick and easy web interface to graph the time series data.

I plan on continuing to expand the Relay to give new functionality. Not every change I make will become a blog post, so you might want to keep an eye on the Github project if you find this interesting and want to be kept aware of the changes. At some point I want to be able to pull the data into the data analysis package R and eventually do machine learning from the data, I will probably end up writing posts about those things.


Monday, November 23, 2015

Sensor Data To The Cloud: Part 1

I have been experimenting with sensing data in my home and storing that data in the cloud. This is a project in progress and I thought I would write about it as all of the pieces come together. This series will discuss

  1. capturing sensor data, in this case temperature and light level data, on a Raspberry Pi,
  2. communicating the sensor data to the cloud over MQTT
  3. storing the data in a time series database on the cloud server
  4. providing access to that data
Step 4 will be done in the second blog post.

Do be aware, there are a LOT of steps here to get everything configured. For some of you, this may all go quickly, for others it will seem like a lot and perhaps get a bit confusing because of all the steps, some will handle some parts quickly and other parts not so quickly. If you have trouble with any of the pieces, see if you can find someone who understands those pieces to help you out. Also, for some things, the online documentation is good, for instance creating a Google Compute Engine instance, so I won't be going into details there.

There will also be links to github repos containing all of the code written by me.

This post has several major sections. The first will be about setting up the cloud server, including the installation of all needed software. The second section will talk some about how I actually went about figuring out how to do all of this stuff, as the post presents it in an order better suited to giving step by step instructions rather than the messy process of figuring out what the step by step process is. The final section shows the sensor circuit used with the Raspberry Pi and how to configure the Pi to get the data to the cloud.

And with that, let's get started!

Setting Up The Cloud Instance


We'll start with the cloud side first.

The first thing I did was obtain and setup a Google Compute Engine instance. You can find more details about that here: https://cloud.google.com/compute/. I won't give the complete details of every step that I went through to create the instance as the online documentation is good, but will list some of the highlights.

  1. The image on the machine is a Linux instance running Debian Jessie
  2. It has a static IP address that I connected to the domain name smartspaces.io
  3. I used the Metadata > SSH Keys tab on the Google Developer's Console to install an SSH key created on my laptop so that I could ssh and scp content from my laptop into the instance without a password
  4. The cloud instance by default only opens the firewall for HTTP ports, but network ports for MQTT and for a non-standard HTTP port for an AJAX call are necessary

Opening Network Ports


It is necessary to open up a couple of network ports to allow MQTT and non-standard HTTP AJAX traffic. To open these ports you will need to go to your VM Instance in the Google Developer's console for your Compute Engine instance and click the default link under Network. You can see this area circled below.




This will take you to the network page. Click on Add Firewall rule to add in the MQTT port. It is the blue button in the image below.




You will see a window that looks like this


I set the Name to MQTT and the Description is MQTT Ports.

Under Allowed Protocols and Ports, I put tcp:1883; tcp:8883.

Everything else I left alone.

I also created a firewall rule for accessing the web server that provides the time series data. This is different than the web server on the instance, though I could configure the existing web server to direct traffic to the time series web server. This is more detail than we want to worry about here, so just creating a second web server port keeps it easy.

For the timeseries web server, I set the Name to Web Timeseries. The Description is Timeseries HTTP Ports.

Under Allowed Protocols and Ports, I put tcp:8095.

Everything else was left alone.

If you were doing this for real, you would probably want some sort of authentication for accessing the timeseries web server, but we won't worry about that here.

Installing Cloud Software


There are a few pieces of software that need to be installed on the cloud instance. These will help with the later installations.

$ sudo apt-get install update-rc.d
$ sudo apt-get install unzip

Installing MQTT


The MQTT server I chose was from the Moquette project. You can get details of the project and download the software from http://andsel.github.io/moquette/.

I placed the software in the /opt/moquette folder of the Google Compute Engine instance.

The config/moquette.conf file was modified as follows. I will give the lines that I changed and the value that I gave them.

The allow_anonymous property will either allow or prevent just anyone from using your broker. We will set it to not allow anonymous logins so that clients will need to log into the broker to use it. All of the sample code on the github links provide login into the broker.

allow_anonymous false

Since the broker does not allow anonymous connections any more, you will need a user name and password. Moquette's password are coded as SHA256 hashes. Say you want to create a user named keith and have a password of yourpassword. First get the SHA256 hash of yourpassword. I ran the following command on the Google Compute Engine instance.


$ echo -n "yourpassword" | sha256sum
e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951  -

That really long string after the echo command is the SHA256 hash of the password. The hash does not include the - on the end. Now that you have the password hash, you need to edit the file config/password_file.conf. This file can contain as many lines as you want, one line for each different user name and its associated password.


keith:e3c652f0ba0b4801205814f8b6bc49672c4c74e25b497770bb89b22cdeb4e951

I do recommend that if you are using this blog that you do not make your password yourpassword as people may see that password here and try it first.

Now we need to make sure that Moquette will start running every time your cloud instance reboots.

You have a choice here of using init.d scripts or Upstart. Upstart is the default system on Ubuntu instances, whereas Debian by default uses init.d. You can get the startup scripts here as well as directions on how to make them work: https://github.com/kmhughes/robotbrains-system/tree/master/messaging/mqtt/moquette.

Installing Cassandra


The backend database for the timeseries data is the Apache project Cassandra.

Installing Cassandra is fairly easy. First you add in the location of the Cassandra packages to the Debian package manager, and then make sure Cassandra will be autostarted when the machine reboots.

First let's get the Cassandra repository location added to the Debian package manager.

$ sudo nano /etc/apt/sources.list.d/cassandra.list

The contents of cassandra.list will be

deb http://www.apache.org/dist/cassandra/debian 21x main
deb-src http://www.apache.org/dist/cassandra/debian 21x main

Now let's get Cassandra installed.

$ gpg --keyserver pgp.mit.edu --recv-keys F758CE318D77295D
$ gpg --export --armor F758CE318D77295D | sudo apt-key add -
$ gpg --keyserver pgp.mit.edu --recv-keys 2B5C1B00
$ gpg --export --armor 2B5C1B00 | sudo apt-key add -
$ gpg --keyserver pgp.mit.edu --recv-keys 0353B12C
$ gpg --export --armor 0353B12C | sudo apt-key add -
$ sudo apt-get update
$ sudo apt-get install cassandra cassandra-tools

All of the gpg commands only need to be done once and tells your local machine to trust the Cassandra repositories. 

Once Cassandra is installed, you should tell the operating system to automatically start Cassandra when the machine reboots. While you are at it, you should also start the service so you can continue with the instructions here.

$ sudo /usr/sbin/update-rc.d cassandra defaults
$ sudo service cassandra start

Installing KairosDB


Now we need to provide an easy way to store time series data into Cassandra and retrieve the data. The package we will use is KairosDB.

You can find out details about KariosDB at https://github.com/kairosdb/kairosdb. This site has links to the documentation on how to use KairosDB if you are interested in learning how to use it yourself rather than just using the code I will provide.

It is easiest to install KairosDB by getting the Debian package.

$ wget https://github.com/kairosdb/kairosdb/releases/download/v1.0.0/kairosdb_1.0.0-1_all.deb
$ sudo dpkg -i kairosdb_1.0.0-1_all.deb

You will then find KairosDB in /opt/kairosdb.

We now need to modify config/kairos.properties so that we change the web port to use for accessing the data and so that it uses the correct backend database.

By default, KairosDB exposes its web server port at 8080. This conflicted with a regular web server running on my server, so I decided to change the port to 8090. Look for the line

kairosdb.jetty.port=8080

Now change the value on the end to 8090.

kairosdb.jetty.port=8090

By default, KairosDB uses H2 as its backend database. We need to change it so that it uses Cassandra. Look for the following line

kairosdb.service.datastore=org.kairosdb.datastore.h2.H2Module

Put a # at the beginning of this line so it looks like

#kairosdb.service.datastore=org.kairosdb.datastore.h2.H2Module

Now look for the line like the following

#kairosdb.service.datastore=org.kairosdb.datastore.cassandra.CassandraModule

and change it to

kairosdb.service.datastore=org.kairosdb.datastore.cassandra.CassandraModule

Save the file.

Now start KairosDB. It will automatically connect to Cassandra and create all necessary tables.

$ sudo service kairosdb start

The KairosDB Cassandra tables need to be modified from the defaults that KairosDB creates them with. First, start up the Cassandra command line shell.

$ cqlsh

After the shell is running, type the following commands. Everything before the > is the prompt that cqlsh is giving you, you should type everything after the >.

cqlsh> use kairosdb;
cqlsh:kairosdb> alter columnfamily data_points with read_repair_chance = 0.1;
cqlsh:kairosdb> alter columnfamily row_key_index with read_repair_chance = 0.1;
cqlsh:kairosdb> alter columnfamily string_index with read_repair_chance = 0.1;

If the KairosDB namespace is not available, (you would see an error like InvalidRequest: code=2200 [Invalid query] message="Keyspace 'kairosdb' does not exist" if it doesn't exist) then KairosDB is not connecting properly to Cassandra. Look at the logs in the terminal window for information, perhaps you don't have a port set correctly for contacting Cassandra, or perhaps KairosDB's network ports clash with another port being used on your Compute Engine instance. You might also have forgotten to change KairosDB from using H2 to Cassandra for its database.

Installing the Data Relay Server


OK, hopefully by now you have the MQTT broker, Cassandra, and KairosDB running. However, now the MQTT broker needs to talk to KairosDB to store your data in Cassandra. For this I have implemented the Data Relay Server. This server has an MQTT client that will receive data from the Raspberry Pi and then call KairosDB to ultimately store the data in Cassandra.

You can find the code for the Data Relay Server on github:
    https://github.com/kmhughes/robotbrains-examples/tree/master/data/cloud/org.robotbrains.data.cloud.timeseries.server

You can find a zipped version of the server on Google Drive: 
    https://drive.google.com/file/d/0BxZiP9lENIZjbEZqM2d5dmRmdW8/view?usp=sharing

For now, download the Relay from Google Drive and unzip it in your home folder on the Compute Engine instance. It will eventually be placed in /opt, but for now let's just have it running if you are logged into the Compute Engine instance.

Now you need to create a YAML file that will contain some configuration properties for the Data Relay Server. Here is a sample file I called smartspacescloud.yaml.

mqtt.server.host: smartspaces.io
mqtt.server.port: 1883

mqtt.username: keith
mqtt.password: yourpassword

smartspaces.cloud.timeseries.topic.incoming: /smartspaces/cloud/timeseries/incoming
smartspaces.cloud.timeseries.node.name: /smartspaces/cloud/timseries/relay/server

database.kairosdb.connection.host: localhost
database.kairosdb.connection.port:8090

The first two properties set the host and port of the MQTT server that is being used to transfer your sensor data. Notice here it is pointing at my MQTT server. I am using the standard MQTT port. The values should match those for your server.

The next two properties give the user name and password you set up for your MQTT server account. Notice here the password is not given as its SHA256 hash, it is in plaintext.

The next two properties specifies the MQTT topic that the sensor data will be transferred on and how the MQTT client identifies itself.

The Data Relay Server will subscribe to the specified topic to receive the MQTT data from the Raspberry Pi, whereas the Raspberry Pi will be a publisher on this topic to send the data. Pick a name meaningful to you, here the name /smartspaces/cloud/timeseries/incoming was chosen to be self documenting, this is the SmartSpaces topic for timeseries data coming into the cloud. If you wanted, you could call it something like /foo/bar, but after you have a bunch of topics, it might then be hard to know what they are actually being used for if you use names like /foo/bar.

The node name should also be meaningful, here /smartspaces/cloud/timseries/relay/server. Notice that my Relay instance that runs on the Compute Engine instance ends with server. The instance I run on my laptop for debugging ends in laptop.

The final two properties specify which KairosDB instance the Server should talk to. The values supplied should be fine unless you changed the port that KairosDB listens on or placed it on another machine. If so, change the values accordingly.

You will make a file very similar to this one on your Raspberry Pi as well so you have the same information on both sides as both the Data Relay Server and the Raspberry Pi are MQTT clients, even though one is a subscriber to the data and one is a publisher. The copy on the Raspberry Pi can delete the two lines about KairosDB since you won't be running KairosDB on the Pi.

Say you unzipped the Data Relay Server and stayed in the folder that you unzipped the server into. You can start the server by typing

$ org.robotbrains.data.cloud.timeseries.server/bin/org.robotbrains.data.cloud.timeseries.server -c smartspacecloud.yaml

You should see a bunch of logging information going by telling you that everything is working. If not, double check your configuration parameters.

Now let's install the Relay in its final home. Move the server directory to /opt. Once it is there then copy your configuration file into the config folder. Finally copy the systemd file that will start the Server on machine boot into the proper system folder and enable it and then start it running.

$ sudo mv org.robotbrains.data.cloud.timeseries.server /opt/robotbrainsclouddataserver
$ sudo mv smartspacecloud.yaml /opt/robotbrainsclouddataserver/config
$ sudo cp /opt/robotbrainsclouddataserver/bin/robotbrainsclouddataserver.service /etc/systemd/system
$ sudo systemctl enable robotbrainsclouddataserver.service
$ sudo systemctl start robotbrainsclouddataserver.service

If you have everything set up correctly, you should output appearing in 
/var/log/robotbrainsclouddataserver/robotbrainsclouddataserver.log telling you that the Relay Server has started.

The Server has a bunch of pieces and explaining them all in detail would take a lot of room, but let's look at the highlights.

Start with ServerMain and see how it pulls the server pieces together.

To understand the Raspberry Pi to Database Relay communication, look at the PahoMqttRemoteDataRelay class. Its  startup() and shutdown() methods show how to connect to and disconnect from the MQTT broker. JSON messages are used to encode the data between the Pi and the Relay, you can see the details starting from the handleMessageArrived() method.

To see how data is transferred into KairosDB, look at the KairosDbDatabaseRelay class. The startup() and shutdown() methods show how to set up and shut down communication with KairosDB. The processSensorData() method takes data being transferred from the MQTT client and sets up the data structures needed to transfer it to KairosDB.


The Server Is Done!


Wow, that was a lot of work! But now you have your server configured, though it isn't doing anything yet but sitting there patiently waiting for your data.

While I wrote the software I did things very differently so I could test things as I went along. Here were my steps, more or less, in case you're curious about my approach,.

First I installed Moquette on the Compute Engine instance. I then wrote an MQTT publisher and an MQTT subscriber in Java and ran both on my laptop, making sure I could get roundtrip data. You can see my test clients here: https://github.com/kmhughes/robotbrains-examples/tree/master/comm/mqtt

I then wrote the beginnings of the MQTT publisher in Python and made sure I could have the Python publisher write any sort of data whatsoever to the Java subscriber. In fact, I had it write Hello, World, which is the time-honored first message that any program should send. The publisher was then copied to a Raspberry Pi to confirm I could get data from a Pi to the Java subscriber on my laptop.

Next was building the circuit for the Raspberry Pi and writing a small Python program to read the data from the sensors.

The sensor reading Python program was then combined with the MQTT publisher Python program and cleaned up. The data format was decided. I then checked that the newly combined sensor publisher Python program could send data to my Java subscriber.

I then took that Java subscriber and copied it into a new project and started creating the Data Relay Server.

Next I installed Cassandra and made sure I could connect to it with csqlsh. Once I knew that worked, I installed KairosDB and configured it so that it could talk to Cassandra. That took a couple of tries, but I was finally able to see the Cassandra namespace for KairosDB.

Next I took my nascent Relay Server and made it place random data into KairosDB. Once I saw the fake data successfully going in, I tried reading the fake data out.

Finally I connected the Kairos component of the Data Relay Server to the MQTT client in the Server and tested whether data went in and whether I could query it.

It was a lot of steps, but it meant at each step I understood that piece and knew it worked before I added in another step. If I had tried everything all at once it would have been VERY difficult to debug. It also meant I could really learn how to use each component in isolation. Trying to write everything all at once when not understanding any of the pieces would have been very hard.

Setting Up Your Raspberry Pi


I am using a Raspberry Pi 2 for this project. If you are not using a Pi 2, these directions may or may not work, I have not tested them on a Pi 1.

To keep things simple, let's use Python on the Raspberry Pi to access the sensors and send data to the cloud.

I tried using the Ubuntu installation for the Pi but had problems getting the Python library for accessing the Pi GPIO pins to work, so gave up and switched to Raspbian.

Python uses a package called pip to install Python packages. If you don't already have pip installed, you can install it with the following commands, depending on whether you use Python 2 or Python 3.

$ sudo apt-get install python-pip
$ sudo apt-get install python3-pip

You now need to install a variety of Python packages used by the sample code.

$ sudo apt-get install python-dev
$ sudo pip install rpi.gpio
$ sudo pip install pyyaml
$ sudo pip install paho-mqtt

If you are using Python 3, replace pip in the above commands with pip3.

Wiring


I chose to measure temperature and light levels in this post to keep things simple. I used a TMP36 for reading the temperature as it gives a very simple analog signal linear in the temperature being sensed. The light level was measured with a photoresistor, also giving an analog signal.

The Raspberry Pi does not come with analog to digital converters, so if you want to read analog signals, you will need an external A/D converter. I chose the MCP3008 A/D converter which gives an SPI interface for reading the analog data. I will not be describing the SPI protocol in this post, perhaps a later one, but there are good articles online if you want to Google them.

The complete circuit is given below. The resistor is a 10k, the photoresistor was just something I had in the house, I think its highest resistance was around 10k. And that big blue thing on the left side is a Raspberry Pi 2 Cobbler. The Cobblers make it really easy to breadboard with a Raspberry Pi, I recently bought a lot of them.

First the Fritzing diagram.



The following pictures show a bit of a rat's nest of wiring, the Fritzing will make things much more clear.



And with the Pi...


I am leaving this on a large breadboard in case I want to add more sensors to it over time.

The Python Program


The Python program can be found on github at https://github.com/kmhughes/robotbrains-examples/tree/master/data/cloud/org.robotbrains.data.cloud.timeseries.client.

Copy the files robotbrainsclouddataclient.py and robotbrainsclouddataclient to your /home/pi directory.

Now we need to have the client know about the MQTT broker in the cloud and what topics to publish on. Create a file called smartspacescloud.yaml and place the following contents in it

mqtt.server.host: smartspaces.io
mqtt.server.port: 1883

mqtt.username: keith
mqtt.password: yourpassword

smartspaces.cloud.timeseries.topic.incoming: /smartspaces/cloud/timeseries/incoming

The properties here should match the properties in the Data Relay Server's smartspacescloud.yaml. If you don't get them correct, the Data Relay Server will not get the data from your Pi. The host and port values should match the host and port for your MQTT server on your cloud instance.

Now let's try your client and see if you get data in the cloud. But first, let's see if we get any data at all. The Python program is initially configured to not contact the cloud and to write out sensor data every 5 seconds on the console.

Run the following commands on your Pi.

sudo ./robotbrainsclouddataclient.py smartspacescloud.yaml

The sudo is necessary as you can only access the GPIO pins on the Pi as the root user.

If you have wired everything properly, you should see sensor measurements being printed every 5 seconds on the Pi. If you don't see anything, check that your circuit is wired properly. If you do see data being printed, type ^C to end the program.

If your circuit worked, let's get data to the cloud.

On the cloud instance type the following command. This will let you see log output as it is written into the logs.

$ tail -f /var/log/robotbrainsclouddataserver/robotbrainsclouddataserver.log 

Now open the Python script in your favorite editor and look for the line where the variable CONNECT is set equal to 0. Change the 0 to 1 and run the program again.

If you have connected everything properly, you should see sensor measurements coming into your cloud server every 5 seconds. If you don't see anything, make sure you have the same MQTT server connection information in the smartspacescloud.yaml files on both machines and that the MQTT topic is the same on both sides.

Measurements every 5 seconds might be a bit much, I have my instance reading the sensors every 10 minutes. Once again, open up the Python script in your favorite editor. Change the value of DEBUG from 1 to 0 to stop debugging output and change SAMPLING_PERIOD to how many seconds you want between sensor measurements. As I said, I record measurements every 10 minutes so set the sampling period to 600.

 We would like the Data Client to run whenever the Pi boots. To do so, copy robotbrainsclouddataclient from /home/pi to to /etc/init.d. Then type the following command so that the Pi knows to run this script every time it boots.

$ update-rc.d robotbrainsclouddataclient defaults


The Python script has 5 major sections.

The first contains various parameters like DEBUG, CONNECT, and SAMPLING_PERIOD that are used and which pins the sensors are attached to on the A/D converter chip.

The second section contains code from the folks at Adafruit for communicating with the A/D chip. It uses a technique called bit-banging, where the SPI signals are generated by turning digital pins on and off and reading digital pins, rather than using specialized pins and processor instructions that handle things like the SPI clock for you.

The next section creates a handler for the MQTT client for handling data connections and also sets up a handler to detect when a kill signal is sent to the script. The handler will shut down the MQTT client and perform any other needed cleanups.

The 4th section reads the configuration file and sets up the MQTT client.

The final section creates a loop that will read the sensor signals, format them into a JSON message, and then send them over the MQTT channel to the Data Relay for storage into the time series database.

Done For Now!


If you made it through everything, you are now reading sensor signals on your Raspberry Pi and storing them in a time series database in the cloud. As I said before, this is a lot of steps, so if it isn't working, find people you know who understand the pieces that aren't working and see if they can help.

Now that we have data going into the database, we need to get it out of the database so we can process it. I will soon be writing the code that allows queries for reading the data, uploading it to github and then write another post on how reading the data works.

Until next time...






Saturday, July 4, 2015

Greenhouse: Phase 3, Sensors Working

It has been a while since I posted, but a lot has been going on.

As far as this project is concerned, the barometer ended up being a real pain. I was using the Adafruit MPL3115A2 breakout board and, as you can see from the previous post, I tied it in on the same I2C bus as one of the AM2315 Temperature and Humidity Sensors on the DUE. I modified the Adafruit library for the MPL3115A2 in the same way I modified the AM2315 to work on any I2C bus I want it to and fired everything up.

Nothing. Not a peep from the sensor.

I tried it on the second I2C bus.

Nothing.

Now that's annoying. I checked the wiring a bunch of times, checked my code, and yet... nothing.

I then dug out an Arduino UNO to try it out on and everything worked perfectly.

I wrote a post on the forums at Adafruit to see if anyone had any ideas. Someone else tried the MPL3115A2 on a DUE and also never got it to work. So I went over to Sparkfun and bought their same board. I know, kinda silly, but I was hoping it was something about the Adafruit board and that the Sparkfun board would just work. But no, it was not to be and the Sparkfun board remained silent on the DUE. Ultimately the folks at Adafruit decided that something was odd about the DUE I2C libraries just for the barometer board, perhaps a timing issue or something, even if it was working with the temperature/humidity sensors.

Poking around, I found the Teensy 3.1 has 2 I2C buses so gave it a try. Everything worked the first time, though the Teensy libraries do not use the I2C interfaces from twi.h, and I found they had their own I2C library called i2c_t3. So much for uniform library naming.

My final modifications for the Adafruit libraries for the AM2315 and the MPL3115A are found here and here, respectively.

Here is a picture of the final breadboard. I am leaving it as a breadboard for the next year in case I want to add in other sensors. I left it with the Sparkfun barometer board, which is on the right. The left side is the Teensy 3.1.




Below is the Fritzing diagram of the circuit. I couldn't find a part that shows the pins on the underside of the Teensy so just drew the wires with a note to label them. You can see the wires coming out from under the Teensy on its right side in the above picture.




My Arduino sketch is still not sending the binary data to the host processor, but here is the current code.



#include <i2c_t3.h>
#include <Adafruit_AM2315.h>
#include <Adafruit_MPL3115A2.h>

// The inside AM2315 temperature and humidity sensor.
Adafruit_AM2315 am2315Inside(&Wire1);

Adafruit_MPL3115A2 barometric(&Wire1);


// The outside AM2315 temperature and humidity sensor.
Adafruit_AM2315 am2315Outside(&Wire);

// The sensor data packet containing all data to be transmitted
// to the host.
typedef struct _SenseData {
  float temperatureInside;
  float humidityInside;
  float temperatureOutside;
  float humidityOutside;
  float altitude;
  float barometricPressure;
  float barometicTemp;
} SenseData;;

SenseData senseData;

void setup() {
  Serial.begin(9600);
}

void loop() {
 if (am2315Outside.begin()) {
    am2315Outside.readTemperatureAndHumidity(senseData.temperatureOutside, senseData.humidityOutside);
    Serial.print("Outside Hum: "); Serial.println(senseData.humidityOutside);
    Serial.print("Outside Temp: "); Serial.println(senseData.temperatureOutside);
  } else {
    Serial.println("Cannot see outside AM2315");
  }

  if (am2315Inside.begin()) {
    am2315Inside.readTemperatureAndHumidity(senseData.temperatureInside, senseData.humidityInside);
    Serial.print("Inside Hum: "); Serial.println(senseData.humidityInside);
    Serial.print("Inside Temp: "); Serial.println(senseData.temperatureInside);
  } else {
    Serial.println("Cannot see inside AM2315");
  }
 
  if (barometric.begin()) {
    senseData.barometricPressure = barometric.getPressure();
    Serial.print(senseData.barometricPressure/3377); Serial.println(" Inches (Hg)");

    senseData.altitude = barometric.getAltitude();
    Serial.print(senseData.altitude); Serial.println(" meters");

    senseData.barometicTemp = barometric.getTemperature();
    Serial.print(senseData.barometicTemp); Serial.println("*C");
  } else {
    Serial.println("Couldnt find barametric sensor");
  }

  delay(1000);
}

You may notice the conditionals inside the loop that are constantly calling the begin() method for the sensor. I did this so that if a physical sensor is removed, the other sensors can continue to report their data. The else section will probably set those fields to minus infinity or NaN if the sensor is not read.

When I am finally ready to send this to the host computer, I will use a line like

Serial.write(&senseData, sizeof(senseData);

This line will send out the packet with the data in the order that the fields are defined in the structure SenseData above.

In the next installment of the Greenhouse Saga, I will show the Java end that will read the data from the sensor and place it in an Interactive Spaces activity.