New in Smart Spaces: November, 2016

As I near getting a deployment of the prototype of my company's product (http://www.inhabitech.com) in my home, I have been making quite a few changes to Smart Spaces to support the new usages I have. There have been modifications to the core open source system and to some items in the sandbox that I thought I would mention. This isn't an exhaustive list, but contains some of the highlights.

First let's start with something fun. Recently support for GPIO on a Raspberry Pi has been added to Smart Spaces through the Pi4J project. This was added to support reading a PN532 RFID/NFC scanner through SPI, but has also given the ability to directly control GPIO pins. This is leading to much better native hardware support for Smart Spaces. I will soon have example Smart Spaces activities that will use the GPIO pins to demonstrate how this is used.

Zeroconf/mDNS support is now a key part of Smart Spaces, with support for mDNS Service Advertising and Discovery. This is moving Smart Spaces towards being autoconfigurable, though it also still supports manually putting a network together..

Smart Spaces now has a lot of support for the Internet of Things protocol MQTT, all designed as I have built a collection of ESP8266-based sensor boards using MQTT to communicate with Smart Spaces. The Interaction API allows sensors to attach to the Sensor portion of of the API over MQTT (potentially with future CoAP support). The original MQTT client did not support automatic reconnect to an MQTT broker if the connection was lost, this has now been implemented, including automatic resubscription to MQTT topics registered with the client if not using MQTT persistent sessions, helping make the sensor and communication layer much more reliable. The Smart Spaces Master will also soon support an MQTT broker so that an additional one need not be installed in a Smart Spaces installation unless there is a reason to run an external broker. Future plans also include support for SSL-based connections so that a Smart Space can communicate with other spaces through the cloud or monitoring/alerting systems in the cloud.

The Core platform now also supports ReactiveX programming using rxjava. The core platform now contains an EventObservableService that allows components to register named ReactiveX Observable instances for easy discovery by other components in the system that want to observe events from those components. This is being used a lot in the Interaction API that now publishes ReactiveX events for people entering and exiting a room.This provides a uniform way of locating event observables in an application.

The core platform no longer uses ROS for Smart Space master and Space controller control communication. Though ROS support is still in Smart Spaces, it did not work well in an environment where the master or space controllers could go up and down. We would often see a lot of XmlRpc errors in the Experience Centers at Google, and this change was wanted way back then, but it has only finally happened. There is now a TCP server running on each Space Controller that a TCP client will attach to from the Master for control. This will evolve over time so that the Master can start a reconnect schedule if it loses contact with a Space Controller. This is evolving towards being able to run a space without a master constantly running, but rather have automatically starting Space Controllers as hardware comes up and down in the space.

All of the above changes are now findable on the master branches of the SmartSpaces github repos. If you want to track the evolution of the open source project, do be aware that the smartspaces and smartspaces-sandbox repositories have a development branch.

I am also planning on integrating Smart Spaces with the Eclipse Smart Homes project. This will make it easy to access a lot of hardware without needing to reduplicate that work in many cases.

Evolving Interaction API and Scala in Smartspaces

It has been quite some time since I wrote anything in this blog. I started my own company in February, so have been pretty busy, without a lot of time for writing in this blog. I have about 4 posts in draft about building a sensor network in LUA on an ESP8266, my configurations of Raspberry Pis as sensor platforms, etc. Some day I will post those.

The evolution of Smart Spaces continues apace. Recently I started writing a new Interaction API that makes it simpler to model a physical space with sensors in it. Though Smart Spaces supported sensors in spaces, it was very low level in the Interactive Spaces days and I have been trying to up the abstraction game.

If you want to see some of what I mean in the file

https://github.com/smartspaces-io/smartspaces-sandbox/blob/development/models/smartspaces-interaction/src/main/scala/io/smartspaces/sandbox/interaction/test/testdescription.yaml

Eventually this model will be built in a GUI, but for now I just needed something simple, so created a quick YAML format.

Look for the section labeled sensors. This contains a listing of sensors and a description of them. Eventually this will include reference to a detailed list of what the sensor senses, but for now, there is just an ID, a name, and a description.

The section physicalLocations contains similar information about physical locations in the model. This model will eventually evolve into being able to describe the connectivity of the space, which rooms are connected to which other rooms.

The section sensorAssociations then links sensors to a physical location via the sensor and physical location IDs. The API has the concept of a Sensed Entity Model that will contain all the sensor values currently known to be valid for that Sensed Entity. An example of a Sensed Entity is one of the physical locations discussed above, but really, they could be anything that is being sensed, such as a person. One of these models is built for every item that can have a sensed state in the API.

One of the interesting parts of the Physical Location Sensed Entity Model is that it can model the occupants that are currently in the room. To do that, it is necessary to be able to model people. The descriptions of people known in the model are found in the persons section of the YAML file.

One of the ways that I am considering knowing where someone is in the space is caught up in the Marker concept. A marker is something that marks something else's location in the space. Often a marker will be something transmitting a signal, such as a BLE node or perhaps an RFID tag that is being carried around by a person. You can see this by looking at the markers section and the markerAssociations section. The sample file contains a BLE marker, identified as such with the format of the Marker ID, though there should probably be a type field in here specifying that it is a BLE device. This marker is then associated with a person. Marker detectors in the space, such as the sensor with ID /home/livingroom/proximity can then detect these markers and update the occupants of the room.

Yes, I know RSSI is not a great way of knowing someone is in a room, but it was a simple technology that allowed me to quickly get something working.

This API is very knew and rapidly evolving as I build a prototype system that lets me model what I want to model. My house is getting more and more sensors in it and I am using that to help drive the modeling possible with the API, rather than building the API in my mind, hoping that it is actually useful for the tasks I want to achieve.

You can browse through the current state of the code here:

https://github.com/smartspaces-io/smartspaces-sandbox/tree/development/models/smartspaces-interaction/src/main/scala/io/smartspaces/sandbox/interaction

Finally, you will notice the Interaction API is written in Scala. As I build more of the new modeling framework, I decided I needed a richer language for doing it in. So now Smart Spaces is compiled as a Scala project, though the bulk of the system code is still Java. No way I want to immediately rewrite all that code in Scala. New stuff, sure, you can see some simple Scala in the workbench and there will be more in the future. I have also done some of the examples in Scala as it is now possible to compile Activity and Library projects in Scala. Be aware, I am new to Scala, so I am sure I am not writing idiomatic Scala. Yet, anyway.

I got the "Figurin' out DBus" BlueZ

I have been thinking about a context-aware framework for some time now. "Context aware", in the usage I mean, is about software having an idea of the location of things it is tracking, making it possible to do things based on the context, or location, where that thing is.

Though I am not fully convinced that Bluetooth Low Energy (BLE) is a good final solution for my context aware needs, it is an easy to use solution, particularly with nice platforms like the Intel Edison, TI Sensor Tag, and all the nice gadgets that Sparkfun and Adafruit are putting out. If I make use of Bluetooth's signal strength property (RSSI), I should be able to know when someone is within a given distance of a bluetooth master if they are carrying the bluetooth device around with them.

This has been an "interesting" project as I had to learn enough about Bluetooth to know what was possible. Then I had to sort out the BlueZ implementation that comes standard with Linux. And to do that I had to learn about DBus, particularly the Java version. Documentation was rather sparse for Bluez and DBus, so much of this has been banging my head against the table and wall, Googling anything to do with BlueZ, DBus, Java, PropertiesChanged signals, face palming, heavy sighing, elevated stress levels, and all. Fortunately I eventually figured it out and I will talk you through how I did it.

This post will not cover how to use the Bluetooth library I am creating, it will only discuss deciphering the DBus API for the BlueZ bluetooth library and how the code to use it all is written in Java. Later posts will discuss the API I am writing.

The BLE functionality was added into Bluez v5+, so make sure you have version 5 or higher.

If you are interested in taking a gander at the code, it can be found here: https://github.com/smartspaces-io/smartspaces-sandbox/tree/master/services/comm/bluetooth/smartspaces-sandbox-comm-bluetooth2

Do be aware it will be changing as I add more functionality and discover how clumsy some of the APIs will be.

Hardware for Experimenting

There are two kinds of Bluetooth hardware I used for learning the Bluez API.

The first was a TI Sensor Tag. This handy little gadget from Texas Instruments is a collection of sensors, such as accelerometers, gyroscopes, temperature, and humidity. You can get them with a variety of wireless technologies, I bought the one that did BLE. This is easy, you just push a button to make it start advertising, and push the button again to stop it from advertising.

I also used a Raspberry Pi and a Bluetooth 4 dongle. The dongle I got was this one. This is a little harder to use than the Sensor Tag where a button push starts advertising, but the dongle allowed experimenting with much more functionality.

To use the Dongle I had to install the Bluez 5 libraries on a Raspberry Pi. There are some nice instructions over at Adafruit on doing this as part of their directions for installing their Python library for working with BLE (https://learn.adafruit.com/bluefruit-le-python-library). Just be aware you will be compiling C++ code.

The Adafruit python BLE library is nice for experimenting and for learning some about DBus and BlueZ when reading the code. However, it mostly supports working with Bluetooth devices being used as UARTs, which is not my use case. But it provided invaluable information on things like the DBus ObjectManager. Thank you, Adafruit!

Once you have Bluez installed on the Pi, you need the following commands to make everything work.

sudo hciconfig hci0 up
sudo hciconfig hci0 leadv 0
sudo hciconfig hci0 piscan

The first command brings the bluetooth adapter up. The next tells it to start advertising itself on the Low Energy bands. The final makes the dongle discoverable.

DBus

The standard Linux implementation for Bluetooth makes use of DBus, so DBus is where we will start as none of the other code will make sense unless you have some understanding of DBus.

The following will be a lot of words, but it is best to wade through them. Things may not make sense the first time around, but once you go through the section about the Bluez DBus interface, you may find this section makes a lot more sense. I would read through each section at least a couple of times to make sure you understand the pieces.

What is DBus? It is framework found on Linux machines to support Inter-Process Communication (IPC) and Remote Procedure Calls (RPC). I won't discuss the sorts of things it was trying to consolidate and replace, things like CORBA and DCOP, you can find lots of web resources for that.

DBus makes it possible for processes running simultaneously on the same machine to talk with each other very efficiently and in a uniform way. If each process had a different mechanism for communicating with outside processes, it could rapidly get messy. If Process A used one system and Process B another, either the programs running in each would have to be modified to include each other's messaging protocols, or a 3rd process could potentially translate between the separate mechanisms.

An example of DBus usage would be that little icon at the top of your laptop screen that tells you how much battery power you have. The battery service is a DBus service that sends out signals giving the current amount of charge in the battery. It has no idea who is using this information, it just throws it out into the world hoping that someone cares. The program that controls the icon attaches itself to DBus and says it is interested in those battery events. Now, when the battery process sends out a signal giving the battery level, the icon program will receive that signal and update the image you see at the top of the screen.

DBus is a hub and spoke model. There is a DBus daemon that runs on the machine, providing the hub. Each process will create a connection to this hub. This connection is the spoke. For 2 processes to talk to each other, the process sending a message would send it to the hub which would then send it down the spoke for the destination process.

Each process that wants to use a DBus service ends up with at least 1 DBus bus of its own. This bus gives it access to the hub. Each bus has a unique name that is created when the computer boots up and processes start. These bus names have names like :1.15, which aren't particularly readable, but are what DBus uses internally. DBus also supports the concept of well-known names, which allow a program to access services exposed by a process without using names like :1.15 (which will potentially be different each time the system boots up), but rather a more easily remembered name. In the case of Bluez, the well known bus name is org.bluez.

Each process can then expose objects on a bus it owns. These objects will be a collection of methods that can be called remotely from another process. Each object will expose at least 1 DBus interface, an interface specifying a collection of methods, properties, and signals. Examples of DBus interface names are org.freedesktop.DBus.Properties and org.bluez.Device1. A single object can expose multiple interfaces, but the caller will need to specify which interface it wants to call for a given method. These objects are referenced by giving their path. The path is hierarchical and has names like / and /org/bluez/hci0. Only one object can be at each node in the path,

Methods provide functionality. They are like methods you would write in a program, but in DBus the actual method implementation is running in another process, so calling that method is called a Remote Procedure Call.

Properties give information about an object. In the Bluetooth case, properties will say whether a Bluetooth device is paired, if connected, what the address of the radio is, etc.

Signals are initiated by the object which exposes the signals when something happens that another process should know about. For example, a Bluetooth DBus object will raise a signal the first time a new Bluetooth device appears. This is different than a method call, method calls are initiated by a process and go to the process implementing the method, whereas signals are initiated by the process implementing the signal and are sent to everyone who registers to be notified when the signal is sent. The difference is only in who initiates the communication.

Sorting out the BlueZ Dbus API

There are a variety of command line tools for experimenting with DBus. They will allow you to monitor the bus and send messages to DBus objects. These tools are nice, but are usually the most useful if you kinda already know what you are doing. Fortunately, since I had no idea what I was doing, I discovered D-Feet, a tool that gives you a GUI interface for the entire DBus system. You can look at services, click through their entire object hierarchy, find what interfaces and signals they provide. You can also call methods on DBus objects. Using both D-Feet and the dbus-monitor command line program, I was able to gradually piece together how BlueZ was working and how to talk to it to get thing done.

The first thing you should do is make sure you can access DBus and the BlueZ services it exposes without having to be the root user on your machine. The lp Linux group has unfettered access to the BlueZ DBus services, so first I placed myself in the lp group.

sudo usermod -a -G lp keith

Make sure you don't leave off that -a. If you do, you will erase every other group you are in and it will be messy. The -a means add to the groups you already belong to.

For this new group to become effective, you then have to log out and then log back in.

Now install D-Feet. I did it with apt-get.

sudo apt-get install d-feet

Now run it with the command

d-feet

You should see a window like the following.

The BlueZ service sits on the DBus System Bus, so make sure the System Bus tab is selected. The select the org.bluez service. You should now see something like the following. Doubleclicking on the image will make the picture larger.

Here you can see the entire object hierarchy for the Bluez service

• the root of the object hierarchy at /, 
• various managers at /org/bluez, 
• a Bluetooth adapter at /org/bluez/hci0, 
• and a bluetooth device at /org/bluez/hci0/dev_B0_B4_48_BD_D0_83

Let's go through each of these objects.

The Root Object

The uppermost object in the Bluez object hierarchy implements two interfaces org.freedesktop.DBus.Introspectable and org.freedesktop.DBus.ObjectManager. You can see them by opening the / object in D-Feet.

I have no idea, but I think most DBus objects implement the interfaces org.freedesktop.DBus.Introspectable interface. This interface has a single method Introspect, which allows you to get a string containing a complete an XML description of the object. This description will give all interfaces implemented, their method descriptions, all signals provided by the interfaces, and all properties the object has. You will see examples of properties once we open up some of the later objects.

The most interesting object here is the  org.freedesktop.DBus.ObjectManager object. This object is usually found at the root of a service's object hierarchy and provides the ability to access other objects in the hierarchy. The method GetManagedObjects returns a very complex data structure that gives all the other objects in the hierarchy. If you look at the D-Feet output, you can see it returns a data structure of the type

Dict of {Object Path, Dict of {String, Dict of {String, Variant}}}

The outer dictionary uses object paths for its keys. In the Bluez case here, the keys will be /org/bluez, /org/bluez/hci0, and /org/bluez/hci0/dev_B0_B4_48_BD_D0_83. The value for each key in this outer dictionary is another dictionary. This next dictionary will be keyed by DBus interfaces that each object implements, for example org.freedesktop.DBus.Introspectable or org.bluez.Adapter1. The value for each key will be yet another dictionary, keyed by the names of any properties the interface supports, and their value. Notice the type of the value for this innermost dictionary is Variant. This is a DBus type for dictionaries or methods that can return any of a number of types, such as numbers, strings, etc.

Notice that the org.freedesktop.DBus.ObjectManager object also exposes 2 signals, InterfacesAdded and InterfacesRemoved, which are sent when new objects are added to the hierarchy or are removed.

InterfacesAdded is signaled with 2 arguments. The first is an Object Path and gives the DBus object path to the new object that has been handed to the object hierarchy.  The second argument is a dictionary whose key gives one of the interfaces that the object implements and whose value gives a dictionary of all property/value pairs that the particular interface contains. Once again, we will see examples of those properties later.

You can use the dbus-monitor command to see what an actual InterfacesAdded signal looks like. We want to look at the System Bus, so use the following command line.

dbus-monitor --system

The following output from the monitor shows my computer seeing the TI Sensor Tag for the first time.

signal sender=:1.3 -> dest=(null destination) serial=1252 path=/; interface=org.freedesktop.DBus.ObjectManager; member=InterfacesAdded
   object path "/org/bluez/hci0/dev_B0_B4_48_BD_D0_83"
   array [
      dict entry(
         string "org.freedesktop.DBus.Introspectable"
         array [
         ]
      )
      dict entry(
         string "org.bluez.Device1"
         array [
            dict entry(
               string "Address"
               variant string "B0:B4:48:BD:D0:83"
            )
            dict entry(
               string "Name"
               variant string "CC2650 SensorTag"
            )
            dict entry(
               string "Alias"
               variant string "CC2650 SensorTag"
            )
            dict entry(
               string "Paired"
               variant boolean false
            )
            dict entry(
               string "Trusted"
               variant boolean false
            )
            dict entry(
               string "Blocked"
               variant boolean false
            )
            dict entry(
               string "LegacyPairing"
               variant boolean false
            )
            dict entry(
               string "RSSI"
               variant int16 -72
            )
            dict entry(
               string "Connected"
               variant boolean false
            )
            dict entry(
               string "UUIDs"
               variant array [
                     string "0000aa80-0000-1000-8000-00805f9b34fb"
                  ]
            )
            dict entry(
               string "Adapter"
               variant object path "/org/bluez/hci0"
            )
            dict entry(
               string "ManufacturerData"
               variant array [
                     dict entry(
                        uint16 13
                        variant array of bytes [
                              03 00 00
                           ]
                     )
                  ]
            )
            dict entry(
               string "TxPower"
               variant int16 0
            )
         ]
      )
      dict entry(
         string "org.freedesktop.DBus.Properties"
         array [
         ]
      )
   ]

InterfacesRemoved is signaled with 2 arguments. The first is the Object Path of the object being removed from the object hierarchy. The second argument is an array of the names of DBus interfaces that are being removed along with the object.

The /org/bluez Object

Now let's look at the object at /org/bluez. Click on its line in D-Feet, you will see a window like

Here are interfaces for various Bluez services AgentManager1, Alert1, HealthManager1, and ProfileManager1. I have totally sorted out what these are for yet, for now they are here to show you yet another set of interfaces that an object can expose. Notice this object also implements the org.freedesktop.DBus.Introspectable interface.

The Bluez Bluetooth Adapter at /org/bluez/hci0

The next object down the hierarchy is for the single Bluetooth adapter on my machine. You can have multiple adapters on your machine and each will have object paths like /org/bluez/hci0, /org/bluez/hci1, etc.

There are a lot of methods and properties in an adapter interface, so there are 2 screen shots,

and

The first screenshot shows the org.bluez.Adapter1 interface. This interface exposes 4 methods.

The RemoveDevice method will remove a Bluetooth device from the DBus object hierarchy when it is no longer needed. It takes a single argument, the DBus object path to the device to be removed. An example of such a Bluetooth device's object path is /org/bluez/hci0/dev_B0_B4_48_BD_D0_83. This path includes the name of the adapter and the address of the Bluetooth device in a different format than you usually see Bluetooth addresses.

The StartDiscovery method will tell the adapter to start scanning for new Bluetooth devices. StopDiscovery will tell the adapter to stop scanning. Notice neither of these methods have arguments.

The SetDiscoveryFilter method sets a filter for any devices that the adapter will look for when it is in discovery mode.

This is the first interface we have seen with DBus properties. Let's look at some, but not all of them. The UUIDs property gives the set of UUIDs of Bluetooth services that the adapter supports. Discoverable will have a value of true or false and determines if the adapter will be discoverable by other Bluetooth devices. Discovering will be true if the adapter is scanning, and false otherwise. Notice some of these properties are read only, some of them are read/write.

A very important interface that the adapter implements is the DBus interface org.freedesktop.DBus.Properties. This interface contains 3 methods and 1 signal. The method Get lets you read the value of a property for a particular interface, such as the Discovering property mentioned above in the org.bluez.Adapter1 interface. GetAll will return all properties exposed by a given interface and the current values of those properties. Finally, Set will let you set the value of a property on a given interface.

The Properties signal PropertiesChanged is sent every time any of the properties of the object implementing Properties changes. The first argument of the signal gives the name of the interface that exposes the changed properties. The second argument gives a map of the properties and their values for just the properties that have changed. The final argument gives an array of the properties that are no longer available.

As an example, if the StartDiscovery method is called on the adapter, a PropertiesChanged signal will be raised. The interface in the first argument of the signal will be org.bluez.Adapter1 since it contains the property that will be changing. If the only thing that changes is whether or not the adapter is scanning, the map will contain a single entry showing that Discovering is now true. The last argument would be an empty list in this case. The output shown by dbus-monitor for this signal is shown below.

signal sender=:1.3 -> dest=(null destination) serial=1250 path=/org/bluez/hci0; interface=org.freedesktop.DBus.Properties; member=PropertiesChanged
   string "org.bluez.Adapter1"
   array [
      dict entry(
         string "Discovering"
         variant boolean true
      )
   ]
   array [
   ]

The BlueZ Bluetooth Device Object

The final object to look at is an actual Bluetooth device. In my case I was using a TI Sensor Tag with Bluetooth address B0:B4:48:BD:D0:83. Adapter hci0 is the one who found it, so its DBus object path is /org/bluez/hci0/dev_B0_B4_48_BD_D0_83.

Once again there will be two screenshots since there is a lot of interfaces, methods, signals, and properties.

and

The first interface is org.bluez.Device1. This interface has 6 methods. Pair initiates a paring between the adapter and the remote Bluetooth device. The pairing is cancelled with CancelPairing. Connect will connect to the remote device, Disconnect disconnects from it.
ConnectProfile connects to a profile on the device with the given profile UUID, while DisconnectProfile will disconnect from the profile specified.

The properties for the org.bluez.Device1 interface includes the UUIDs exposed by the device, whether or not it is Connected and/or Paired. For my current usage, it fortunately also exposes RSSI, which gives the signal strength of the Bluetooth device.

Notice that the Bluetooth device object also exposes the org.freedesktop.DBus.Properties interface methods and signals, which will work the same as the description given in the Adapter section.

The Java Library

Now that we have a rudimentary knowledge of the Bluez DBus objects, interfaces, signals, and properties, it is time to get all of this into a Java library.

The dbus-java Library

The dbus-java library supplies a pure Java implementation of the DBus protocol. It allows the user to access remote objects, call methods on them, and receive signals from them. It also allows the user to implement their own objects and expose them on the DBus.

If Java isn't your favorite language, there are DBus libraries for most of the popular languages.

Writing DBus Java Interfaces

The DBus Java library works by having Java interfaces and classes defined that extend specific DBus Java interfaces and classes. The DBus library looks at these interfaces and figures out how to encode and decode the remote procedure calls and signals from and to their Java equivalents. Fortunately you just need to implement the interfaces and classes and not have to worry about the complexities of encoding and decoding, you just use objects that the DBus Java library creates that match your interfaces.

First of all, how is a remote interface defined for the org.freedesktop.DBus.Properties interface? The following is the Java interface that should be written for this DBus interface.

package org.bluez.DBus;

public interface Properties extends DBusInterface {
  <A> Variant<A> Get(String interfaceName, String propertyName);
  Map<String, Variant> GetAll(String interfaceName);
  void Set(String interfaceName, String propertyName, Variant value);
}

How did we know this is what the interface should be? We got this by looking at the interface we got from D-Feet, which said the interface was

Get(String interface, String name) -> (Variant value)
Set(String interface, String name, Variant value) -> ()
GetAll(String interface) -> (Dict of {String, Variant} properties)

and that the interface name was org.freedesktop.DBus.Properties. Notice the package for the Java interface is org.bluez.DBus and the interface name is Properties. The fully qualified name of the interface should be the name of the DBus interface.

Notice that the interface extends DBusInterface, an interface supplied by the DBus Java libraries. All DBus interfaces must extend DBusInterface.

Here we wrote out the Java interface by hand. However, the DBus Java library comes with a program called CreateInterfaces that does all this work for us. However, CreateInterfaces wasn't working for me, something about a bad XML DTD, so I gave up and implemented everything by hand. It didn't take that long.

Now, we can't actually use the Java package org.freedesktop.DBus for our package as there is already a class at that location in the DBus Java libraries. The DBus library allows us to place our classes in any package we want, but only if we tell the DBus library the ultimate name we want the DBus interface to have. Here is the same example as above, but placed in a different Java package. We want the DBus interface to be org.freedesktop.DBus.Properties. However, we placed the class in the package io.smartspaces.support.bluez. The name we want the interface to have can be specified with the annotation DBusInterfaceName. You can see this below.

package io.smartspaces.support.bluez;

@DBusInterfaceName("org.freedesktop.DBus.Properties")
public interface Properties extends DBusInterface {
  <A> Variant<A> Get(String interfaceName, String propertyName);
  Map<String, Variant> GetAll(String interfaceName);
  void Set(String interfaceName, String propertyName, Variant value);
}

Finlly we need to define a class for the signal PropertiesChanged. Looking back at the definition of the signal in D-Feet, we see it needs to have the following signature.

PropertiesChanged(String, 
    Dict of {String, Variant}, Array of [String])

The first argument is the DBus interface whose properties has changed, the second is a dictionary of only the properties whose values have changed and their new values. The final argument is a list of properties that are no longer available for the interface.

The signal classes in the DBus Java library must extend DBusSignal. They also need to call the superclass's constructor with all of the arguments in the signal. Notice the class below has an extra argument, path, that isn't in the definition of the signal we saw in D-Feet. This argument will give the DBus object path for the sender of the signal, for example path will have the value /org/bluez/hci0 when the StartDiscovery method is called on object /org/bluez/hci0 and /org/bluez/hci0 sends the PropertiesChanged signal.

public static class PropertiesChanged extends DBusSignal {
  private final String iface;
  private final Map<String, Variant> propertiesChanged;
  private final List<String> propertiesRemoved;

  public PropertiesChanged(String path, String iface, 
      Map<String, Variant> propertiesChanged,
      List<String> propertiesRemoved) throws DBusException {
    super(path, iface, propertiesChanged, propertiesRemoved);

    this.iface = iface;
    this.propertiesChanged = propertiesChanged;
    this.propertiesRemoved = propertiesRemoved;
  }

  public String getIface() {
    return iface;
  }

  public Map<String, Variant> getPropertiesChanged() {
    return propertiesChanged;
  }

  public List<String> getPropertiesRemoved() {
    return propertiesRemoved;
  }
}

This then needs to be placed inside of an interface that gives the DBus interface that emits the signal. So for the org.freedesktop.DBus.Properties interface and its PropertiesChanged signal, we need the following complete Java interface definition.

package io.smartspaces.support.bluez;

@DBusInterfaceName("org.freedesktop.DBus.Properties")
public interface Properties extends DBusInterface {
  <A> Variant<A> Get(String interfaceName, String propertyName);
  Map<String, Variant> GetAll(String interfaceName);
  void Set(String interfaceName, String propertyName, Variant value);

  public static class PropertiesChanged extends DBusSignal {
    private final String iface;
    private final Map<String, Variant> propertiesChanged;
    private final List<String> propertiesRemoved;

    public PropertiesChanged(String path, String iface, 
        Map<String, Variant> propertiesChanged,
        List<String> propertiesRemoved) throws DBusException {
      super(path, iface, propertiesChanged, propertiesRemoved);

      this.iface = iface;
      this.propertiesChanged = propertiesChanged;
      this.propertiesRemoved = propertiesRemoved;
    }

    public String getIface() {
      return iface;
    }

    public Map<String, Variant> getPropertiesChanged() {
      return propertiesChanged;
    }

    public List<String> getPropertiesRemoved() {
      return propertiesRemoved;
    }
  }

As an aside, it took me quite some time to make signals work. The first argument of the PropertiesChanged signal was String. When I looked at examples of signals in the DBus Java documentation, they all had their first argument as a string, so my signal class had the path, propertiesChanged, and propertiesRemoved arguments only. I never received an instance of the signal. I spent a long time Googling how to register signal handler, particularly the PropertiesChanged signal and never found anything at all. Eventually I noticed the interface argument in the dbus-monitor output and noticed it was a DBus interface name, not an object path that all of the online examples showed. Once I added the interface argument, I started receiving the signals.

The DBus Connection

Once we have our interfaces and signal classes, the next thing needed is a connection to the DBus daemon. You get this by instantiating a DBusConnection object. Look at StandardBluezBluetoothProvider in my Java code and you will see a line

dbusConnection = DBusConnection.getConnection(DBusConnection.SYSTEM);

This gets a connection to the DBus System bus, which is the bus that contains the Bluez service. You will need the connection before you can do anything.

Getting a Remote Object

Now that we have a connection, we can create a local object that looks like the remote object in the other process. We do this with the getRemoteObject() call on the DBus connection.

ObjectManager bluezObjectManager = 
    dbusConnection.getRemoteObject("org.bluez", "/", ObjectManager.class);

This get a connection to the DBus ObjectManager that is found at /, the root of the Bluez object hierarchy. The first argument gives the well-known bus name org.bluez for the Bluez service.

As I said, this Java object now looks like the actual DBus ObjectManager. For instance, it has a method called GetManagedObjects() that calls the GetManagedObjects method on the remote object and returns the results as Java data structures to our Java program.

Map<Path, Map<String, Map<String, Variant>>> values =
    bluezObjectManager.GetManagedObjects();

These calls will block until they return from the remote object. If you don't want to wait for the result you can use an asynchronous response, either by periodically checking a method result object periodically or by getting a callback when the response returns.

Next let's register some signal handlers. For example, let's register one for the InterfacesAdded signal from the ObjectManager.

Unfortunately the version of the DBus Java library does not allow you to use service well known names like org.bluez to register your signals, you must use the unique DBus address, something like :1.15. Unfortunately this can be different on different machines or different when your machine boots up, so we need to use the well-known name. Argh!

Fortunately DBus provides a service that let's you find out what unique bus name the DBus daemon assigned to the org.bluez service.

DBus dbus = dbusConnection.getRemoteObject(
    "org.freedesktop.DBus", "/org/freedesktop/DBus", DBus.class);
bluezDbusBusName = dbus.GetNameOwner("org.bluez");

Once we have the Bluez service bus' unique name, we can register the signal handler for InterfacesAdded.

DBusSigHandler<ObjectManager.InterfacesAdded> interfacesAddedSignalHandler =
     new DBusSigHandler<ObjectManager.InterfacesAdded>() {
  @Override
  public void handle(ObjectManager.InterfacesAdded signal) {
    System.out.println("Interfaces added");
  }
};
dbusConnection.addSigHandler(ObjectManager.InterfacesAdded.class, 
  bluezDbusBusName, bluezObjectManager, interfacesAddedSignalHandler);

The interfacesAddedSignalHandler.handle() method will now be called whenever Bluez raises an InterfacesAdded signal. Here we are just printing at an interface has been added.

Now we can call GetManagedObjects and also get notified about InterfaceAdded signals without having to ask every time a new object or set of interfaces to an existing object are added.

Conclusion

That was a lot!

We covered DBus and its basic concepts. We looked at various objects exposed via DBus from the Bluez API. And we looked at the DBus Java library and how it can be used to access the Bluez objects and put Bluetooth functionality into our Java program.

There is still much to do on this library, I want to be able to register listeners for things like new devices coming into range of the bluetooth radio on the computer, the RSSI of a device as it moves around in a space and other things. But this was a good start. Stay tuned for how to actually use the library, at which point you won't need to think about BlueZ and DBus at all.

Announcing the Smart Spaces Project

The fork of the Interactive Spaces project is complete and I spent a very long time removing every use of the word "interactive" and replacing it with the word "smart".

Smart Spaces is now available at https://github.com/smartspaces-io/smartspaces.

Smart Spaces will not be just concerned about interactivity in a physical space, but also smarts. It is time to get interesting learning algorithms and rules based systems into the package. I also want to increase the abstraction level for talking about events happening in a physical space, both in detecting them and also affecting them. I'm not entirely sure what this will all mean, but it will be fun figuring it out.

The initial effort was merely changing the root package name from interactivespaces to io.smartspaces. I imagine I will be removing instances of interactive for some time.The next major project will be removing all deprecated APIs from the Interactive Spaces days and adding the ability to use MQTT for activity to activity communication, routes in Smart Spaces parlance.

Despite the low initial version number, 0.1.0, this is feature complete to Interactive Spaces 1.8.2 (1.8.1 runs in production in the Google Experience Centers) and is ready for deployment. I just numbered low as I want 1.0.0 to have some of the things that were wanted in Interactive Spaces for some time, such as ROS out of master/space controller communications.

The discussion board is found at https://groups.google.com/forum/#!forum/smartspaces. 

Join us on this adventure.

Forking the Interactive Spaces Project

I was the initial creator of the Google Interactive Spaces project. I have really appreciated the opportunity Google gave me to develop this project and use it in the production environment of the Google Experience Centers. I learned so much and coming to work was so fun. But now it is time for a change.

I am leaving Google on February 2nd to pursue a startup. I will be forking the project to continue its evolution, the new project will be called SmartSpaces. I want to move it towards the ability to create intelligent spaces, where interactivity is only part of the picture. The code will be found at https://github.com/smartspaces-io. If you wish to follow the new project I will be discussing it here and on the discussion group smartspaces@googlegroups.com.

The code will stay in the interactivespaces namespace until February 2, after that I will change all the package names. I also plan on cleaning up a lot of the inconsistencies in the APIs that appear after 3 years of evolution.

Greenhouse Phase 5: XBee On Sensor Board

The first several articles in the Greenhouse Series were about

• Solar Power
• Temperature and Humidity Sensors
• The Sensors Finally Working
• Data To The House

In Data To The House we talked about XBee sensor meshes and configured a couple of XBee radios to get data from the greenhouse into the house. In this article, we will 

• wire an XBee Series 2 radio to the greenhouse sensor board
• modify the Teensy sketch to send the data from the sensors to the XBee on the sensor board
• write a small Python script to read the sensor data from the Coordinator radio

The Greenhouse Sensor Board

The Greenhouse sensor board needs to have the XBee radio added and the Teensy sketch modified to send data through the sensor board radio to the Coordinator radio.

Wiring in the XBee Radio

Wiring the XBee radio to the sensor board is very easy. The XBee radio uses standard serial communication to talk to a host processor. The RX and TX pins on the radio will be attached to TX1 and RX1 respectively on the Teensy. RX1 and TX1 are the first hardware-based serial pins on the Teensy 3.1. The serial connection and power are all that are needed. The XBee is in the lower right of the image below.

The current version of the board then looks like this.

This is a more up close shot of the board.

Originally I thought I was going to have to use an XBee Series 2 Pro radio to get data all the way across the yard from the greenhouse to the house, going through outer walls of both buildings and th Coordinator possibly in the basement of the house. The Pro is 63 mW of power and has a distance of about 1 mile, while the standard radios are 2 mW and good for about 400 feet. A quick test with a 2 mW radio got signals from inside the closed greenhouse into the basement of the main house, so I stayed with the 2 mW radio. This was a relief as the 2 mW radio uses up only 40 mA of power, while the Pro uses 295 mA and I would have had to build an external power board as the power regulator on the Teensy couldn't have powered the Pro.

Modifying the Teensy Sketch

The code running on the Teensy must now be modified to use the radio. The code can be found in the github repository here.

First make sure you have Andrew Rapp's XBee library for Arduino installed on your machine. At some point I will probably write a new Arduino XBee library as Andrew's code is GPL. Though I fully support people using the Gnu license, I much prefer the Apache 2 license and so will most likely write a library that will be licensed as such.

First you will need to include the XBee.h file in your sketch.

#include <XBee.h>

The sensor data will be sent as an XBee TX packet. TX packets allow arbitrary data to be sent between radios and are received as an RX frame on the destination radio. Adding the following lines will create an XBee radio object and some other data structures for the XBee TX packet.

XBee xbee;

XBeeAddress64 addr64 = XBeeAddress64(0x00000000, 0x00000000);

ZBTxRequest zbTx = ZBTxRequest(addr64, (uint8_t *)&sensorData, sizeof(sensorData));

The variable addr64 contains the 64 bit address of the radio that will receive the sensor data packets. Typically I use the Coordinator radio to be the main destination for sensor data, so here set the destination address to the special address 0000000000000000 for the Coordinator. You could also use the exact address for the Coordinator by looking at the underside of the radio, but by using the special address you can switch out your Coordinator radio for another radio and the Teensy code would not have to change.

The variable zbTx creates the TX request packet to be sent by the XBee. It uses our address for the Coordinator Radio. The last argument sizeof(sensorData) will give the number of bytes of sensor data to be transmitted. This way we can add or subtract data from the packet and not have to worry about counting bytes on the Teensy side.

The second argument (uint8_t *)&sensorData needs some explanation. The ZBTxRequest constructor requires a pointer to an array of bytes containing the data to be transmitted to the destination radio, an expression of type uint_8 *. sensorData contains our data, but it is a bunch of float data. How to we get those float values into an array of bytes? A lot of people create a byte array for the TX packet and use various tricks to get the bytes from the floats into the byte array, but that is way too much work. Can we do better?

Let's remind ourselves of the sensor data data structure.

typedef struct _SensorData {
  float temperatureInside;
  float humidityInside;
  float temperatureOutside;
  float humidityOutside;
  float altitude;
  float barometricPressure;
} SensorData;

SensorData sensorData;

When sensorData is laid out in the memory of the Teensy, the first 4 bytes will contain the value of temperatureInside, the next 4 bytes will contain the value for humidityInside, the next 4 temperatureOutside, and so on. Whatever order the fields are found in the struct will be the order of the values in the Teensy memory.

Though it will not matter in this article, the float values are laid out in memory lowest order byte first. This is called little-endian. Knowing the byte order will matter when we start processing the data in Java, but won't matter for the Python script.

Going back to our second argument, the expression &sensorData will give us the memory address of the lowest byte of the sensor data. Just to reinforce, this byte will be the low order byte (remember little-endian) of temperatureInside. However, the type of this pointer will be SensorData * and we need a pointer type of uint_8 *, so we use the type coercion (uint_8 *).

The program then places data in the structure as before. It doesn't matter what order we make the assignments in, the only thing that matters as far as the radios are concerned is the order of the fields in the struct.

Initializing the xbee object is pretty easy. First, the serial object needs to be initialized. Since the TX1/RX1 pins are being used on the Teensy for the XBee, we need to use the Serial1 object.

Serial1.begin(9600);
xbee = XBee();
xbee.begin(Serial1);

The final new line sends the actual packet to the destination radio.

xbee.send(zbTx);

Let's look at the output coming across the USB serial connection for the Teensy. Once the Python script is written, we will want to confirm that the values being output here are the same being output by the Python script.

So far so good!

Reading the sensors after 1 second pauses is probably too frequent. When the board is finally deployed, I will probably sample every 10 minutes, which means that the SLEEP_DELAY constant at the beginning of the file should be set to 60000. But waiting 10 minutes while debugging the system is painful, so for now the value is 1000 for those 1 second pauses.

Checking Radio Communication

It is always best to test in as simple an environment as you can. Now that the sensor board has its radio, let's use XCTU to see if the board is communicating with the Coordinator radio before writing the Python script to read the data.

If you remember the last article, we talked some about Networking mode in XCTU. This mode lets us look at all of the radios in the mesh and see who can talk to whom. Let's use that now.

First  I placed the greenhouse sensor board in the greenhouse and plugged it into power. Then I went back into the house and plugged the Coordinator radio into a Sparkfun USB Explorer and plugged the USB cable into my laptop and set up XCTU to look at the radio. A quick check in Networking mode (remember to hit the Scan button) showed the radios talking to each other.

I then placed another XBee Router next to the window in the main house and carried my laptop into the basement, expecting to see the greenhouse radio transmitting through the Router radio by the window which would then communicate the sensor packets to the Coordinator radio. But if you look at the picture below, you can see that the greenhouse radio is talking directly to the Coordinator radio despite the distance from the house to the greenhouse, 2 outer walls, and the distance into the basement.

Not bad!

If you remember in the last article we talked about XCTU's Consoles mode which allows us to look at the packets coming into a radio. I switched over to the Consoles window and clicked the Open button. You can see the XBee API RX packets coming into the radio from the greenhouse radio. They are the red lines in the center section of the picture below. They are labeled Receive Packet and have a length of 36 bytes.

There is a window along the right side of the Consoles window that shows the actual contents of the packet. If you scroll this window to the bottom and hit the Hex tab, you can see the sensor data coming through.

Receiving the Data from the Coordinator Radio

Now that we know data is coming into the Coordinator radio from the sensor radio, it is time to write a Python script to show us the data. The complete script can be found here on github.

The first thing to do is install the Python XBee library on the computer you want to run the script on.

$ sudo pip install xbee

First we need the Python object that interfaces to a Series 2 radio in API mode. This is done by importing the ZigBee package from the xbee Python library.

from xbee import ZigBee

Next we will create a serial connection to the Sparkfun USB Explorer and hand this serial connection to the ZigBee object.

serial_port = serial.Serial(serial_port_name, SERIAL_BAUD_RATE)
 
xbee = ZigBee(serial_port, escaped=True)

The ZigBee object can now be polled to see if a new API frame has been read. The wait_read_frame() call blocks until a new frame is read.

Once a frame is received, we will first check to see if it is from the greenhouse radio. The reason for this check is that eventually all radios will be sending TX packets to the Coordinator and we need some way to tell which radio has sent a particular packet. This can be handled in several ways. We could preface each packet with an identifier saying what set of sensors the data is from. Or we could just know the address of each radio, assuming each radio only sends one type of data packet. I chose in this case to go with the latter.

The frame that the ZigBee class creates is a dictionary with multiple fields in it, including the address of the source radio of the TX packet, as well as the data. To get the 8 bytes of source address, we use the expression frame['source_addr_long']. We take these 8 bytes and create a hexadecimal string for easy comparison with the address supplied as a command line argument when we start running the script.

If we find that we have a packet from the greenhouse radio, we then need to decode the binary data in the packet to the series of floats that were sent. We use the expression frame['rf_data'] to get this data.

To read the data we need to know what order the floats are in the packet. Looking back at the struct in the Teensy code we know the order is

1. The inside temperature
2. The inside humidity
3. The outside temperature
4. The outside humidity
5. The altitude
6. The barometric pressure

We can read these values from the TX data by specifying the start location and the number of bytes to use for a data value. In our case, every value being sent is a float and a float is 4 bytes.

inside_temperature = struct.unpack('f',rf_data[0:4])[0]
inside_humidity = struct.unpack('f',rf_data[4:8])[0]
outside_temperature = struct.unpack('f',rf_data[8:12])[0]
outside_humidity = struct.unpack('f',rf_data[12:16])[0]
altitude = struct.unpack('f',rf_data[16:20])[0]
barometric_pressure = struct.unpack('f',rf_data[20:24])[0]

Notice the inside temperature is first in the packet with its first byte at position 0 in rf_data, so we look at rf_data[0:4]. The inside humidity is second in the packet with its first byte at position 4, so we need to look at rf_data[4:8]. The rest of the values follows the same pattern.

The script can be run with something like the following command. For me the Sparkfun USB Explorer had ended up at /dev/ttyUSB0 and the greenhouse sensor radio has the 64 bit address 0013a200407bd2e6.

$ ./GreenhouseSensors.py /dev/ttyUSB0 0013a200407bd2e6

The screenshot below shows the data being output by the script.

Conclusion

At long last there is finally data making it from the greenhouse into the main house. You have seen how easy it is to create and process XBee packets both on the Teensy and in a Python script. Ultimately I will write an Interactive Spaces activity to process the data, but for now the Python script is enough. I will also soon modify the Python script to send the data up into the time series database in the cloud as discussed in the article Sensor Data To The Cloud: Part 1.

Greenhouse Phase 4: Data To The House

The first several articles in the Greenhouse Series were about

• Solar Power
• Temperature and Humidity Sensors
• The Sensors Finally Working

Now it would be nice to get the data from the sensors in the Greenhouse into the main house for capture, processing, etc.

To do this, I decided to use Series 2 XBee radios. These powerful little radios are very easy to interface to and have a very nice feature, they support mesh networking. A lot of people look for cheaper radios, XBees are not the cheapest in the world, but these little radios have a whole lot of features and, if you need these features, spending some extra money rather than implementing those features yourself can be the right decision to make.

XBees

Let's start off by talking a little bit about Series 2 XBee radios.

What Is Mesh Networking?

So what is mesh networking? Mesh networking is useful in situations where you have a bunch of radios that need to talk to each other, some are too far away from each other to communicate directly, or perhaps there is a wall in the way that doesn't let the signal thorough. Also, sometimes noise prevents a signal from getting through, or something breaks. So how can we have a reliable network that makes sure that all data is received?

Suppose you have radios A and B, and A wants to talk to B, but B is too far away from A, or there is a wall in the way or something that means that the connection won't happen. One way of handing this is to have a 3rd radio C that is within range of both A and B. A could send its message for B to C and C could then forward the message to B, A -> C -> B.

One issue here is if C stops working for some reason, suddenly A can no longer talk to B.

So suppose we have a radio D that is also reachable by A and by B. It doesn't matter if C can talk to D or not, just that A and B can talk to D. But say C is working. Now there are 2 different routes from A to B, either A -> C -> B, or A -> D -> B. If radio A finds that C isn't working, it can route its message for B through D. Of course, if both C and D are not working for some reason, once again there is no way for A to talk to B.

It is possible for these chains to be longer. Each additional radio in the chain is called a hop. Say radio A can't reach B, but can reach C. C can't reach B, but can reach E. And suppose E can reach B. Now A can send a packet to B via the path A  -> C -> E -> B. This path has 3 hops.

So radios don't necessarily have to be able to talk directly to each other to transmit information to each other, they can use intermediate radios. Not only that, but they needn't use the same path each time. Some radios in a path that used to work may not be available at any given time, but if there is a path of radios who can talk to each other, the packet can eventually get from A to B. This is the basis of mesh networking and is similar to how the Internet works.

Other XBee Concepts

There are a few more concepts useful for understanding XBee radios. Every mesh has to have 1 radio that is the Coordinator. A mesh can have only one Coordinator. This radio controls the network. It determines the ultimate local addresses that each radio in the mesh gets. Radios have 2 addresses, one short one local to the mesh the radio is part of, and a much longer one that is unique to the radio no matter what mesh it is in.

Another radio type is a Router. Router radios can act as intermediaries and radios who cannot talk directly to each other must go through one or more Routers.

Finally there can be End Devices. These End Devices can be battery powered and lower powered. They can go to sleep and turn themselves back on. They can talk to Routers or Coordinators, but cannot talk to each other. This limited functionality means they can be lower cost or can work on battery power for very long periods of time because they can sleep between transmissions.

An example mesh network might look like below.

One more important concept is that of the PAN (Personal Area Network) ID. All XBees must have a PAN ID, and all radios that are going to communicate with each other must have the same PAN ID. The PAN ID is determined by the Coordinator, and can be manually set or automatically created by the Coordinator. Finally, you have have multiple independent XBee networks in the same physical location, each with its own PAN ID. These networks will not be able to communicate with each other, only radios with the same PAN ID can communicate with each other.

In this post we will set the PAN ID manually.

The XBee Hardware

You can use the XBee hardware in several ways.

You can talk to it over a serial connection and control it from a host computer or microcontroller. This style of usage gives you the most flexibility, but as it requires some programming, it is also the most complex usage. The radios have standard RX and TX pins for communication.

It is also possible to wire digital and analog sensors directly to an XBee, eliminating the need for a host computer or microcontroller. In this style of usage, circuitry and programming are minimized. It is also possible then to make the radio sleep for the intervals between sensor samples, which means that a battery powering the XBee could last a very long time The number of sensor pins is limited, but if this is all you need, it is a great option for the simplicity and long battery life.

Configuring the XBees

XBee radios are configured with software from Digi, the manufacturer of XBee radios, called XCTU. When I first started using XBees, XCTU only ran on Windows machines, which meant I had to use virtual machines or Wine on my Linux box. But Digi have since rewritten the program in Java and it is now possible to run it on all major OSes.

The program is really nice now. Not only does it work on all OSes, but it also has a much improved user interface. It also has new functionality that is very useful for debugging, you can visualize your entire XBee network and see which radios can talk to which other radios, transmit data to any radio by typing the data into the UI. All in all, XCTU has become a very nice tool.

So jump into your favorite search engine and search for "digi xctu download" and pick the version appropriate for your computer and follow the directions for installation.

My Initial Network

The wireless network I build will eventually cover both the greenhouse and the house itself, but this post is about getting data from the greenhouse into the house. To do this I will use 3 XBees. Two will be XBee Pros. These radios have a lot more output power and I figured they would be good for getting through the outside walls of the greenhouse and the house and cover the distance between the 2 buildings. They might also be overkill, but I thought I would give them a try. The Coordinator Radio will be a standard XBee in the house and doesn't have to be as high powered.

Hardware for Configuring

It is necessary to have a serial connection to the radios to program them. To help with this, I bought several Sparkfun XBee Explorers. These little boards are awesome. You plug the radio into the board, hook up a USB cable and you are working with them in no time. I often hook the Coordinator Radio up to a host computer using an Explorer in production.

The Radios are blue and the Explorers are red in the picture below. You may also notice tape on the antenna for the radios. I usually use tape to mark the PAN ID and whether the radio is a Coordinator, Router, or End Device when building a network so I know which is which.

Running XCTU

Running XCTU is easy. If you follow the directions on the Digi site on Linux, it will install the software in /opt/Digi/XCTU-NG. I have not yet puzzled through how to set permissions so that I can use XCTU without running as root, for now I use the command

sudo /opt/Digi/XCTU-NG/app &

This will run the program as root and run it in the background so that I can do other things in the same terminal window.

The initial window you should see is the following.

Now we need to scan for the radios. First I want to program the Coordinator radio, so I labeled the Coordinator Radio with tape that had C written on it, plugged it carefully into the Sparkfun USB Explorer, and then plugged the Explorer into my computer.

If you look at the initial XCTU window, you will see an image kinda shaped like an XBee with a magnifying glass next to it. It is the second large icon from the left just under the menu bar. Click this and you will see the following dialog box.

On my machine the Explorer got assigned /dev/ttyUSB0, so I click the checkbox and hit Next >.

The next dialog box will ask me how I want to talk to the XBee, and will let me pick serial properties like baud rate, number of stop bits, etc.

The default baud rate for XBee radios is 9600 baud, in fact all the values seen are the defaults, so just clock Finish.

Another dialog box will pop up and give messages about its search for radios on the port given. If everything is wired property and you left the connection values  alone you should eventually see the following dialog box.

Select the checkbox next to the radio and click Add selected devices.

You should then see a window like below. To be honest, this window is from after I made the radio a Coordinator, so your picture will be slightly different. But that won't matter at all.

Now double click on the radio entry in the left hand pane. This will bring up data in the right hand pane about the radio.

Configuring the Coordinator

The radio plugged in now is to become our Coordinator. If you look over the above window, you will see that the radio is currently a Router. I have found that most brand new radios I buy have the Router AP firmware in them. That's fine, XCTU makes it pretty easy to change. There are a series of 5 large icons in the right pane labeled things like Read, Write, Update and others, and we will use Update.

Before we continue, let's discuss what is going on. XBee Series 2 radios are basically small computers and each radio can either be a Coordinator, a Router, or an End Device. What determines the type of radio is the firmware uploaded to it. Right now the radio seen in that last window shown has the Router AT firmware and so is a Router, and we want it to contain the Coordinator API firmware so that it will be a Coordinator. We won't worry about the different between AT and API in this post, we need API. So click the Update button to see the following dialog.

Make sure the product family is selected as XB24-ZB. If this isn't chosen, you have the wrong kind of radio and should go make sure you order a Series 2 radio. Scroll through the middle list until you find ZigBee Coordinator API. As for the Firmware version, I always make sure I chose the latest firmware. I have never tried to find out if my Coordinator, Router, and End Device radios are running the latest firmware, but it probably is safer to make sure they are always running the latest at the same time.

Then click Update and wait.

Once the update is done, XCTU will read the values from the radio again and you should see the following.

Notice the function set for the radio is now labeled as Coordinator, though if your window is like the picture I have, you will see a ... and some of the letters of Coordinator. Rolling the mouse over this area will give a tooltip giving the function set name without the ... in the middle of the text. If it is not ZigBee Coordinator API, trying doing the update again, you didn't select the correct firmware.

Now we need to set the PAN ID. Remember every radio that needs to talk to each other needs to have the same Pan ID. For this post, we will make the Pan ID have the value 2000, though you can pick any value you want as long as all the radios get the same value. Scroll through the radio parameters until you see PAN ID. You probably won't need to scroll at all, it is the top value. Type 2000 into the text box.

Next, the software libraries that will be talking with the XBee are using Escaped Mode. Without going into too much detail, this mode makes it easier to detect XBee radio frames, so many software libraries use it. To use Escaped Mode, we need to tell the radio to set its configuration parameter AP to the value 2.

Now we need to write these values into the permanent memory in the radio so that they are kept when the radio is turned off and turned back on. If you look at those large icons again in the right hand pane, you will see Write. Click that button.

If you want to confirm that your values are set correctly, click the Read button and scroll through the window to make sure PAN ID and AP have the proper values.

Cool, now we have a Coordinator Radio to rule our network!

Configuring the Greenhouse Radio

Now it is time to configure the XBee that will go in the greenhouse. I decided to make this radio a Router for no particularly good reason. My general rule of thumb is that if the radio is not going to go to sleep between sensor measurements and will not be battery powered, I usually make it a Router to have the most flexibility.

Since my greenhouse is across the lawn, I chose an XBee Series 2 Pro, which has a higher power output than a standard XBee Series 2.

I labeled the greenhouse XBee with a piece of tape with the letter R to label it as a Router, carefully seated it on a second Sparkfun USB Explorer, and plugged it into my computer. I then clicked the Search Radio icon on the left side of XCTU and got the following dialog.

Notice the additional serial port /dev/ttyUSB1. Click Next >. You will again see the Port Values dialog where you can chose baud rate, stop bits, etc. Just click Finish. After a little while you will see the following dialog.

Make sure both radios are checked and click the Add selected devices button. You will now see two radios in the left panel of the XCTU window.

If you are looking at these images carefully, the above picture is not exact. This is actually from after the radios were configured, so the radio already has the Router API firmware. But ignore this for now, and imagine it claims the radio is Router AT, which is how the radios are usually set up when bought brand new.

Now double click on the new radio, which is below the Coordinator radio in the above picture, and get to its main screen.

First off is to update its firmware to be XBee Router API by clicking the Update button.

After the firmware update finishes, set its PAN ID to 2000.

Also set its API Mode to Escaped Mode, which is done the same way as the Coordinator Radio, setting the AP configuration parameter to 2. Then click Write.

Confirm the values by clicking Read and scrolling to check PAN ID and AP.

Are The Radios Communicating?

One of the features in XCTU that I really like that the previous version didn't have is the ability to look at the network of radios and see who talks to whom and exactly what packets they are sending and the contents of those packets. You can also construct packets and have the radio send them. All of this really helps with debugging.

To get into this mode, look at the toolbar immediately under the menu bar and click the rightmost icon. As you mouse over it the tooltip will read Switch To Networking Mode. This will show the following window.

Now click on the leftmost icon in the righthand pane, it is labeled Scan. If the radios are properly configured and powered up, you will see a window like the following.

In this window, the Coordinator radio is on the right and the Router is on the left. You can see that they talk directly to each other by the line between them.

If your network were more complicated, you would see more radios and who is talking to whom by the line between the various pairs of radios.

The mode you are in now shows the network as a visual graph. You can also see a table of radios by clicking the Mode button next to the Scan button and selecting Table. You will then see the following.

By clicking on the Connections dropdown on the right hand side of each radio row you can see a list of the radios that radio is connected to.

Consoles Mode

As stated earlier, you can also connect to a radio and tell it packets to send. Though we won't go into much detail here, I will at least show the window. The Consoles Mode is entered by clicking the button just to the left of the Networking Mode button. If you mouse over the button it will show Switch to Consoles working mode. Clicking on this will show the following.

Some day perhaps I will talk about how to use this window.

Conclusion

So now we have a couple of XBee radios configured to get data from the greenhouse into the house network for storage and processing. We have discussed some of the basics of XBee Series 2 radios and mesh networking. You have seen how to use Digi's XCTU tool for configuring radios and for looking at who is talking to whom. I also set up another Pro radio to sit between the greenhouse and the Coordinator, but have not shown that radio as it is configured the same way as the greenhouse Router.

In the next installment of this series we will wire the Greenhouse XBee Router to the sensor platform built before and write the software to capture the sensor data, get it into the house, and then upload it to a time series database in the cloud.

Be seeing you!