Integrating Apache HBase (Big Data) with Arduino Uno (IOT)

For some time I had been blogging about the intersection of Big Data and IOT. The combination of Cloud, Big Data and IOT provide a lot of opportunities. In this blog we will look on how to integrate Apache HBase with Arduino Uno. HBase is a NoSQL database of columnar type. HBase has got some nice documentation here.

Adding Ethernet Shield to Arduino Uno

arduino-uno-ethernet-shieldArduino Uno by itself doesn’t come with a facility to connect to the internet. So, an Ethernet Shield has to be added on top of the Arduino Uno board. Here we see the Arduino Uno connected to the USB cable on the right and the Ethernet Shield connected to RJ45 network jack on the left. Now the Ethernet Shield has to be mounted on the top of the Arduino Uno for gaining the network capabilities.arduino-uno-ethernet-shield

Shields are expansion boards to provide additional functionality to the Arduino Board. More than one shield can be stacked on each other. Many shield are available in the market. A custom shield can also be built using the Arduino Protoshield.

In the coming blogs, we will look at the Arduino Motor Shield and build some cool things using it.

Starting the HBase and the REST service

HBase provides different interfaces (Java – REST – Phoenix – Protobuf) for the DDL/DML operations. In this tutorial the REST interface will be used. After starting the HBase, the REST server which is part of the HBase installation has to be started. More about the HBase REST interface here and 1, 2, 3.

The REST interface runs on port 8080 and uses the HTTP verbs like POST, PUT, GET and can be  invoked from the browser as shown below. The schema for the table mytable is displayed in the browser here. Because of the language agnostic behavior of REST, applications can be built using any language on top of HBase.

browser-rest-call
Creating a table in HBase and querying it in Arduino Uno

Now a HBase table has to be created using the command create ‘mytable’, ‘metrics’ from the HBase shell. Once the table has been created, the same can be accessed from Arduino Uno. It should be made sure that the Arduino Uno and the HBase are within the same LAN.

hbase-shell

Now that the table has been created,  the Arduino sketch has to be executed to make a REST call. Here (GET, POST) are the complete Arduino Sketch. The HBase-Arduino-GET.ino Sketch is getting the schema for a table in HBase and the HBase-Arduino-POST.ino Sketch is for inserting a row in HBase. Note that the code is not optimized, but is good enough to know the basic concepts and also to get started.

The bottom screen is the Arduino IDE for developing/compiling/deploying the  Arduino Sketches. Bottom right in the same screen is the Arduino Serial Monitor which shows the output of the HBase-Arduino-GET.ino Sketch. Note that the Arduino Sketch does a simple HTTP GET operation to get the schema for a particular table.

arduino-hbase-output

Similarly, the HTTP POST/PUT can be used to put some data in HBase. HBase stores the data as Base64, so the input data has to be converted into Base64 before insert it. The arduindo-base64 library can be used to encode/decode to Base64.

Conclusion

In this blog we have seen how to integrate the Arduino Uno directly with the HBase REST Server, which is a bit restrictive approach because Arduino Uno can’t process much and also there aren’t too many libraries for the Arduino Sketch when compared to languages like Python. Arduino Uno can also be interfaced with an intermediate (through pySerial or Processing) and the intermediate communicates with the HBase REST Server. We will look into the pySerial and the Processing approach in a future blog.arduino-hbase-interfaceIntegrating Big Data with IOT directly  was fun, but had not been straight forward. Big Data and the IOT exist in their own space, may be down the line the different m2m platform vendors will provide some sort of connectors to the Big Data space to make the connection between the Big Data and the IOT worlds a bit more smoother.

In the coming blogs we will look at how to get the ambient temperature from a temperature sensor and put it into HBase at regular intervals.

Leave a Reply

Your email address will not be published. Required fields are marked *