I had been assisting a (now defunct) IoT startup in identifying bottlenecks in their data ingestion and aggregation process when I realised that a lot of the data processing could be managed by underutilised "Things", mainly Intel Edison devices used across the deployment. I started to examine how this could be implemented and managed as part of a new IOT ecosystem.
HIP (Hierarchical IOT Protocol) is a communication protocol designed to increase the scalability and interoperability of large scale IOT deployments and includes specifications for both Platform Side (e.g. SaaS solution) and Deployment Side (e.g. sensors and actuators in a "smart building") messaging. Core concepts within the protocol are Aggregators - which allow for "deployment-side" data processing, Commanders and Delegators (previously referred to as Brokers), which provide local (again,deployment-side) control of actuators and Health messaging which can be used in conjunction with Configuration messages to identify over and under utilised devices and spread roles across them.
Aggregators provide for a Hierarchical structure across a deployment, where relevent data from one or more sensors can be processed locally before being uploaded to the Platform. In theory, this enables data from unlimited numbers of sensors to be processed and fully addresses the practical issues within the scope of Fog computing. (Fog Computing refers to extending cloud computing to the edge of an enterprise's network)
The scope of the protocol includes all aspects of the configuration and management of a deployment. Health messages can be sent from devices to the platform to identify potential bottlenecks while Configuration messges can be used to transfer "roles" from under resourced device to devices with spare capacity. Functionality (or HIP roles) can be added or removed from devices through the use of "handler" messages, which include Base64 encoded compiled or interpreted classes or scripts.
The protocol is based around a number of defined roles which can be added or removed from individual devices. Each role has a specific set of messages associated with it.
To date MQTT has been used to implement the majority of the messaging (some REST APIs and websocket communication to support user-facing tools has been included but can possibly be considered optional)
Through HIP, localised processing or preperation of data is intended to reduce the connectivity and central processing requirements within IOT - much like the contemporary Fog computing paradigm, but without the need for sophisticated edge computing solutions.
Base Device messages enable core functionality supports each "thing" in a deployment, with additional roles being configured through Configuration messages, and functionality enabled through Handler messages and files
In the above diagram, we have a very small and simple deployment (left of the dashed line) connected to the Platform on the right. Looking at the dashed line, all communication between the deployment and the platform is via a Coordinator on the deployment side [ CR ] and the Coordinator Link [ CrL ] on the Platform side, alllowing us to isolate other devices from the Internet.
The Orange Flow in the diagram is a command sent by a user to a particular actuator or controller (e.g. turn lights on/off), the blue flow represents data collection and aggregation across the deployment.
[ D ] - these are HIP enabled devices and can have a number of different roles which are described below.
Scalability is achievable through a hierarchical architecture whereby data collected from sensors can be processed and aggregated locally before being transferred to a central (cloud or on-premise) platform. Any device can be assiged the role of an "Aggregator" - allowing it to summarise, aggregate or otherwise process data from relevent sensors or aggregators so that the platform receives a single message with all the relevant readings (if desired) in addition to the processed output data. For example, rather than collecting and processing data from multiple sensors in an open plan office-space, HIP can transmit a single message summarising all the relevant data and store this without the need for further processing. In contrast to this, traditional IOT solutions tend to have a flat structure, whereby each sensor feeds raw data to the platform for subsequent, centralised processing.
Management is supported through the use of Health and Configuration messages. Roles such as "Aggregator" or "Controller" above can be assigned and configured centrally from the central platform UI. The "Health" messages update the central platform with the status of each device - allowing administrators to remove roles for under-provisioned devices based on their CPU or memory utilisation. For example, an Aggregator role could be moved from one device to another under-utilised device automatically from the central platform without the need for direct access to the relevant devices.
In addition to the protocol specification, a partial NodeJS implementation of the protocol is available across a number of GitHub repositories:
Platform: This is the main component within the ecosystem and is responsible for the vast majority of logic and control within HIP - it provides REST APIs for client (UI) solutions as well as websockets notifications and also communicates with the Coordinator-link via REST APIs and MQTT messaging
Device: This is any "Thing" deployed as part of an IOT installation, Any Device can have one or more of the following roles and all communication is via MQTT.
- Sensor - this is a local sensor (e.g. DHT 22 for temperature and humidity), or a remote device which is not HIP compatible. Invidivual handlers (e.g. JS file or Kura Bundle) can be deployed from the Platform to enable the Device to integrate with the Sensor and forward readings on a poll or event-driven basis.
- Controller (actuator) - Similar to Sensors, this can be a local or remote actuator and handlers can be deployed from the platform to enable integration. A collection of commands and their parameters are maintained on the Platform and can be pushed to local Commanders to enable local control/integration
- Coordinator - this sits on the edge of a deployment/installation and acts as a single point of contact bwtween the deployment (the Things) and the Platform
- Delegator - Commands from the platform or a local Commander can be routed through one or more Delegators. Different paths/rules etc. can be managed by the Platform and custom handlers deployed on the relevant device
- Aggregator - Multiple Sensors can have their readings forwarded to one or more Aggregators and aggregator handler s can process that data locally before forwarding the output to the Coordinator and/or one or more additional Aggregators. For example, an open plan space in a large office may have a number of lighting/temperature sensors, in addition to uploading the individual readings, these can also be collated by an Aggregator to give a summarised view of the data - e.g. the temperature on the first floor or the ambient light levels on th easterly aspect of the building.
Coordinator-Link: This is an MQTT based solotion which is deployed between the Platform and the deployment (accessed via the Coordinator role on a specified device)
UI: this is a responsive frontend for the solution built in Angular and includes functionality for both administrators and end-users. The UI (and derrived Cordova apps) communicate directly with the Platform
Commander: this is (or will be) a standalone UI which communicates with the local Coordinator rather than the Platofrm in order to monitor associated Sensors and control associated Controllers
To date, a significant portion of the functionality has been developed - sensors can push data, via Aggregators and the Coordinator to the Platform and the Coordinator can accept MQTT messages and route them to the relevent controller/actuator. A number of additional components are under development (e.g. "health and status" messages from devices which will simplify the optimisation and role allocation across a deployment. The development of other elements, e.g., particular UI controls for viewing device and sensor status has not begun yet
I believe that Eclipse HIP could complement and even underpin Eclipse IOT. To continue outside of Eclipse would lead to significant duplication and could adversely impact the quality of HIP as a number of synergies may be overlooked. HIP is at a critical point where it needs to move from a conceptual to a functional project. I believe that certain aspects of the solution are close enough to completion to allow them to be reused across other IOT projects with minimal additional effort, while others need further evaluation and validation to determine how or if they should continue.
N/A - all code and concepts are original
Ideally, I would like to draft a complete specification in Q2 2018 and release v1.0 later in the year
As part of the specification to date, I have been working on a number of use cases. I would hope to expand on this list and extend the protocol based on feedback and discussions with the Eclipse community.
I've also been reviewing some of the main Eclipse IOT projects and would like to engage with their communities to ensure HIP is aligned with and supports them