Eclipse Ditto Creation Review

Type: 
Creation
State: 
Successful
End Date of the Review Period: 

Reviews run for a minimum of one week. The outcome of the review is decided on this date. This is the last day to make comments or ask questions about this review.

Wednesday, April 5, 2017
Project: 
Proposal: 

Eclipse Ditto

Parent Project: 
Background: 

In the Internet of Things two disciplines of software development come together. On one hand, a hardware influenced view of software for devices and protocols and on the other hand the view of web/mobile/business application development.

To bring these both sides closer together the metaphor of Digital Twins can be used, where a Digital Twin is kind of holistic view of all capabilities and aspects of a device/product asset including its digital representation.

It is important to make these Digital Twins accessible in a convenient way, independent of the protocols used to integrated them and the current connectivity state of the devices. The core of this Digital Twin is the information of the state of the device regarding properties and meta data.

As of today a lot of IoT solutions implement functionality to manage something similar to this idea of Digital Twins. They do this by using simple databases covering some aspects of Digital Twins, which is totally fine until a certain point in time.

Probably more complex topics come into play soon:

  • Reflect changes of the physical world in the back end representation
  • Notification of different application layers on relevant changes
  • Providing access to the Digital Twins via API
  • Harmonizing the persistence and access for multiple different device types
  • Providing access control for the Digital Twins
  • On increasing load still provide a scalable, robust and high-performance implementation
  • ...

Gradually the initially good and simple starting point gets more and more work and requires to invest in technical fundamentals instead of business level features.

It would be helpful to have a domain-independent component that takes care of these mentioned aspects and can be used out-of-the-box in IoT solutions.

Beyond that, this component could be used in a "back end as a service" approach to move into the direction of serverless architectures. Without the need to develop and operate a custom back end, IoT solution developers can focus again on business requirements, on connecting devices to the cloud / back end and implementing business applications.

Scope: 

Eclipse Ditto is the open-source project of Eclipse IoT that provides a ready-to-use functionality to manage the state of Digital Twins. It provides access to them and mediates between the physical world and this digital representation.

To achieve this Ditto addresses the following aspects:

  • Device-as-a-Service
    Provide a higher abstraction level in form of an API used to work with individual devices.
  • State management for Digital Twins
    Differ between reported (last known), desired (target) and current state (live) of devices, including support for synchronization and publishing of state changes.
  • Organize your set of Digital Twins
    Support finding and selecting sets of Digital Twins by providing search functionality on meta data and state data.
Description: 

Device-as-a-Service

IoT solutions have to interact with a heterogeneous set of device types, device protocols and communication patterns.

To bring back simplicity to IoT developers, Eclipse Ditto exposes a unified resource-based API that can be used to interact with devices, abstracting from the complexity of different device types and how devices are connected. It helps to structure the devices into their distinct aspects of functionality and can optionally enforce data types and data validation based on a formal device meta model (Eclipse Vorto).

As devices are equipped with a public API (potentially public on the Internet), it is crucial to define on a device level which individuals are allowed interact with the devices. Ditto ensures that access to the device API is only granted for authorized individuals. Authentication is not scope of Ditto and delegated to existing identity providers.

With this approach your devices are turned into services with a hosted, always accessible and available API.
Devices managed by Ditto are usable as easy as any of the other services (like weather, maps, ...) within your application.

State management for Digital Twins

A digital representation of physical devices consists at its heart of the state these devices.

For IoT solutions the following information regarding state is most relevant:

  • Device and sensor properties like temperature, location, level, fault information, etc.
  • Configuration properties of sensors and actors like thresholds, intervals, ranges, toggles and limits, etc.

A good representation of this state in a Digital Twin should support different perspectives for these properties:

  • Reported property values based on the last transmission to the back end
  • Desired target property value for configuration properties
  • Live perspective reflecting the properties values at the current point of time

The state management provides access to all three different perspectives and helps in synchronizing between them.

Organize your set of Digital Twins

When interacting with huge amounts of devices, it can get difficult to keep track of which devices and devices types are there and how they are related (e.g. spatially).

It is wise to add meta data like e.g. manufacturer, model information, static geographic location, serial number or software version to devices in order to find them again later.
That meta data as well as the state data is automatically indexed by Ditto which leads to fast search responses provided by the search API, even when there are millions of devices to search in.

When working with sets of devices authorization information is used to determine the visibility of devices to individuals and control who can change device data.

Features at a glance

The described functionality is offered by Ditto in form of the following features:

  • Unified resource-based HTTP JSON API representing devices
  • Definition of a simple "Digital Twin State Management Protocol" using JSON for command- and events-based interaction with devices
  • Websocket API based on this protocol in addition to the resource-based HTTP API
  • Managing device meta data via APIs
  • Optionally configure and enforce a schema for device state (via Eclipse Vorto)
  • Schema evolution support for schema based device state
  • Accessing and setting different state perspectives
    • Live
    • Reported
    • Desired
  • Notification about changes of device resources via HTTP Server-sent events (SSE)
  • Authorization/Access control at device API enforcing that only allowed individuals may read/write
  • Search HTTP API accepting predicate-based query language
  • Emit events resulting from state changes of devices which can be used
    • for building up a "transaction log" (e.g. using Apache Kafka)
    • as source for stream processing (e.g. via Apache Spark Streaming)
    • for building up additional persistence representations (e.g. into an InfluxDB providing optimized access to the history of a device's state properties)
    • for transmitting data into data analytic tools (e.g. into an Apache Hadoop HDFS)
  • Out-of-box integration with Eclipse Hono for communication with devices using standard or custom device protocols

Integrate Ditto in a IoT landscape

Ditto is especially useful in the context of a larger IoT landscape. In this landscape the other important aspects of a IoT solution like device communication or data analytics are covered by distinct components.

The following diagram shows a typical landscape for IoT solutions where Ditto brings in its functionality:

The most important integration aspect is the device communication. This is essential for the Digital Twins to really be twins of real-world physical devices.

In addition to provide a custom device communication layer Ditto will provide an out-of-the-box integration with Eclipse Hono to support device communication.

In this scenario Ditto uses Eclipse Hono in order to receive messages (e.g. state changes) from devices and to send messages (e.g. configuration changes) to devices.

Why Here?: 

The Eclipse IoT community already brings a lot of very useful technologies and components for IoT applications. As the scope of Ditto has a strong focus on the management of Digital Twins it is essential for it to integrate with these other components to provide a helpful end-to-end platform for IoT application developers.

That's why Ditto does explicitly not cover aspects like connectivity and communication with devices, communication protocols, the development of (embedded) software for devices and the modeling of devices.

To support this end-to-end platform idea the following projects, deal with relevant aspects that are interesting for Ditto:

  • Eclipse Hono for the message exchange with devices
  • Eclipse Vorto for the modeling of device structures reflected by the Digital Twins
  • Eclipse Hawkbit for rolling out software updates based on meta-data of the Digital Twins
  • Eclipse Kapua as integration framework as easy quick-start for end-to-end IoT solutions leveraging the Digital Twins approach

A potential orchestration of Ditto with these Eclipse IoT projects could result in the following end-to-end platform:

Initial Contribution: 

The initial contribution of Ditto will contain several ready-to-run microservices bundled as Docker images.

The contributed source code is structured as Maven multi module build providing services as well as libraries on which the services are based on.

Libraries

  • ditto-json
    • JSON library inspired by minimal-json (https://github.com/ralfstx/minimal-json) library
    • provides convenient access for JSON fields, JsonPointer based operations (get/set), definitions of JSON fields, etc.
    • uses minimal-json as default JSON serializer/deserializer
  • ditto-model
    • includes domain model classes for Ditto, e.g. "Thing" and "Feature"
    • all domain model classes are serializable to JSON and deserializable from JSON
  • ditto-commands-events
    • includes command and event classes used for CQRS pattern, event sourcing + event publishing
    • all commands and events are serializable to JSON and deserializable from JSON
  • ditto-protocol-adapter
    • provides a definition of the "Digital Twin State Management Protocol" including an adapter of the command and event objects to JSON format

Microservices

The services are implemented in Java and make use of the Akka Toolkit (http://akka.io/).

The services use an externally provided MongoDB as database (https://www.mongodb.org). This is a prerequisite that has to be provided by the user and is not part of Ditto.

  • ditto-api
    • provides REST-like HTTP API for CRUD and search operations on Things
    • provides a Websocket API for the same CRUD operations and for receiving events
  • ditto-things
    • responsible for persisting/restoring Things into a MongoDB
    • applies event sourcing pattern: accepts commands for modifying which result in events which are persisted + published
  • ditto-search-updater
    • subscribes to events published by "ditto-things" service and updates its own search index in MongoDB
    • does synchronizations if events for specific Things were missed in order to keep eventual consistency
  • ditto-search
    • responsible for answering to search queries directly from optimized search index from MongoDB
  • ditto-amqp-bridge
    • consumes AMQP 1.0 interface and forwards the received "State Management Protocol" messages to the responsible services
    • emits published events from "ditto-things" towards the AMQP 1.0 interface via the defined "State Management Protocol"

Project Scheduling: 

We aim to provide the initial contribution in Q2/2017.

Future Work: 
  • Alternative search index implementation based on Elastic Search
  • Support for geolocation information within search and notifications (e.g. for geofencing)
  • Integration of business logic executing using Function-as-a-Service providers (e.g. OpenWhisk)
  • In general: scoping towards more aspects of the Digital Twin metaphor
    • seamless integration of Digital Twin history data
    • integration with technologies covering high-level aspects of Digital Twins like semantics, simulation and orchestration/federation
People