This proposal has been approved and the Eclipse Streamsheets™ project has been created.
Visit the project page for the latest information and development.

Go to Project

Eclipse Streamsheets

Tuesday, November 5, 2019 - 13:31 by Gabriele Raue
Basics
This proposal is in the Project Proposal Phase (as defined in the Eclipse Development Process) and is written to declare its intent and scope. We solicit additional participation and input from the community. Please login and add your feedback in the comments section.
Project
Eclipse Streamsheets™
Parent Project
Eclipse IoT
Proposal State
Created
Background

The integration of different software systems is time consuming and expensive. Low-code and no-code development platforms can speed up the development and the integration of software systems. End-users of low-code and no-code development platforms do not need to have any knowledge of a programming language because the development is carried out using a visual approach. The development of an application is thus more accessible, even for people who are not software developers. In addition, applications can be created more efficiently which in turn increases productivity.

Streamsheets facilitate the creation and integration of applications in a familiar spreadsheet environment without a single line of program code. Users define logic for analyses, data transformation and processes in an already known and accepted way, which minimizes familiarization times and fears of contact, thereby acceIerating implementations of digital solutions.

Unlike traditional spreadsheet applications, Streamsheets are not static but are designed for dynamic processing, analysis, and controlling of data flows. Input data (primarily from but not limited to IoT devices / sensors) enters the inbox of a Streamsheet, using a concept similar to an e-mail program, then flows through the logic defined in the Streamsheet, before the results are forwarded to downstream IoT devices, processes or databases.

Once the modeling phase of a process is complete, the graphical interface for the Streamsheet can be closed and its process logic will continuously run in real-time as a microservice in the server environment.

Scope

The Streamsheets project provides a platform for processing stream data using a spreadsheet-like interface and spreadsheet-like formulas. In order to achieve this it provides:

  • Web UI Service: A spreadsheet-like user interface for visually creating and managing Streamsheets.
  • Gateway Service: An API for managing Streamsheets.
  • Machine Service: A backend runtime for executing Streamsheets.
  • Streams Service: A pluggable system for adding connectors to other protocols, databases, etc.
  • Graph Service: A service for synchronizing Streamsheets interactions between different users.
  • A JavaScript/Node.js client for interacting with the Streamsheets API.
Description

With Streamsheets the everyday technical or business end user can create stream processing applications just by using their existing spreadsheet knowledge (e.g. from Microsoft Excel or Google Sheets). Streamsheets give non-programmers the opportunity to work with event streams with a power and flexibility that would otherwise only be available to an experienced software programmer.

Applications for stream processing are widespread and as generic as spreadsheets can be. Areas Streamsheets can be used in include the following:

  • IoT: Control, analysis, monitoring, visualization and simulation of industrial and other IoT processes (Smart Factory, Smart Building, Smart City, Smart Energy, Smart Farming, …) using MQTT, AMQP or other messaging protocols.
  • Stream Processing: Any kind of business or technical processes that publish and subscribe to a streaming platform (e.g. Kafka) and does analysis, transactions, monitoring, statistics, automated alerts, conversions, semantic mappings, enriching of data sets, etc.
  • Protocol conversion or gateways: Edge to Cloud, IT to OT, Protocol to Protocol, Request/Response to Publish/Subscribe, Logfile to Stream, etc. using MQTT, AMQP, UDP, REST, POP, SMTP protocols or simple input from text files.

     
Licenses
Eclipse Public License 2.0
Legal Issues

Our main dependencies are licensed under a MIT license. Our own code will be licensed under the EPL v2.0. We do not expect any legal issues.

Why Here?

Eclipse already hosts many mature projects related to IoT. The proposed Streamsheets project is a very good complement to the existing projects, including Eclipse Paho, Eclipse Milo, and most notably the Eclipse Mosquitto project. Streamsheets uses the MQTT protocol both for internal and external communication and uses Mosquitto as its standard MQTT broker. Streamsheets also complements the Mosquitto project in the sense that it provides a standard user interface for Mosquitto.

Future Work
  • Extend Chart Modules: the current charts (line charts, bar charts, etc.) will be extended with additional functionality.
  • Built-in tutorials: to help users to get started with Streamsheets we will integrate built-in tutorials (text and/or video) that show how to use different features of the software.
  • UX/UI improvements: we will further enhance the software for usability and user experience.
Project Scheduling

Having released version 1.4 on the 16th of October 2019 we are planning future releases every 10-12 weeks.

People
Project Leads
Philip Ackermann
Roger Light
Committers
Philip Ackermann
Arnd Houben
Roger Light
Michael Raue
Andreas Sayn
Evangelos Vlachogiannis
Mentors
Jens Reimann
Interested Parties
  • Festo
  • Siemens
  • Real-time Automation
  • Edgeworx
Source Code
Initial Contribution

The initial code contribution to Streamsheets will contain the code from the GitHub repository located at https://github.com/cedalo/streamsheets. The backend services of the Streamsheets platform are written in JavaScript and intended to run on Node.js. The main dependencies for the backend are Express.js and MQTT.js. The frontend is as well written in JavaScript and mainly depends on the React framework using the Material UI component library and on the self-developed “JavaScript Graph Library” (JSG) for Canvas interactions, which will be included in the Streamsheets project.

Source Repository Type
GitHub
Source Repositories
https://github.com/cedalo/streamsheets

Submitted by Thomas Jaeckle on Sat, 11/09/2019 - 14:41

Congratulations on publishing the proposal. I think Eclipse IoT is a great environment for Streamsheets.

We already got a small demo of Eclipse Ditto together with Streamsheets (processing change events of digital twins) running at EclipseCon. We had to use http polling which was not ideal.

So my questions:

  • When trying out Streamsheets I could not consume from MQTT brokers, only publish to them. Did I miss something or is consuming from MQTT not included? 
  • Can Streamsheets be operated in a high available cluster and is it possible to scale horizontally? 
  • For integration with Eclipse Ditto a Websocket consumer would have been great as Ditto is able to publish change events via Websocket. Do you have plans on supporting that? Integration with Ditto would be very easy then..

Best regards Thomas

Submitted by Philip Ackermann on Fri, 01/17/2020 - 04:33

Hi Thomas,

Thank you for your interest in our project! :-) And sorry for the late reply. Somehow the notification of your comment did not reach us.

Regarding your questions:

  1. To consume data from a broker, you need to set up a consumer. There are three resources which can be part of the configuration of a data stream: always a connector and then a consumer or a producer. 
    • Connectors set up the connection to brokers including information, such as the URL and certificates or login details. Every data stream is based on a connector. However, you can define many connectors and therefore connect to many brokers at the same time.
    • To consume data, there are consumers. Based on a connector you define which topics you want to subscribe to.
    • To produce/publish data, there are producers. Producers are based on connectors, too.
    • Thus, connectors are used to establish a connection to a broker whereas consumers and producers define which data (streams) on that broker are addressed.
    • You can find more information about streams in the official documentation here and here.
  2. The architecture of streamsheets was created having scalability in mind, but we do not yet officially support cluster mode or horizontal scalability. However, since those features are very important for us, this is something we will definitely address in the future.
  3. We, indeed, have plans to support WebSockets fully, that is to say not only as consumers, but as producers, too. This integration is currently on the mid-term roadmap. Btw.: are there already other options to integrate with Ditto, e.g., MQTT? 

Best regards,

Philip