This proposal has been approved and the Eclipse Oniro Compliance Toolchain project has been created.
Visit the project page for the latest information and development.

Eclipse Oniro Compliance Toolchain

Thursday, May 5, 2022 - 05:14 by Luca Miotto
This proposal is in the Project Proposal Phase (as defined in the Eclipse Development Process) and is written to declare its intent and scope. We solicit additional participation and input from the community. Please login and add your feedback in the comments section.
Parent Project
Proposal State
Created
Background

The Eclipse Oniro Core project hosts an open-source operating system that can run on a variety of IoT devices - from microcontrollers powered by an RTOS (such as Zephyr) to powerful multi-core CPUs running a general purpose kernel such as Linux.

Thus, given the very broad scope of the Oniro Core project, a key element to ensuring full compliance and OpenChain conformance is to develop tools that can automate as much as possible the process of Open Source Compliance monitor. Without these tools the work for the Audit team would be challenging and would lead to a very long time for processing data and giving feedback to the development team.

Since OpenChain conformance is a prerequisite of the entire Oniro project, it appears convenient that the tools specifically developed to tackle the specific challenges of Oniro (see further for details) are also developed in a shared fashion by the entire Eclipse community of committers and therefore made a project in its own right.

Scope

Eclipse Oniro Compliance Toolchain provides a complete set of tools to enable, enforce, guide and continuously monitor compliance to Open Source licenses of the developed software in the Oniro project.

This toolchain will also enable generation of a constantly updated Software Bill of Material (SBOM) and other compliance artifacts that can be used by any device maker or vendor that wants to adopt Oniro as a base for their own device software and reduce legal friction for those using the platform.

Eclipse Oniro Compliance Toolchain is based on free and open industry standards and tools like Fossology, Scancode, SPDX and REUSE and it is designed to provide as much as possible accurate information about IP compliance both to Oniro Audit team during development process and to the final users of the Oniro Core Platform.

Eclipse Oniro Compliance Toolchain also provides  an online dashboard to quickly visualize the status of the audit process and current state of the compliance process.

Description

Eclipse Oniro Compliance Toolchain implements a Continuous Compliance workflow for Oniro repositories. The goal is to continuously scan (through ScanCode and Fossology) the source code of all first party and third party components that are included in Oniro Core Platform reference builds for multiple possible targets, so that the scan results can be continuously validated by the Audit Team, and potential IP issues can be spotted and solved early in the development process. The logos in the flowcharts are those of the original developers of the made-on-purpose tools and glue. After approval of this proposal, the code will be donated to the Eclipse Foundation and made publicly available under an open source license compatible with the Eclipse IP Policy.

The unique nature of the Oniro Core Platform project has relevant implications for the continuous compliance process which is being automated through CI/CD pipelines:

  • Package and metadata aggregation
    • Oniro uses Yocto as a build system; Yocto does not have a package manager but is based on stratified layers of build recipes that may interfere with each other (overrides etc.) so the final result (target image) depends on a specific combination of layers and configuration parameters and can be analyzed only after having built it;
    • Yocto can create source and binary packages in various formats, but only post-mortem (i.e. after build, just for software composition analysis purposes) and only specific to a particular built image;
    • different targets in the build matrix may partly share the same software components, but while the upstream source of such components would be the same, usually downstream modifications applied by Yocto recipes would be often different for each different target;
    • if we used Yocto functionality to create source packages to analyze, it would create a different package for each different target even if the upstream component is the same across different targets, and we would end up with a package proliferation that would be hardly manageable for the Audit Team which has to review source packages on Fossology;
    • therefore we need to aggregate sources and source metadata from all possible build targets in the build matrix to create source packages that aggregate all possible source variants for different targets in the same project release or snapshot;
    • to do that, we need to build all possible targets in the build matrix for a given release or snapshot (opposite from build pipelines, where each target is independently built by a separate job), and scan all the build directories with TinfoilHat in order to collect metadata and create "aggregated" source packages to be uploaded and reviewed on Fossology;
  • Pipeline triggers
    • given the structure of the Oniro project, the final build result across the whole target matrix may be affected:
      • by a change in the source code component integrated by a recipe
      • by a change in the recipe/layer that integrates it
      • by a change in another layer that overrides that recipe
      • by a change in the manifest that adds or removes (or changes the revision of) a certain layer
    • Thus compliance pipelines (as build pipelines) need to be run against every commit and merge request to the main branch(es):
      1. in the manifest repo,
      2. in all first party layer repos (even if most of them have been included
      3. in the same repo as the manifest, so cases 1 and 2 mostly coincide), and in all first party source repos.
    • In case 2 (layer changes), if the changed layer stays in a different repo from the manifest, the pipeline needs to fetch the manifest and the layers, and then checkout the cloned git repo of that specific layer that has changed to the commit that needs to be tested, and only then fetch source repos and start the build;
    • In case 3 (source component changes), the pipeline needs to fetch the manifest and the layers, and then use a dedicated tool provided by Yocto to override the specific recipe which integrates the changed component, so that it points to the component's source revision that needs to be tested.
  • Pipeline types and run frequency:
    • The "complete" compliance pipeline cannot be run too frequently, not only because it relies on tools (like ScanCode and Fossology license scanners) that may have long execution times and/or are highly resource-intensive (this could be solved by adding more hardware horsepower, to a point), but also because it would otherwise unnecessarily overload the audit team work (the compliance process is an asynchronous process, because automated scanner results need to be validated in Fossology by the audit team, and the final results are collected by the compliance pipeline at a later stage; so continuously feeding Fossology with new source uploads may technically work, but it would be unmanageable for the audit team).
    • So, especially in the early stages of development of a new project release, there should be a pipeline that uploads to Fossology only new source packages that are "aliens" (i.e. not well-known packages), leaving all the remainder packages to a scheduled pipeline run at fixed times (eg. every week); to spot "alien" packages, we use an aliens4friends functionality that filters out packages that do not have a good match (or are not found at all) in the Debian distribution. On the contrary, at the release candidate stage, such a double-track pipeline is not needed any more.
    • There could be also another pipeline, running periodically (eg. nightly) that will perform just the final 2 steps of aliens4friends’ workflow (fossy and harvest), in order to update json stats for the dashboard and therefore to monitor audit work progress on Fossology.
Why Here?

Eclipse Oniro Compliance Toolchain will help device makers and vendors to adopt Oniro Core Platform providing ready-to-use compliance reports such as Software Bill of Material (SBOM), and it is an integral part of the concept behind the Oniro Core Platform project.

Project Scheduling

Eclipse Oniro Compliance Toolchain has been developed to support past and current development of Oniro Core Platform. It is now ready to move on Eclipse Foundation infrastructure to efficiently provide this support for future Oniro Core Platform releases and allow the Eclipse Foundation to exert control and oversight as an owned project.

Future Work

Next key features of the Eclipse Oniro Compliance Toolchain will be:

  • Integration of binary file analysis and generation of a dependency tree to check license incompatibilities.
  • Development of a knowledge base where to store metadata and licenses information to be exposed through APIs (to enable connection with other tools like SW360).
  • Integration of a shared, publicly accessible and developed database of license obligations as a collaboration task.
Project Leads
Interested Parties

The following individuals, organizations, companies and projects have expressed interest in this project:

  • SECO
  • Linaro
  • Array
  • NOI Techpark
  • Synesthesia
  • Huawei
Initial Contribution

Eclipse Oniro Compliance Toolchain is developed to provide Continuous Compliance workflow for Oniro repositories. Eclipse Oniro Compliance Toolchain is a combination of first party developed tools (Aliens4Friends, TinfoilHat, SBOM Dashboard) and well known, widely used industry standard compliance tools like Fossology and ScanCode. Pipelines that enable Continuous Compliance are also developed as first party code.

Source Repository Type