Creation Review

Eclipse Oniro Compliance Toolchain implements a Continuous Compliance workflow for Oniro repositories. The goal is to continuously scan (through ScanCode and Fossology) the source code of all first party and third party components that are included in Oniro Core Platform reference builds for multiple possible targets, so that the scan results can be continuously validated by the Audit Team, and potential IP issues can be spotted and solved early in the development process. The logos in the flowcharts are those of the original developers of the made-on-purpose tools and glue. After approval of this proposal, the code will be donated to the Eclipse Foundation and made publicly available under an open source license compatible with the Eclipse IP Policy.

The unique nature of the Oniro Core Platform project has relevant implications for the continuous compliance process which is being automated through CI/CD pipelines:

• Package and metadata aggregation
  • Oniro uses Yocto as a build system; Yocto does not have a package manager but is based on stratified layers of build recipes that may interfere with each other (overrides etc.) so the final result (target image) depends on a specific combination of layers and configuration parameters and can be analyzed only after having built it;
  • Yocto can create source and binary packages in various formats, but only post-mortem (i.e. after build, just for software composition analysis purposes) and only specific to a particular built image;
  • different targets in the build matrix may partly share the same software components, but while the upstream source of such components would be the same, usually downstream modifications applied by Yocto recipes would be often different for each different target;
  • if we used Yocto functionality to create source packages to analyze, it would create a different package for each different target even if the upstream component is the same across different targets, and we would end up with a package proliferation that would be hardly manageable for the Audit Team which has to review source packages on Fossology;
  • therefore we need to aggregate sources and source metadata from all possible build targets in the build matrix to create source packages that aggregate all possible source variants for different targets in the same project release or snapshot;
  • to do that, we need to build all possible targets in the build matrix for a given release or snapshot (opposite from build pipelines, where each target is independently built by a separate job), and scan all the build directories with TinfoilHat in order to collect metadata and create "aggregated" source packages to be uploaded and reviewed on Fossology;
• Pipeline triggers
  • given the structure of the Oniro project, the final build result across the whole target matrix may be affected:
    • by a change in the source code component integrated by a recipe
    • by a change in the recipe/layer that integrates it
    • by a change in another layer that overrides that recipe
    • by a change in the manifest that adds or removes (or changes the revision of) a certain layer
  • Thus compliance pipelines (as build pipelines) need to be run against every commit and merge request to the main branch(es):
    1. in the manifest repo,
    2. in all first party layer repos (even if most of them have been included
    3. in the same repo as the manifest, so cases 1 and 2 mostly coincide), and in all first party source repos.
  • In case 2 (layer changes), if the changed layer stays in a different repo from the manifest, the pipeline needs to fetch the manifest and the layers, and then checkout the cloned git repo of that specific layer that has changed to the commit that needs to be tested, and only then fetch source repos and start the build;
  • In case 3 (source component changes), the pipeline needs to fetch the manifest and the layers, and then use a dedicated tool provided by Yocto to override the specific recipe which integrates the changed component, so that it points to the component's source revision that needs to be tested.
• Pipeline types and run frequency:
  • The "complete" compliance pipeline cannot be run too frequently, not only because it relies on tools (like ScanCode and Fossology license scanners) that may have long execution times and/or are highly resource-intensive (this could be solved by adding more hardware horsepower, to a point), but also because it would otherwise unnecessarily overload the audit team work (the compliance process is an asynchronous process, because automated scanner results need to be validated in Fossology by the audit team, and the final results are collected by the compliance pipeline at a later stage; so continuously feeding Fossology with new source uploads may technically work, but it would be unmanageable for the audit team).
  • So, especially in the early stages of development of a new project release, there should be a pipeline that uploads to Fossology only new source packages that are "aliens" (i.e. not well-known packages), leaving all the remainder packages to a scheduled pipeline run at fixed times (eg. every week); to spot "alien" packages, we use an aliens4friends functionality that filters out packages that do not have a good match (or are not found at all) in the Debian distribution. On the contrary, at the release candidate stage, such a double-track pipeline is not needed any more.
  • There could be also another pipeline, running periodically (eg. nightly) that will perform just the final 2 steps of aliens4friends’ workflow (fossy and harvest), in order to update json stats for the dashboard and therefore to monitor audit work progress on Fossology.