niklas/ros2_tracing
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Add quality declaration for LTTng and update tracetools' QD

Browse source 
Backport of 3c3e052ae3366c0b5f9abc3a8a536b0337675636 from !179

Signed-off-by: Christophe Bedard <bedard.christophe@gmail.com>
This commit is contained in:
Christophe Bedard 2020-06-17 08:37:37 -04:00
parent 3b717eb113
commit f45370fabd
3 changed files with 221 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

212
LTTng_QUALITY_DECLARATION.md Normal file
View file

@ -0,0 +1,212 @@
This document is a declaration of software quality for LTTng (*Linux Trace Toolkit: next generation*), an external dependency, based on the guidelines in [REP-2004](https://www.ros.org/reps/rep-2004.html).
# LTTng Quality Declaration
This quality declaration claims that LTTng is in the **Quality Level 1** category.
Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Requirements for Quality Level 1 in REP-2004](https://www.ros.org/reps/rep-2004.html).
Note that LTTng itself is split into multiple repositories.
Each repository may provide more than one (Debian or other) package.
* [`lttng-tools`](https://github.com/lttng/lttng-tools): components to control LTTng tracing
* [`lttng-ust`](https://github.com/lttng/lttng-ust): userspace instrumentation and tracing
* [`lttng-modules`](https://github.com/lttng/lttng-modules): Linux kernel instrumentation and tracing
In general, LTTng follows the strict guidelines of the Linux kernel.
Currently, LTTng is used by downstream (ROS) packages through Ubuntu packages.
This means that the LTTng version is fixed for a given Ubuntu distro, and thus it is fixed for the corresponding ROS distro(s).
If any issue regarding the following requirements arises, it should be possible to fork the LTTng packages, apply a fix, and create and use vendor packages.
## Version Policy [1]
The mitigation strategy mentioned in the first section of this document applies here.
### Version Scheme [1.i]
LTTng does not declare any versioning scheme, but seems to follow `semver`.
Upstream [issue #1269](https://bugs.lttng.org/issues/1269) tracks the declaration of a formal version scheme.
### Version Stability [1.ii]
All LTTng packages are at or above a stable version, i.e. `>= 1.0.0`.
### Public API Declaration [1.iii]
LTTng packages clearly define their public APIs, which is their installed headers (`include/` directories).
### API Stability Within a Released ROS Distribution [1.iv]/[1.vi]
API stability within a released ROS distribution is achieved through Ubuntu package restrictions for each distribution.
### ABI Stability Within a Released ROS Distribution [1.v]/[1.vi]
ABI stability within a released ROS distribution is achieved through Ubuntu package restrictions for each distribution.
## Change Control Process [2]
### Change Requests [2.i]
All changes occur through the [LTTng mailing list](https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev) or through their [code review platform](https://review.lttng.org/).
### Contributor Origin [2.ii]
LTTng uses DCO as its confirmation of contributor origin policy.
For more information, see the [documentation on their Gerrit instance](https://review.lttng.org/Documentation/user-signedoffby.html).
All commits are [signed-off by the author(s)](https://lttng.org/community/#create-and-send-a-patch) and the reviewer(s).
* https://github.com/lttng/lttng-tools/commits/master
* https://github.com/lttng/lttng-ust/commits/master
* https://github.com/lttng/lttng-modules/commits/master
### Peer Review Policy [2.iii]
All changes have at least one peer review.
### Continuous Integration [2.iv]
Most changes are tested on CI before merging.
Release branches and `master` are periodically tested on their [Jenkins instance](https://ci.lttng.org/).
### Documentation Policy [2.v]
LTTng does not have an explicit documentation policy.
All features which are currently utilized by the dependent packages are documented.
If they depend on features which are undocumented, it will be necessary for them to provide their own documentation or contribute it upstream.
## Documentation [3]
### Feature Documentation [3.i]
Features are listed and well documented on the [LTTng website](https://lttng.org/docs/).
### Public API Documentation [3.ii]
LTTng packages have embedded API documentation.
It can be viewed on their man pages:
* [`lttng-ust`](https://lttng.org/man/3/lttng-ust/v2.11/)
### License [3.iii]
All repositories have a `LICENSE` file.
All relevant files have a license identifier.
* `lttng-tools` is licensed under LGPLv2.1 and GPLv2, see [`LICENSE` file](https://github.com/lttng/lttng-tools/blob/master/LICENSE)
* `lttng-ust` is licensed under LGPLv2.1, the MIT license and GPLv2, see [`LICENSE` file](https://github.com/lttng/lttng-ust/blob/master/LICENSE)
* `lttng-modules` is licensed under LGPLv2.1, GPLv2 and the MIT license, see [`LICENSE` file](https://github.com/lttng/lttng-modules/blob/master/LICENSE)
### Copyright Statement [3.iv]
All relevant files have a copyright statement and include a list of copyright holders.
## Testing [4]
### Feature Testing [4.i]
LTTng packages have tests in their `test/` directories which simulate typical usage.
### Public API Testing [4.ii]
LTTng packages have tests in their `test/` directories which test their public APIs.
### Coverage [4.iii]
LTTng does not have an explicit coverage policy and does not appear to track code coverage.
However, that does not impact the quality of dependent packages, because it mostly serves as a metric to show that it is properly tested.
Looking at the tests that LTTng has, it does seem to be properly tested.
In any case, the mitigation strategy mentioned in the first section of this document applies here if an issue is found.
### Performance [4.iv]
LTTng does not have an explicit performance regression policy.
Some relevant packages have performance tests in the form of benchmarks:
* [`lttng-ust`](https://ci.lttng.org/view/Benchmarks/job/lttng-ust-benchmarks_master_build/lastSuccessfulBuild/artifact/src/lttng-ust-benchmarks/benchmarks.json)
* [general benchmarks](https://ci.lttng.org/view/Benchmarks/job/baremetal_benchmarks_kmaster_lmaster/)
### Linters and Static Analysis [4.v]
Coverity, `cppcheck` and `scan-build` are regularly run against LTTng packages:
* [lttng-tools_master_cppcheck](https://ci.lttng.org/view/Code%20quality/job/lttng-tools_master_cppcheck/cppcheckResult)
* [lttng-tools_master_scan-build](https://ci.lttng.org/view/Code%20quality/job/lttng-tools_master_scan-build/HTML_20Report/)
* [lttng-tools_master_coverity](https://ci.lttng.org/view/Code%20quality/job/lttng-tools_master_coverity/)
* [lttng-ust_master_cppcheck](https://ci.lttng.org/view/Code%20quality/job/lttng-ust_master_cppcheck/cppcheckResult)
* [lttng-ust_master_scan-build](https://ci.lttng.org/view/Code%20quality/job/lttng-ust_master_scan-build/HTML_20Report/)
* [lttng-ust_master_coverity](https://ci.lttng.org/view/Code%20quality/job/lttng-ust_master_coverity/)
* [lttng-modules_master_cppcheck](https://ci.lttng.org/view/Code%20quality/job/lttng-modules_master_cppcheck/cppcheckResult)
* [lttng-modules_master_scan-build](https://ci.lttng.org/view/Code%20quality/job/lttng-tools_master_scan-build/HTML_20Report/)
## Dependencies [5]
### Direct Runtime non-ROS Dependency [5.iii]
LTTng packages have some direct runtime dependencies, including:
* [`liburcu`](http://liburcu.org/): userspace RCU (read-copy-update) library
## Platform Support [6]
As its name suggests, LTTng only supports Linux-based systems.
## Security [7]
### Vulnerability Disclosure Policy [7.i]
LTTng does not have an explicit vulnerability disclosure policy.
Upstream [issue #1268](https://bugs.lttng.org/issues/1268) tracks the declaration of a formal VDP.
The mitigation strategy mentioned in the first section of this document applies here.
# Current Status
The table below compares the requirements in REP-2004 with the current state of the LTTng package.
|Number| Requirement | Current state |
|--|--|--|
|1| **Version policy** ||
|1.i| Version policy | ✓ |
|1.ii| Stable version | ✓ |
|1.iii| Strictly declared public API | ✓ |
|1.iv| API stability policy | ✓ |
|1.v| ABI stability policy | ✓ |
|1.vi| API/ABI stablility policy within ROS distribution | ✓ |
|2| **Change control process** ||
|2.i| All changes occur through change request | ✓ |
|2.ii| Confirmation of contributor origin | ✓ |
|2.iii| Peer review policy | ✓ |
|2.iv| CI policy for change requests | ✓ |
|2.v| Documentation policy for change requests | ✓ * |
|3| **Documentation** ||
|3.i| Per feature documentation | ✓ |
|3.ii| Public API documentation | ✓ |
|3.iii| Declared license(s) | ✓ |
|3.iv| Copyright in source files | ✓ |
|3.v.a| Quality declaration linked to from README | ✓ |
|3.v.b| Centralized declaration available for peer review | ✓ |
|3.v.c| References any Level N lists the package belongs to | ✓ |
|4| **Testing** ||
|4.i| Feature items tests | ✓ |
|4.ii| Public API tests | ✓ |
|4.iii.a| Using coverage | ✓ * |
|4.iii.b| Coverage policy | ✓ * |
|4.iv.a| Performance tests | ✓ |
|4.iv.b| Performance tests policy | ✓ * |
|4.v.a| Code style enforcement (linters) | ✓ |
|4.v.b| Use of static analysis tools | ✓ |
|5| **Dependencies** ||
|5.i| Must not have lower level ROS dependencies | ✓ |
|5.ii| Optional ROS lower level dependencies | ✓ |
|5.iii| Justifies quality use of non-ROS dependencies | ✓ * |
|6| **Platform Support** ||
|6.i| Support targets tier 1 ROS platforms | ✓ |
|7| **Security** ||
|7.i| Vulnerability Disclosure Policy | ✓ * |
\* : mitigated/does not apply
Comparing this table to the [Quality Level Comparison Chart of REP-2004](https://www.ros.org/reps/rep-2004.html#quality-level-comparison-chart) led us to conclude that this package qualifies for Quality Level 1.

2
README.md
View file

@ -74,7 +74,7 @@ Package containing a `ros2cli` extension to enable tracing.
Library to support instrumenting ROS packages, including core packages.
This package claims to be in the **Quality Level 3** category, see the [Quality Declaration](./tracetools/QUALITY_DECLARATION.md) for more details.
This package claims to be in the **Quality Level 2** category, see the [Quality Declaration](./tracetools/QUALITY_DECLARATION.md) for more details.
See the [API documentation](https://ros-tracing.gitlab.io/ros2_tracing-api/).

17
tracetools/QUALITY_DECLARATION.md
View file

@ -2,7 +2,7 @@ This document is a declaration of software quality for the `tracetools` package,
# `tracetools` Quality Declaration
The package `tracetools` claims to be in the **Quality Level 3** category.
The package `tracetools` claims to be in the **Quality Level 2** category.
Below are the rationales, notes, and caveats for this claim, organized by each requirement listed in the [Package Requirements for Quality Level 1 in REP-2004](https://www.ros.org/reps/rep-2004.html).
@ -63,8 +63,6 @@ All merge requests must resolve related documentation changes before merging.
## Documentation [3]
`tracetools` does not currently follow all the recommended guidelines for ROS Core packages in the [ROS 2 Developer Guide](https://index.ros.org/doc/ros2/Contributing/Developer-Guide/#documentation).
### Feature Documentation [3.i]
`tracetools` has [documentation](../doc/design_ros_2.md) for all of its features, and new features require documentation before being added.
@ -137,7 +135,10 @@ This package does not have any optional runtime ROS dependencies.
### Direct Runtime non-ROS Dependency [5.iii]
`tracetools` has a run-time dependency on [LTTng](https://lttng.org/docs/v2.11/). A Quality Level review for LTTng has yet to be done.
`tracetools` has a run-time dependency on [LTTng](https://lttng.org/docs/v2.11/).
LTTng provides tracing capabilities.
It is **Quality Level 1**, see its [Quality Declaration document](../LTTng_QUALITY_DECLARATION.md).
## Platform Support [6]
@ -182,6 +183,7 @@ The table below compares the requirements in REP-2004 with the current state of
|3.iv| Copyright in source files | ✓ |
|3.v.a| Quality declaration linked to from README | ✓ |
|3.v.b| Centralized declaration available for peer review | |
|3.v.c| References any Level N lists the package belongs to | ✓ |
|4| **Testing** ||
|4.i| Feature items tests | ✓ |
|4.ii| Public API tests | ✓ |
@ -194,7 +196,7 @@ The table below compares the requirements in REP-2004 with the current state of
|5| **Dependencies** ||
|5.i| Must not have lower level ROS dependencies | ✓ |
|5.ii| Optional ROS lower level dependencies | ✓ |
|5.iii| Justifies quality use of non-ROS dependencies | |
|5.iii| Justifies quality use of non-ROS dependencies | ✓ |
|6| **Platform Support** ||
|6.i| Support targets tier 1 ROS platforms | ✓ |
|7| **Security** ||
@ -202,7 +204,4 @@ The table below compares the requirements in REP-2004 with the current state of
\* : going forward
Comparing this table to the [Quality Level Comparison Chart of REP-2004](https://www.ros.org/reps/rep-2004.html#quality-level-comparison-chart) led us to conclude that this package qualifies for Quality Level 3.
Missing for Quality Level 2:
* 5.iii Justifies quality use of non-ROS dependencies
Comparing this table to the [Quality Level Comparison Chart of REP-2004](https://www.ros.org/reps/rep-2004.html#quality-level-comparison-chart) led us to conclude that this package qualifies for Quality Level 2.