Test reporting tool#

How do we test?#

In our Yocto based builds we are making use of Automated runtime testing feature of the build system.

This is done automatically for all supported virtual platforms like qemux86-64 and can be manually run on real hardware as well.

Automated testing support is enabled using the Build matrix in CI feature.

Run a tests on real hardware#

We are making use of the same framework when we test on actual hardware. As part of our scotty release we offer the scotty-test tool to guide you through an interactive test session.

To install see Scotty.

Run tests on graphical applications#

In these tests, we want to make sure that the graphical applications are running as expected in an automated way. This is done comparing screenshots of the application to reference pictures of what is expected to be displayed by the graphical application.

Actually, for each graphical test, a screenshot is taken and is compared to a reference image.

To compare the screenshot and the reference image, their hashes are calculated using the python library imagehash. Then the difference between the two hashes gives the hamming distance. The smaller the hamming distance is, the more the images are similar.

Different methods can be used to calculate the hash of an image. In our tests, we mainly use the phash method because it allows to detect if there are some differences between the two images. When we want to detect a smaller difference between the images, we use the dhash method which is more precise.

Testing the SDK#

For every SDK we do run tests of the same kind, but with the difference that we do test for every SDK, no matter of the target architecture.

Automated pull request comments#

The automatically conducted tests are rendered into

  • an overview of the run testsuites, with detailed outputs for all failed tests

  • an overview of recipes that haven’t been build by any of the images we build in CI

The result is automatically posted as a comment to any pull request in the SimpleCore™ software landscape.

Merge report#

For every merged contribution a more detailed test report will be generated. This report contains all the run tests with

  • test name

  • test description

  • image the test was conducted on

  • test result

the output is stored as a markdown document in the repo’s source tree.

From an installed version of Scotty run

scotty-testreport release \
 <path to test results> \
 <path to store the report>