[Feature Request] Introduce testcase-specific reporting

[mbuechse]

Currently, our report assigns to every testcase a result (PASS/FAIL/ABORT). Everything else is buried in blanket stdout/stderr capture, which has its own internal structure, but that is only apparent if you see all the intervening debug messages, which are redacted on the compliance monitor. It's not great UX.

Also stderr should not be relevant to anyone except test developers! That need not even be part of the UI (except for the full report view).

• change all scripts so that relevant output is done on stdout, prefixed by testcase-id
• make sure stderr only contains debug info and unexpected operational errors
• hide stderr from CM UI (except in full report view)
• make testcase-specific stdout messages available in CM UI (no longer calling it stdout)

[mbuechse]

I instrumented the logger in Tests/iaas/openstack_test.py (which contains all iaas testcases) to automatically output for each line the testcase (if the line can be connected to a single testcase). In the following, lines starting with DEBUG:, INFO: etc. originate from logging, whereas the others are printed on stdout.

Exhibit A:

$ ./iaas/openstack_test.py -c artcodix-ro scs-0100-semantics-check scs-0100-syntax-check
DEBUG: ** scs-0100-semantics-check
INFO: scs-0100-semantics-check: Non-standard disk size for 'SCS-2V-4-2s': 2 not in (5, 10, 20, 50, 100, 200, ...)
INFO: scs-0100-semantics-check: Non-standard disk size for 'SCS-1V-2-1s': 1 not in (5, 10, 20, 50, 100, 200, ...)
scs-0100-semantics-check: PASS
DEBUG: ** scs-0100-syntax-check
scs-0100-syntax-check: error parsing SCS-16V-48-4GPU: Extra characters: GPU
scs-0100-syntax-check: FAIL

Here one can now see that the two INFO lines are in fact related to the testcase mentioned in the DEBUG line above (the instrumented logger tells us as much by prepending the name of the testcase to the log message).

Exhibit B:

$ ./iaas/openstack_test.py -c pco-prod1 scs-0100-syntax-check scs-0100-semantics-check  
DEBUG: ** scs-0100-syntax-check
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-4V:8:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-4V:32:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-8V:32:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-2V:4:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-8V:8:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-16V:64:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-16V:32:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-2V:16:50
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-8V:16:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-4V:16:100
WARNING: [scs_flavors] Name is merely tolerated v1: SCS-2V:8:100
scs-0100-syntax-check: PASS
DEBUG: ** scs-0100-semantics-check
scs-0100-semantics-check: PASS

Here one can now see that the WARNING lines are in fact NOT related to the testcase mentioned in the DEBUG line above (the instrumented logger tells us as much by prepending something in brackets that is not a testcase id).

The same thing happens in Exhibit C:

$ ./iaas/openstack_test.py -c syseleven-dus2 scs-0101-entropy-avail
DEBUG: ** scs-0101-entropy-avail
DEBUG: [collected_vm_output] Selected image: Ubuntu Noble 24.04 (bb11afde-d24b-4121-a936-2f04d6b44559)
DEBUG: [collected_vm_output] Unable to pick flavor with hw_rng:allowed=true for image 'Ubuntu Noble 24.04'
DEBUG: [collected_vm_output] Using external network 2d5c0a31-8b40-4def-8dce-8c8ca4b5184c.
DEBUG: [collected_vm_output] Creating instance of image 'Ubuntu Noble 24.04' using flavor 'SCS-1V-2-25p' and 8 GiB ephemeral boot volume
DEBUG: [collected_vm_output] Using userdata:
#cloud-config
apt:
  primary:
    - arches: [default]
      uri: http://az1.clouds.archive.ubuntu.com/ubuntu/
packages:
  - rng-tools5
bootcmd:
  - systemctl mask serial-getty@ttyS0.service
runcmd:
  - >-
    cat /proc/sys/kernel/random/entropy_avail | awk '$0="_scs-test-entropy-avail: "$0'
  - >-
    cat /dev/random | rngtest -c 1000 2>&1 | awk '$0="_scs-test-fips-test: "$0'
  - >-
    sudo systemctl status rngd 2>&1 | awk '$0="_scs-test-rngd: "$0'
  - >-
    cat /sys/devices/virtual/misc/hw_random/rng_available 2>&1 | awk '$0="_scs-test-virtio-rng: "$0'; sudo /bin/sh -c 'od -vAn -N2 -tu2 < /dev/hwrng' 2>&1 | awk '$0="_scs-test-virtio-rng: "$0'
  - echo '_scs-test-end'
final_message: "_scs-test-end"
DEBUG: [collected_vm_output] Server '_scs-0101-server' ('bf6d14c7-8e14-403e-8aa6-6861f2f728af') has been created
DEBUG: [collected_vm_output] Deleting server '_scs-0101-server'
scs-0101-entropy-avail: PASS

[mbuechse]

I continued my experiment of "instrumenting the logger" and improved it to the point where stderr didn't look too shabby. Find the code at this branch: https://github.com/SovereignCloudStack/standards/tree/experimental/log-testcase-id
An example report can be found here: https://compliance.sovereignit.cloud/page/report_full/c457b51a-160d-4e6d-abfa-d59c3c7eab14#088d3049-110d-43d2-afd1-1fc80c50167a

It looks a bit like this:

scs-0100-semantics-check: DEBUG: ** begin testcase
scs-0100-syntax-check: DEBUG: ** begin testcase
scs-0101-entropy-avail: DEBUG: ** begin testcase
scs-0101-entropy-avail [collected_vm_output]: DEBUG: Selected image: Debian 13 (5837233d-82ab-4bde-9be1-c9d141f087d5)
scs-0101-entropy-avail [collected_vm_output]: DEBUG: Unable to pick flavor with hw_rng:allowed=true for image 'Debian 13'
scs-0101-entropy-avail [collected_vm_output]: DEBUG: Using external network e1f4e556-6fab-47ee-b395-b10af44920b8.
scs-0101-entropy-avail [collected_vm_output]: DEBUG: Creating instance of image 'Debian 13' using flavor 'SCS-1L-1' and 8 GiB ephemeral boot volume
scs-0101-entropy-avail [collected_vm_output]: DEBUG: Using userdata:
#cloud-config
apt:
  primary:
    - arches: [default]
      uri: https://mirror.plusserver.com/debian/debian/
  security:
    - arches: [default]
      uri: https://mirror.plusserver.com/debian/debian-security/
packages:
  - rng-tools5
bootcmd:
  - systemctl mask serial-getty@ttyS0.service
runcmd:
  - >-
    cat /proc/sys/kernel/random/entropy_avail | awk '$0="_scs-test-entropy-avail: "$0'
  - >-
    cat /dev/random | rngtest -c 1000 2>&1 | awk '$0="_scs-test-fips-test: "$0'
  - >-
    sudo systemctl status rngd 2>&1 | awk '$0="_scs-test-rngd: "$0'
  - >-
    cat /sys/devices/virtual/misc/hw_random/rng_available 2>&1 | awk '$0="_scs-test-virtio-rng: "$0'; sudo /bin/sh -c 'od -vAn -N2 -tu2 < /dev/hwrng' 2>&1 | awk '$0="_scs-test-virtio-rng: "$0'
  - echo '_scs-test-end'
final_message: "_scs-test-end"
scs-0101-entropy-avail [collected_vm_output]: DEBUG: Server '_scs-0101-server' ('787a197e-c6c9-4688-a75c-149f820d727f') has been created
scs-0101-entropy-avail [collected_vm_output]: DEBUG: Deleting server '_scs-0101-server'
scs-0101-fips-test: DEBUG: ** begin testcase
scs-0101-flavor-property: DEBUG: ** begin testcase
scs-0101-flavor-property: ERROR: Flavor 'SCS-4V-16' missing attributes: hw_rng:allowed=True; additionally, missing optional attributes: hw_rng:rate_bytes, hw_rng:rate_period
scs-0101-flavor-property: ERROR: Flavor 'SCS-8V-16-50' missing attributes: hw_rng:allowed=True; additionally, missing optional attributes: hw_rng:rate_bytes, hw_rng:rate_period
scs-0101-flavor-property: ERROR: Flavor 'SCS-4V-8' missing attributes: hw_rng:allowed=True; additionally, missing optional attributes: hw_rng:rate_bytes, hw_rng:rate_period
scs-0101-flavor-property: ERROR: Flavor 'SCS-4V-16-50' missing attributes: hw_rng:allowed=True; additionally, missing optional attributes: hw_rng:rate_bytes, hw_rng:rate_period

HOWEVER, I decided to change direction. Improving stderr is besides the point; the point is that stderr shouldn't be interesting to anyone but the script hackers anyway. Let's focus on getting ALL relevant messages over to stdout, and then hide stderr away.

[mbuechse]

I'm wondering whether we should use stdout to dump some json object that the main script can incorporate into the overall report (more or less as-is):

• then we no longer need to parse our own idiosyncratic syntax (instead, we would need some kind of JSON schema),
• we no longer need to include stdout verbatim, which is always a pain with YAML.

As for stderr, I'm wondering whether we should put that into a different file altogether... Maybe we can even accept a zip file of the test suite's working directory. If we include sonobuoy output, such a file can become quite large (at least 1 MiB), and I'm not sure we should be accepting and storing uploads that large within the compliance monitor. Maybe we can establish a "professional" WebDAV server with an account for each subject so the complete artifact can be stored there (by the party running the tests) and merely an URL added to the report? (We would have to clean up regularly.)

[mbuechse]

Note: as a simple first step, change channel from ERROR to INFO, because ERROR is just too misleading for most people (and arguably wrong)