TestFramework currently outputs results in its own custom format. Outputting XML so it can be processed by JUnit-compatible formatters can be useful.

Tools such as bazel set an environment variable XML_OUTPUT_FILE (as well as a few other useful environment variables) to point where the XML file is expected to be written. Other test pipelines could use this as input for displaying reports. In Bazel, when tests do not provide JUnit-compatible outputs, small wrappers are written to execute the tests, process the actual native output, and produce a report.

• Example of setting up the environment specifically in Bazel: https://github.com/bazelbuild/bazel/blob/03c3ea01af438abd8319a359aa4fe3ff1a9f7e99/tools/test/test-setup.sh
• Bazel's documentation on this and other test-related environment variables: https://bazel.build/reference/test-encyclopedia

I propose hacking on TestFramework/gnustep-tests.in to make it understand the XML_OUTPUT_FILE envvar, and process the files a bit differently if the envvar is set. For example, it could invoke a test runner instead of running the tests directly, and process their output, finally invoking the same test runner as a helper binary to assemble the XML into one coherent whole.

I propose that the condition to execute can be that XML_OUTPUT_FILE envvar is present, and that a test runner binary is present and executable (that is: XML processing itself does not need to be hacked together directly in bash in gnustep-tests, which is already a rather complicated shell script; if the helper is not present, the file could be not written out, or if it is written out, it could simply be a near-hardcoded file indicating success or failure and not much more). The test runner could ship in gnustep-base, or depend on it being installed in order to execute. Or, given the test framework will function perfectly fine without XML_OUTPUT_FILE, it could be written in an auxiliary non-Objective-C language and simply be unused if the compiler for the language or the interpreter is not present. Despite the dependency loop, it's likely slightly nicer if we have the processor written with GNUstep.

Some other useful envvars that could be set and used by TestFramework -- possibly even used as a more explicit declaration that JUnit file should be written out than just XML_OUTPUT_FILE:

• TEST_SRCDIR
  • If TEST_SRCDIR is set, it can be used as the root of the source tree (not of the tests themselves, but of the 'root of the repo'), to determine test paths relative to repo root.
  • If TEST_SRCDIR is not set, it can either default to CWD (current dir), or TOP_DIR, or we could search up the tree until we run into a directory containing one of the set of files widely indicating 'root of a project' (e.g. configure or configure.ac, INSTALL, Version, ANNOUNCE). Whatever it is set to can propagate down the GNUmakefiles to indicate root of the workspace.
  • Note that in Bazel, TEST_SRCDIR is parent of a directory containing a representation of the workspace; we don't have to set it that way in our case. Or we could set it to the parent of whatever we identify as the topdir (dirname), and set TEST_WORKSPACE to the name of the topdir (basename).
  • I am not certain that there is a strong usecase for this. It might help the helper binary collect the auxiliary status files.
• TEST_TMPDIR
  • Location where we can write any temporary files we want, including intermediate test result files, XML files, etc.
  • If set, perhaps we should have gnustep-tests place tests.log and tests.sum there; on the other hand, that complicates compatibility in case existing specialized tests depend on it being set. It could help the helper write temporary files into a location where they're unlikely to interfere with the existing tests.
  • If not set, we could just use mktemp and guarantee we will remove it afterwards (but only do so if the XML output is requested; we would not want to pollute tempdirs if JUnit output is not even requested, or if the helper binary is not runnable, or if the platform is unsupported otherwise).

Regarding design of the output file:

TestInfo, given it's a good indicator that a directory contains tests, indicates an individual <testsuite>. So does an individual .m file: they contain many <testcase>s.

It would be ideal if we had the total expected number of tests to run pre-declared, so we can declare that in the XML file. However, that seems infeasible: like in many other languages, the testing framework does not pre-declare tests to run, just the targets / binaries that can execute them (in our case, each .m being one).

tests.sum seems like it may contain enough information for a helper program to turn it into a JUnit XML, as long as we are ok ignoring any stdout/stderr that could be included in the .XML file -- here is a slightly cropped version of a file on my machine:

Passed test:     (2025-06-10 17:27:23.260 GMT+1) basictypes.m:157 ... can unarchive ushort from ushort-2.type
Passed test:     (2025-06-10 17:27:23.261 GMT+1) basictypes.m:162 ... can unarchive BOOL from BOOL-1.type
Passed test:     (2025-06-10 17:27:23.261 GMT+1) basictypes.m:241 ... archiving as int - dearchiving as NSInteger
Passed test:     (2025-06-10 17:27:23.262 GMT+1) basictypes.m:243 ... archiving as unsigned int - dearchiving as NSUInteger
Completed file:  basictypes.m
Passed test:     (2025-06-10 17:27:23.391 GMT+1) decoding.m:197 ... decoding current version of class NSArray
Passed test:     (2025-06-10 17:27:23.392 GMT+1) decoding.m:246 ... decoding 32bit version 1 of class NSArray
Passed test:     (2025-06-10 17:27:23.393 GMT+1) decoding.m:246 ... decoding 64bit version 1 of class NSArray
Passed test:     (2025-06-10 17:27:23.422 GMT+1) decoding.m:197 ... decoding current version of class NSNumber
Passed test:     (2025-06-10 17:27:23.422 GMT+1) decoding.m:246 ... decoding 32bit version 0 of class NSNumber
Passed test:     (2025-06-10 17:27:23.423 GMT+1) decoding.m:246 ... decoding 64bit version 0 of class NSNumber
Completed file:  decoding.m

     66 Passed tests

This doesn't make me happy, but we can possibly live with it for the initial iteration.

Small tangential note:

Although I am talking about Bazel here, this is not indicating an intention to switch to Bazel or even produce the rules to do so; currently, Bazel's actual Objective-C rules are still assuming Apple platforms, and it is tricky to even get gcc or clang to build a .m / .mm file. And unless native Bazel rules are used (i.e. if a simple genrule just uses cmake/gnustep-make), remote build execution cannot efficiently cache the intended outputs, or reason about whether they have changed. Writing GNUstep-specific rules, rather than Objective-C rules, for a limited number of targets would not be overly difficult, assuming we support only a few 'toolchains' (in Bazel meaning of the word) and it would give the benefits of remote build execution and caching per-target. Understanding how to write a custom Objective-C toolchain is tricky, and writing one that doesn't make assumptions about running under Apple is even more so.

But -- even if integration with Bazel is tricky, and we'd have a harder time making use of RBE platforms that use it well, JUnit tests have further uses. Github Actions have formatters that will display nicer reports than what's available without using them.