I think streaming the TestRunners output instead of building our tool agnostic on top has the following benefits

• people get the output of their test tool of choice, which they are used to
• we get rid of this huge ugly red block when the initial test runs fails which tries to explain to the end-user how he can debug the problem
• I think we can drop a lot of code in infection namely
  • InitialTestsConsoleLoggerSubscriberFactory
  • CiInitialTestsConsoleLoggerSubscriber
  • InitialTestsConsoleLoggerSubscriber
  • .. and corresponding events
• when using the tool native output we automatically get github annotations with the same problem-matchers, as the output is 1:1

I've tested it locally, and I think it makes the output much much worse, unfortunately.

1. tried to run bin/infection - we have now 5000+ dots, PHPUnit's information (header), all the deprecations and finally those needed tests numbers. Too many things that I don't really need as an Infection user, IMO

What we need from the underlying test framework (again - my opinion) is

• version
• it passes or not (when not - it's hard stop)
• and the number of tests

so only 2 visible short strings would be really needed, like PHPUnit 11.5 and Ran 5432 tests. But now we have hundreds of lines with very verbose output

2. Then I tried to run bin/infection --no-progress

Much less output than in p.1, but still 30-35 lines for Infection itself:

PHPUnit 11.5.25 by Sebastian Bergmann and contributors.
   
   Runtime:       PHP 8.2.28 with Xdebug 3.4.1
   Configuration: /tmp/infection/phpunitConfiguration.initial.infection.xml
   Random Seed:   1751128324
   
   Time: 00:11.943, Memory: 96.00 MB
   
   There was 1 error:
   
   1) Infection\Tests\Process\Runner\InitialTestsRunnerTest::test_it_creates_a_process_execute_it_and_dispatch_events_accordingly
   ArgumentCountError: Too few arguments to function Infection\Process\Runner\InitialTestsRunner::__construct(), 2 passed in /opt/infection/tests/phpunit/Process/Runner/InitialTestsRunnerTest.php on line 98 and exactly 3 expected
   
   /opt/infection/src/Process/Runner/InitialTestsRunner.php:56
   /opt/infection/tests/phpunit/Process/Runner/InitialTestsRunnerTest.php:98
   
   --
   
   2 tests triggered 2 PHPUnit deprecations:
   
   1) Infection\Tests\Config\ValueProvider\TestFrameworkConfigPathProviderTest::test_it_automatically_guesses_path
   onConsecutiveCalls() is deprecated and will be removed in PHPUnit 12. Use $double->willReturn() instead of $double->will($this->onConsecutiveCalls())
   
   /opt/infection/tests/phpunit/Config/ValueProvider/TestFrameworkConfigPathProviderTest.php:130
   
   2) Infection\Tests\Config\ValueProvider\TestFrameworkConfigPathProviderTest::test_it_asks_question_if_no_config_is_found_in_current_dir
   onConsecutiveCalls() is deprecated and will be removed in PHPUnit 12. Use $double->willReturn() instead of $double->will($this->onConsecutiveCalls())
   
   /opt/infection/tests/phpunit/Config/ValueProvider/TestFrameworkConfigPathProviderTest.php:97

This information is not needed and is very noisy.

we get rid of this huge ugly red block when the initial test runs fails which tries to explain to the end-user how he can debug the problem

I wouldn't say it's huge and ugly. We built it based on users' feedback, and it really helps users to think about underlying potential issues, because Infection always run tests in a random order, while project could run them sequentially - this is a very important information. I think if this error message is not useful and ugly, this is a good candidate for a separate issue.

My personal conclusion is: I would better parse PHPUnit's output to get that minimal and nice

Ran 321 tests

rather than displaying all the PHPUnits output for a user.

It feels like "Ran 321 tests" adds on maintenance burden for reasons that I'm struggling to see. How's this number is essential for the users? How's it going to make the user's experience far worse if we don't show it?

(I use Infection with --no-progress most of the time, so there's some bias here.)

let me re-phrase what the actual problem is what I am trying to solve here.

we discussed at several places about --skip-initial-tests and the pre-condition to have a green test-suite before infection should be run. we also discussed whether pre-generated coverage data makes sense since it can easily get out of date because implementation changes can make the coverage data wrong.

so with this PR, my take is:

• run the regular test-suite in the "initial-tests" phase in a similar way the user would, to make sure we have up2date coverage data and a green test suite
• give the user the output he is used to from his test-suite which he currently needs to run separatly before infection
• whether/how we print the progress of the test-suite at this stage is a tiny detail which can be made configurable.

=> the idea: with this PR infection gets a real replacement for the usual project test-runner. I try to simplify how people use infection. it is no longer an additional step the user needs todo after running the regular test-suite.

So what you are saying folks are using vendor/bin/infection as a replacement for calling vendor/bin/phpunit, right?

yes.

I think if infection users could do that, that would ease integration of infection into their projects and simplify the setup -> reduce hurdles. it would simplify ci configs etc.

the PR is just a POC. and a stable solution to achieve that might be more involved.

I guess the main problem is how fast can we run the test-suite (the feedback loop for the enduser with the test-runner) so when infection would replace the call to vendor/bin/phpunit, this can only work if we can be similar fast in reporting test errors back to the user (this might for example mean we need to run the test runner twice behind the scenes, in case coverage collection has a too big impact).

Btw: I am in contact with a php-src dev who will look into making pcov faster