@cason I've already implemented this logic, it's the concept of runner executions and I was using it to better test the e2e nightly failures.

https://github.com/cometbft/cometbft/tree/andy/e2e-preserve-logs

you can run with "executions" enabled

./build/runner -f networks/nightly/gen-group01-0004.toml --execution

I[2024-09-20|13:53:42.069] Starting initial network nodes...            
I[2024-09-20|13:53:42.070] load                                         msg="Starting transaction load (16 workers)..."
....
I[2024-09-20|13:55:17.123] wait until                                   msg="Waiting for all nodes to reach height 51..."
I[2024-09-20|13:55:26.540] Running all tests in ./tests/...             
ok      github.com/cometbft/cometbft/test/e2e/tests     46.244s
I[2024-09-20|13:56:15.085] saving execution                             msg="saving e2e network execution information"

and it will save all the logs and manifest used in each run (could add logic to save the node data/config too), the nice thing you can run the same manifest multiple times, maybe modifying a parameter, and still keep the old results because it saves each run within a time date folder

./networks_executions/gen-group01-0004
└── 20240920_095615
    ├── manifest.toml
    └── nodes
        ├── validator01
        │   ├── docker-errors.log
        │   └── docker.log
        ├── validator02
        │   ├── docker-errors.log
        │   └── docker.log
        ├── validator03
        │   ├── docker-errors.log
        │   └── docker.log
        └── validator04
            ├── docker-errors.log
            └── docker.log

This logic can be expanded on many fronts, like some ideas parse the results and generate automated reports in order to improve the time it currently takes to comb through the results or eventually do diffs of results based on tweaks between runs 😉

I just didn't have time to port this to main because I was busy investigation the nightly failures but if you think this is the right direction and should be prioritized I'd be happy to work on it.

@andynog , this is great, should we open a PR on this?

(could add logic to save the node data/config too)

Yes, I would keep everything.

@cason, I'll try to find time this week to push a PR with this logic 👍

Thank you for this code, @andynog, great work.

I was wondering, however, whether we should add a command to the runner to dump all this information. So we can run the multiple actions, one by one, and after stopping the network but before cleaning-up the execution, we could also "manually" dump and the relevant data.

Ok, it is true that the Comet's home directories for every node are under networks/<manifest> and they won't change once we invoke the stop command. But this is not true for the logs, isn't it?

This issue can be associated with #4113.

PR #4165 was closed since the solution to this problem needs additional design considerations that needs to be addressed/defined before a logic can be implemented.