GNU testsuite comparison:

Skip an intermittent issue tests/tail/overlay-headers (fails in this run but passes in the 'main' branch)

GNU testsuite comparison:

Skipping an intermittent issue tests/timeout/timeout (passes in this run but fails in the 'main' branch)

@Misakait please keep in mind that Humans are reviewing PR, not AI...

• comment 0 isn't useful. please provide only relevant information
• the ptx change is way too big. It needs to be split in different PR to be reviewed.
• test data and references should be generated on the fly

@Misakait please keep in mind that Humans are reviewing PR, not AI...

• comment 0 isn't useful. please provide only relevant information
• the ptx change is way too big. It needs to be split in different PR to be reviewed.
• test data and references should be generated on the fly

Hi @sylvestre,

Thank you very much for your review and valuable feedback. I sincerely apologize for the size of this PR. I understand that large PRs place a significant burden on maintainers, and I struggled for a long time trying to split it into smaller pieces, but found it nearly impossible. The reason is that these changes are, by their nature, highly architecturally coupled.

In response to your first point, I'd like to explain my process and the dependency chain, and I would be extremely grateful for your advice on how to split this, or for any alternative refactoring path I should follow. As a newcomer, this is an area I find very challenging.

My primary goal was to implement the -S (sentence-regexp) mode.

To achieve this, I discovered that a series of tightly linked changes were necessary:

1. Enabling Stream Processing: The first step was to introduce stream-based file reading. I modified the get_config function and the Config struct to accept the context_regex parameter.

2. Refactoring the Core Pipeline: To support -S, the entire processing pipeline (read_input -> create_word_set -> write_traditional_output) had to be adapted.

   • In read_input, I added the logic to read the entire file content into a new full_content field within the FileContent struct, preparing it for create_word_set.

   • In create_word_set, I introduced the core dual-mode logic: a stream-mode path for when context_regex is set, and a line-mode path for the traditional behavior. This fundamental split required introducing the PtxResult struct and the WordRef enum with its LineWordRef and StreamWordRef variants.

3. Decoupling Formatting Logic: Finally, write_traditional_output had to be refactored. The calculation of layout chunks (tail, before, etc.) was moved up from the format-specific functions (format_tex_line, etc.) into a new set of prepare_*_chunks functions. The dispatch to either prepare_line_chunks or the new prepare_stream_chunks is now determined by the WordRef variant.

This is essentially what my first commit does. My difficulty in splitting it is that these changes are an interdependent chain. A PR that only modifies read_input, for example, would introduce an unused field and leave the architecture in a broken state. All these changes must be applied together to be meaningful.

My second commit is dedicated to fixing the numerous compatibility bugs that the new architecture revealed. I could not bring myself to submit a PR full of bugs. This second commit primarily refactors the get_output_chunks function to precisely replicate GNU's behavior by implementing truncation detection, the left_field_start jump optimization, and compatibility quirks like the '-1' effective length for empty fields. It was only after this commit that the output finally matched the GNU version.

Regarding the on-the-fly test generation, you are absolutely right. I will work on modifying that.

Thank you for taking the time to read this long explanation. I have many questions and would be very grateful for any advice you can offer to help me move forward. Thank you!