Nothing Special   »   [go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature request: periodic log entries during large file transfers #294

Closed
NinoSkopac opened this issue Aug 20, 2024 · 8 comments · Fixed by #389
Closed

Feature request: periodic log entries during large file transfers #294

NinoSkopac opened this issue Aug 20, 2024 · 8 comments · Fixed by #389
Milestone

Comments

@NinoSkopac
Copy link
Contributor

When downloading large files (e.g., a 1TB file), it would be helpful to receive periodic log entries at defined intervals (e.g., every n seconds) that report on the progress of the file transfer. Instead of a single log entry at the end indicating the completion of the transfer, the system should generate multiple log entries during the transfer, each corresponding to a specific chunk of data delivered.

This feature would provide real-time visibility into the progress of large file transfers, making it easier to monitor and manage the download process, especially in environments where continuous feedback is crucial.

Totally fine if this is behind a flag.

Thank you and have a great day!

@zh-jq-b
Copy link
Member
zh-jq-b commented Aug 21, 2024

Do you want this feature just for layer-7 HTTP request/response, or for all of the layer-4 tunnel connections?

@NinoSkopac
Copy link
Contributor Author
NinoSkopac commented Aug 21, 2024

Layer 4, please. I do love having socks5, it's so much more efficient. In my tests it consumes way less data for the same URL compared to http proxy.

Design note: please do not log the total bytes received on top of chunks' logs.

Example if downloading a 1GB file and my speed was 1MB/s:

  1. log received 1MB
  2. log received 1MB
  3. ...
  4. do not log 1GB received

Thank you Mr. Zhang!!!

@zh-jq
Copy link
Collaborator
zh-jq commented Aug 21, 2024

For tunnel connections, we don't known whether it's downloading files. What can be done is that, log when reading/writing to client connections at some nearly fixed time interval.

@NinoSkopac
Copy link
Contributor Author

That works!

@zh-jq zh-jq added this to the g3proxy v1.12 milestone Aug 24, 2024
@NinoSkopac
Copy link
Contributor Author
NinoSkopac commented Sep 13, 2024

Note: auth and its refresh interval must be respected. As in, a user could be valid now, but invalid at a later chunk/interval. In that case the connection must be terminated.

Should this be a separate issue? Is this already implemented? (I haven't tested.)

@zh-jq-b
Copy link
Member
zh-jq-b commented Sep 13, 2024

Note: auth and its refresh interval must be respected. As in, a user could be valid now, but invalid at a later chunk/interval. In that case the connection must be terminated.

Should this be a separate issue? Is this already implemented? (I haven't tested.)

You can set a user to blocked state and then it will be blocked after idle_check_interval (5min by default).

@zh-jq-b
Copy link
Member
zh-jq-b commented Nov 20, 2024

The enovy way: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/observability/access_logging#periodic-access-logs.
The plan is to add a task_log_flush_interval and task_log_on_stage_change config options in server config, and just do task log at that flush interval or when task stage changed (new -> connected -> relay).
@NinoSkopac does this work for you?

@NinoSkopac
Copy link
Contributor Author

Sounds good.

I guess for my particular use case I only need to use task_log_flush_interval?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants