[aria2c] Sometimes the download suddenly stops responding #5931

D0LLYNH0 · 2023-01-02T23:50:19Z

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

I understand that I will be blocked if I remove or skip any mandatory* field

Checklist

I'm reporting a bug unrelated to a specific site
I've verified that I'm running yt-dlp version 2023.01.02 (update instructions) or later (specify commit)
I've checked that all provided URLs are playable in a browser with the same IP and same login details
I've checked that all URLs and arguments with special characters are properly quoted or escaped
I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood

Sometimes the downloads suddenly stop responding, sometimes at ~80%, ~90%, when this happens I have to end the process and start it again to complete the download.

After a few hours of downloading random content and witnessing these behaviors, I found a URL that always stops at 100%, the download never completes. The sample URL is in the report, I believe it will be invalid after a few hours, but I can generate a new one anytime, just let me know.

Using --compat-options no-external-downloader-progress works around these behaviors.

Provide verbose output that clearly demonstrates the problem

Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['-vU', '-o', 'files/sample.%(ext)s', '--ffmpeg-location', './', '--downloader', 'aria2c', 'https://apd-vlive.apdcdn.tc.qq.com/defaultts.tc.qq.com/svp_50112/ELoeXke2jnYfJZ2yBDwxjd-gDF0MH69HK6d95Q8ZRsMZqI6Id_bxeEZJGh6jNraHwJvg-QNDY5tyswxs12_9MvbM-mOXaiaE2jSVGXkJr8UEZX-muFH7dGDzzvO7W-A6ZS7cS9YR4dtdwlDARYr9lCLQ1nu3oMU5Sft62ZWRgP8/gzc_1000102_0b53yqaacaaam4adxobvzfr4brgdahbaabka.f322016.ts.m3u8?ver=4']
[debug] User config: []
[debug] System config: []
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2023.01.02 [d83b0ad80]
[debug] Lazy loading extractors is disabled
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.4.0-135-generic-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.4.1-static (setts), ffprobe 4.4.1-static, phantomjs 2.1.1
[debug] Optional libraries: Cryptodome-3.15.0, certifi-2022.09.24, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1754 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2023.01.02, Current version: 2023.01.02
yt-dlp is up to date (2023.01.02)
[generic] Extracting URL: https://apd-vlive.apdcdn.tc.qq.com/defaultts.tc.qq.com/svp_50112/ELoeXke2jnYfJZ2yBDwxjd-gDF0MH69HK6d95Q8ZRsMZqI6Id_bxeEZJGh6jNraHwJvg-QNDY5tyswxs12_9MvbM-mOXaiaE2jSVGXkJr8UEZX-muFH7dGDzzvO7W-A6ZS7cS9YR4dtdwlDARYr9lCLQ1nu3oMU5Sft62ZWRgP8/gzc_1000102_0b53yqaacaaam4adxobvzfr4brgdahbaabka.f322016.ts.m3u8?ver=4
[generic] gzc_1000102_0b53yqaacaaam4adxobvzfr4brgdahbaabka.f322016.ts: Downloading webpage
[debug] Identified a direct video link
[generic] gzc_1000102_0b53yqaacaaam4adxobvzfr4brgdahbaabka.f322016.ts: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] gzc_1000102_0b53yqaacaaam4adxobvzfr4brgdahbaabka.f322016.ts: Downloading 1 format(s): 0
[debug] Invoking hlsnative downloader on "https://apd-vlive.apdcdn.tc.qq.com/defaultts.tc.qq.com/svp_50112/ELoeXke2jnYfJZ2yBDwxjd-gDF0MH69HK6d95Q8ZRsMZqI6Id_bxeEZJGh6jNraHwJvg-QNDY5tyswxs12_9MvbM-mOXaiaE2jSVGXkJr8UEZX-muFH7dGDzzvO7W-A6ZS7cS9YR4dtdwlDARYr9lCLQ1nu3oMU5Sft62ZWRgP8/gzc_1000102_0b53yqaacaaam4adxobvzfr4brgdahbaabka.f322016.ts.m3u8?ver=4"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Fragment downloads will be delegated to aria2c
[hlsnative] Total fragments: 149
[download] Destination: files/sample.mp4
[debug] aria2c command line: aria2c -c --console-log-level=warn --summary-interval=0 --download-result=hide --http-accept-gzip=true --file-allocation=none -x16 -j16 -s16 --allow-overwrite=true --allow-piece-length-change=true --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.41 Safari/537.36' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-us,en;q=0.5' --header 'Sec-Fetch-Mode: navigate' --check-certificate=true --remote-time=true --show-console-readout=true --enable-rpc --rpc-listen-port=46451 --rpc-secret=aa4ddebd-6360-4763-ae80-57b1a9e4fa6f --dir ./files/ --auto-file-renaming=false --file-allocation=none --uri-selector=inorder -i ./files/sample.mp4.part.frag.urls
[download] 100.0% of ~ 775.39MiB at    1.40MiB/s ETA 00:00 (frag 149/149)

The text was updated successfully, but these errors were encountered:

coletdjnz · 2023-01-02T23:56:53Z

cc @Lesmiscore

Lesmiscore · 2023-01-03T07:02:20Z

TL;DR: I'd say to avoid aria2c all the time, at least for yt-dlp. It's not your superhero. If you still want to use, use --compat-options no-external-downloader-progress as always and limit it for just plain HTTP downloads by --downloader http:aria2c

aria2c is particularly weird and unstable downloader amongst the all others - it sometimes stuck without explaining why, or even it can't download those that works 100% with native downloader.

Honestly, I don't want to "fight" aria2c to dig in its behavior. Author doesn't know about functionalities of HTTP servers¹, they don't give any response for Socks5 support request² and leading/trailing spaces are automagically removed... Fighting aria2c is an insane work, anyway.

There is no wonder even if it starts strike while being accessed via RPC... because it's aria2c. Aria2c is unstable most of the time.

@pukkandan I regret I can't say that because the changes were suddenly made, but it should be enabled by user, instead of by default. Thoughts?

You can limit number of connections on server-side ↩
A good tool named proxychains? No, aria2c bypasses proxychains, torsocks etc. The only one option is to "convert" the proxy to HTTP proxy, which requires an another tool to do that ↩

rebane2001 · 2023-01-03T09:03:02Z

I'd argue this is still something that should be fixed or disabled by default in yt-dlp. I've rarely had problems with aria2c in the past, but this new update without the compat options constantly keeps getting stuck.

That being said, if what @Lesmiscore is saying is true, imo the README should mention that aria2c shouldn't be used. At the moment it seems to imply that setting it is a good thing.

pukkandan · 2023-01-03T11:39:20Z

After a number of requests, the RPC simply stops responding. The only reference for this I could find is aria2/aria2#515. The issue isn't even limited to fragmented downloads, but happens for any large file.

For now, the only solution I can see is to revert the progress reporting feature

Lesmiscore · 2023-01-03T11:48:28Z

Can't no-external-downloader-progress be flipped or require an another flag for that? (like ytdl-patched requires --enable-native-progress for it)

pukkandan · 2023-01-03T11:53:47Z

If we could timeout the RPC call, the the feature could be salvaged. But the issue is happening in a way that doesn't obey --socket-timeout. So once the progress is stuck, it's unrecoverable and user is forced to manually close and restart yt-dlp. I'm not sure anyone would want to use the feature with this caveat.

anasty17 · 2023-01-04T04:40:20Z

If we could timeout the RPC call, the the feature could be salvaged. But the issue is happening in a way that doesn't obey --socket-timeout. So once the progress is stuck, it's unrecoverable and user is forced to manually close and restart yt-dlp. I'm not sure anyone would want to use the feature with this caveat.

Check aria2p listener, it's working well. Never missed any completed download.

He using websocket with variable timeout.

https://github.com/pawamoy/aria2p/blob/5c91827dcbb88c7ade012a5762cdf3750c2eeb69/src/aria2p/client.py#L1689

jazz1611 · 2023-01-05T10:06:16Z

hello @pukkandan , i get stuck at 100% when have --downloader aria2c. in old version i don't have problem with aria2c. after update to yt-dlp version 2023.01.02 it always stuck at 100%

[generic] Extracting URL: https://rr5---sn-5goeenes.c.drive.google.com/videoplayback?expire=1672925823&ei=P5q2Y5zPB6SP2LYP-...LhBbnv_jmSv2zX9pYQ==
[generic] mp4&vprv=1&prv=1&dur=3114: Downloading webpage
[redirect] Following redirect to https://rr4---sn-5hnekn7z.c.drive.google.com/videoplayback?expire=1672925823&ei=P5q2Y5zPB6SP2LYP-ZKlkAg&ip=2a01:4f9:1a:995d::2&cp=QVRMVEpfVFhQRFhPOmYxMlMtNGY3R0VzWEdNYzhxUXB1aENBaWpzN0hrU1VCb29VbWE4SW5ZQXU&id=61b3f921cdd77876&itag=22&source=webdrive&requiressl=yes&ttl=transient&susc=dr&driveid=1XWILFllsK6zCCNj5cCDHWiKgW92rpH6j&app=explorer&mime=video/mp4&vprv=1&prv=1&dur=3114.376&lmt=1663399606861306&subapp=NONE&txp=0016224&sparams=expire%2Cei%2Cip%2Ccp%2Cid%2Citag%2Csource%2Crequiressl%2Cttl%2Csusc%2Cdriveid%2Capp%2Cmime%2Cvprv%2Cprv%2Cdur%2Clmt&sig=AOq0QJ8wRQIgIHPCWSx_QtFfmAz740PptJ_wFWvj9L6NmXNBZxxVyPQCIQDGjcpOvY-kWbUeXFABFKZmSNfrtXGdF6dqyd9LPkZGfA==&redirect_counter=1&cm2rm=sn-5goly76&req_id=f01699397d3a3ee&cms_redirect=yes&cmsv=e&mh=_i&mm=34&mn=sn-5hnekn7z&ms=ltu&mt=1672911171&mv=m&mvi=4&pl=44&lsparams=mh,mm,mn,ms,mv,mvi,pl&lsig=AG3C_xAwRAIgcqG-rX7DB4Z8mQWt-WdOI6-5mBSEivAe8Re2iCpMBKwCICRvLMKlhDQIxILN7aiXG7g6GWSgo9YaZJGBZEvo9ISA
[generic] Extracting URL: https://rr4---sn-5hnekn7z.c.drive.google.com/videoplayback?expire=1672925823&ei=P5q2Y5zPB6SP2LYP-...GWSgo9YaZJGBZEvo9ISA
[generic] mp4&vprv=1&prv=1&dur=3114: Downloading webpage
[info] mp4&vprv=1&prv=1&dur=3114: Downloading 1 format(s): fmp4
[download] Destination: /www/wwwroot/domain.com/videos/720P/2023/01/03/bf2c6905da6164356e67f4691485f3e8.mp4

[download]      0.00B at  Unknown B/s (00:00:00)
[download]   0.0% of  518.47MiB at  559.61KiB/s ETA 15:48
[download]   0.0% of  518.47MiB at  315.03KiB/s ETA 28:05
[download]   0.0% of  518.47MiB at    1.07MiB/s ETA 08:02
[download]   0.2% of  518.47MiB at    2.49MiB/s ETA 03:28
[download]   0.3% of  518.47MiB at    3.09MiB/s ETA 02:47
[download]   0.4% of  518.47MiB at    3.50MiB/s ETA 02:27
[download]   0.5% of  518.47MiB at    3.81MiB/s ETA 02:15
[download]   0.6% of  518.47MiB at    3.98MiB/s ETA 02:09
[download]   0.7% of  518.47MiB at    4.16MiB/s ETA 02:03
[download]   0.8% of  518.47MiB at    4.30MiB/s ETA 01:59
[download]   0.9% of  518.47MiB at    4.38MiB/s ETA 01:57
[download]   1.1% of  518.47MiB at    4.48MiB/s ETA 01:54
[download]   1.2% of  518.47MiB at    4.54MiB/s ETA 01:52
[download]   1.3% of  518.47MiB at    4.61MiB/s ETA 01:51
...
[download]  97.7% of  518.47MiB at    5.32MiB/s ETA 00:02
[download]  97.8% of  518.47MiB at    5.34MiB/s ETA 00:02
[download]  98.1% of  518.47MiB at    5.42MiB/s ETA 00:01
[download]  98.2% of  518.47MiB at    5.41MiB/s ETA 00:01
[download]  98.3% of  518.47MiB at    5.43MiB/s ETA 00:01
[download]  98.6% of  518.47MiB at    5.51MiB/s ETA 00:01
[download]  98.7% of  518.47MiB at    5.50MiB/s ETA 00:01
[download]  98.8% of  518.47MiB at    5.51MiB/s ETA 00:01
[download]  99.1% of  518.47MiB at    5.60MiB/s ETA 00:00
[download]  99.2% of  518.47MiB at    5.63MiB/s ETA 00:00
[download]  99.3% of  518.47MiB at    5.62MiB/s ETA 00:00
[download]  99.7% of  518.47MiB at    5.77MiB/s ETA 00:00
[download] 100.0% of  518.47MiB at    5.84MiB/s ETA 00:00
[download] 100.0% of  518.47MiB at    5.78MiB/s ETA 00:00
[download] 100.0% of  518.47MiB at    5.72MiB/s ETA 00:00

it should end line as below

[download] 100.0% of  461.32MiB at      0.00B/s ETA 00:00
[download] 100% of  461.32MiB in 00:01:10 at 6.51MiB/s

dirkf · 2024-04-07T18:17:06Z

Upstream a similar thing has been implemented using aria2p but maybe no-one has used it hard enough to provoke a similar problem.

D0LLYNH0 added bug Bug that is not site-specific triage Untriaged issue labels Jan 2, 2023

coletdjnz added regression Works in youtube-dl/older yt-dlp and removed triage Untriaged issue labels Jan 2, 2023

coletdjnz added the high-priority label Jan 3, 2023

pukkandan mentioned this issue Jan 3, 2023

[aria2c] getting files with .part extension #5928

Closed

9 tasks

pukkandan closed this as completed in ad68b16 Jan 3, 2023

pukkandan mentioned this issue Jan 3, 2023

[Feature request] Show native progress for aria2c #2038

Open

4 tasks

pukkandan mentioned this issue Jan 6, 2023

🚨[IMPORTANT]🚨 KNOWN ISSUES/FAQ #3766

Open

dirkf mentioned this issue Apr 7, 2024

yt-dlp does not pass signals to the external downloader. #9630

Open

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[aria2c] Sometimes the download suddenly stops responding #5931

[aria2c] Sometimes the download suddenly stops responding #5931

[aria2c] Sometimes the download suddenly stops responding #5931

[aria2c] Sometimes the download suddenly stops responding #5931

Comments

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

Checklist

Provide a description that is worded well enough to be understood

Provide verbose output that clearly demonstrates the problem

Complete Verbose Output

Footnotes