Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Youtube] HTTP Error 403: Forbidden Retrying fragment 1 #32857

Closed
tansy opened this issue Jul 17, 2024 · 5 comments
Closed

[Youtube] HTTP Error 403: Forbidden Retrying fragment 1 #32857

tansy opened this issue Jul 17, 2024 · 5 comments
Labels
duplicate

Comments

@tansy
Copy link

tansy commented Jul 17, 2024

Checklist

  • [ x ] I'm reporting a broken site support issue
  • [ x ] I've verified that I'm running youtube-dl version 2024.07.11
  • [ x ] I've checked that all provided URLs are alive and playable in a browser
  • [ x ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
  • [ x ] I've searched the bugtracker for similar bug reports including closed ones
  • [ x ] I've read bugs section in FAQ

Verbose log

PASTE VERBOSE LOG HERE

Main part here:

$ youtube_dl --version
2024.07.11

$ youtube_dl -v -f 242 https://www.youtube.com/watch?v=pXoZQsZP2PYOA

(...)
[dashsegments] Total fragments: 1
[download] Destination: How To Measure The Tiniest Forces In The Universe-pXoZQsZP2PY.webm
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 1 of 10)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 2 of 10)...
(...)
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 10 of 10)...
ERROR: giving up after 10 fragment retries
Traceback (most recent call last):
  File "/home/user/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/dash.py", line 53, in real_download
    success, frag_content = self._download_fragment(ctx, fragment_url, info_dict, headers)
  File "/home/user/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/fragment.py", line 111, in _download_fragment
    success = ctx['dl'].download(fragment_filename, fragment_info_dict)
  File "/home/user/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/common.py", line 380, in download
    return self.real_download(filename, info_dict)
  File "/home/user/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/http.py", line 349, in real_download
    establish_connection()
  File "/home/user/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/http.py", line 116, in establish_connection
    raise err
HTTPError: HTTP Error 403: Forbidden

Whole log is here.

Description

WRITE DESCRIPTION HERE

Every so often I got this error now: `HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 1 of 10)...'.
Another try usually helps.
@dirkf
Copy link
Contributor

dirkf commented Jul 18, 2024

Missing start of log for future reference 
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-f', u'242', u'https://www.youtube.com/watch?v=pXoZQsZP2PYOA']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2024.07.11 [16f5bbc46] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 2.7.3 (CPython i686 32bit) - OpenSSL 1.0.1e 11 Feb 2013 - glibc 2.0
[debug] exe versions: none
[debug] Proxy map: {}
[youtube] pXoZQsZP2PY: Downloading webpage
[debug] [youtube] Decrypted nsig ZDoZYvl1v0qHRzlPEZb => BaQcXN_ZpQR6MA
[debug] [youtube] Decrypted nsig AylcO5S57E5PtqJE42C => aXTmEZHgb9ZKdg
[debug] Invoking downloader on u'https://rr3---sn-u2oxu-3ufs.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&ei=Bz2YZuL2Fuiii9oPyuC1mAY&ip=2a00%3Af41%3A28e3%3A29bf%3A21f%3A3cff%3Afe31%3Af779&clen=10150654&spc=NO7bAVIKmyOhsUlw9KhnR1nnCGTxyxQcQPDKKtecqnI-NQm0-lODGVVxie6-&id=o-AKN1ov5VYhfnj6Kp6yJIEzo5Nn3c3pTpMyV62WXlRV9p&txp=4537434&svpuc=1&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&gir=yes&xpc=EgVo2aDSNQ%3D%3D&requiressl=yes&keepalive=yes&source=youtube&mv=m&sig=AJfQdSswRQIgSm57CY7K6JmWAN0i8ByypNqygNGWgjcJnzhNFPSn-AYCIQCyrON9mfuxSurDxdXm1rTKWNRZiyCwnTzBxtEbhCXr8Q%3D%3D&dur=933.849&ns=P2Zql4U4Ekivwzqq377hlP8Q&initcwndbps=930000&vprv=1&lsig=AHlkHjAwRAIgS4ZfA4iJ7qWJXhHVSqjDjSHbmZYOii-OMhIVBhzSWkMCIHHju3hSCNRjDcGRNrmZXYA4k3-xalu8lHOQszrq059v&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&
lmt=1697328299515221&c=WEB&sefc=1&bui=AXc671L48KVRcvcPV98Cqm92KKKtwQuexSsL1VLX-4ka56NT65nQVFuMqj7A1NfsbF075Lt7TC339jY7&mime=video%2Fwebm&fvip=2&rqh=1&itag=242&mm=31%2C29&mn=sn-u2oxu-3ufs%2Csn-u2oxu-f5fr&mh=bj&n=aXTmEZHgb9ZKdg&mt=1721252692&expire=1721274727&pl=44&ms=au%2Crdu&mvi=3'
[dashsegments] Total fragments: 1
...

Works for me, but then this is acknowledged to be intermittent.

$ ~/Downloads/youtube-dl-20240711 -v -f 242 'https://www.youtube.com/watch?v=pXoZQsZP2PYOA'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-f', u'242', u'https://www.youtube.com/watch?v=pXoZQsZP2PYOA']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2024.07.11 [16f5bbc46] (single file build)
[debug] ** This version was built from the latest master code at https://github.com/ytdl-org/youtube-dl.
[debug] ** For support, visit the main site.
[debug] Python 2.7.18 (CPython i686 32bit) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial - OpenSSL 1.1.1w  11 Sep 2023 - glibc 2.15
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[youtube] pXoZQsZP2PY: Downloading webpage
[debug] [youtube] Decrypted nsig LhPRmKb2VzLGQSTzQse => IBI3bzt7gGQ55Q
[debug] [youtube] Decrypted nsig 7M39DFZ9V6MXTFVIB42 => fp0yTzErjEhAFQ
[debug] Invoking downloader on u'https://rr3---sn-cu-aigss.googlevideo.com/videoplayback?sparams=expire%2Cei%2Cip%2Cid%2Caitags%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&ei=-FaYZv2GEZ-Bp-oP8KOd0A4&ip=46.208.6.101&clen=10150654&spc=NO7bAfmDI6LyTxDKMvASeX54Xj0tctQwhcgRHPOeg_-9Hk7hjmT_9_v88zBc&id=o-ABxsDcX3NLMTC5b5krnycSJokIXO7JUDJFGd8__xSUw-&txp=4537434&svpuc=1&aitags=133%2C134%2C135%2C136%2C137%2C160%2C242%2C243%2C244%2C247%2C248%2C271%2C278%2C313%2C394%2C395%2C396%2C397%2C398%2C399%2C400%2C401&gir=yes&xpc=EgVo2aDSNQ%3D%3D&requiressl=yes&keepalive=yes&source=youtube&mv=m&sig=AJfQdSswRgIhAO9MhuWrcmKBUYjMwvtR965FWRrC-Bmrkmtcv0ewlya1AiEAtTQNGoztE9gVNSJZKgtmWMtlDIMT9U6XFs9y0plklWY%3D&dur=933.849&ns=9Dw_hmBZRT2bXWVROpf803sQ&initcwndbps=1946250&vprv=1&lsig=AHlkHjAwRQIgBmXcV-PpIhTHadOb2D8EfCxnTvp7x1ijIK9vrL-1QO4CIQDUejY6cI3uTnHdW-oq3X3EfIe6AeIcx26xr1l_bPKm8A%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lmt=1697328299515221&c=WEB&sefc=1&bui=AXc671LKT16IyOEnHudIdJsR-tUyVMOfnmNj1u78R_WK2o6diL0iMCfD2JqEtkuDyB9GkUL5C2dgs876&mime=video%2Fwebm&fvip=2&rqh=1&itag=242&mm=31%2C29&mn=sn-cu-aigss%2Csn-cu-c9id&mh=bj&n=fp0yTzErjEhAFQ&mt=1721259423&expire=1721281368&pl=25&ms=au%2Crdu&mvi=3'
[dashsegments] Total fragments: 1
[download] Destination: How To Measure The Tiniest Forces In The Universe-pXoZQsZP2PY.webm
[download] 100% of 9.68MiB in 00:08
$

There is --fragment-retries ... but I expect that each individual run will either fail or succeed depending on which YT server (eg, rr3---sn-u2oxu-3ufs.googlevideo.com vs rr3---sn-cu-aigss.googlevideo.com). Or you may be seeing yt-dlp/yt-dlp#10397: we don't (yet) make the check that closed that issue.

@tansy
Copy link
Author

tansy commented Jul 18, 2024 •
edited
Loading

Missing start of log for future reference

Will include it in post instead of paste next time.

works for me

Standard administrator's response # 1.

There is --fragment-retries ... but I expect that each individual run will either fail or succeed

Indeed.

(...)
[youtube] awjxwVIkvMo: Downloading webpage
[debug] [youtube] Decrypted nsig VEFA-bqqkpop_nN-Bhb => I33IfCp1NVzvfg
[debug] [youtube] Decrypted nsig TmxtWBpHptzmScpXdlO => ujkuuHDDDUwzeQ
[debug] Invoking downloader on u'https://rr2---sn-u2oxu-3ufy.googlevideo.com/videoplayback?...
[dashsegments] Total fragments: 1
[download] Destination: Tsunami 2004-awjxwVIkvMo.webm
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 1 of 100)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 2 of 100)...
(...)
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 99 of 100)...
[download] Got server HTTP error: HTTP Error 403: Forbidden. Retrying fragment 1 (attempt 100 of 100)...
ERROR: giving up after 100 fragment retries
Traceback (most recent call last):
  File "/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/dash.py", line 53, in real_download
    success, frag_content = self._download_fragment(ctx, fragment_url, info_dict, headers)
  File "/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/fragment.py", line 111, in _download_fragment
    success = ctx['dl'].download(fragment_filename, fragment_info_dict)
  File "/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/common.py", line 380, in download
    return self.real_download(filename, info_dict)
  File "/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/http.py", line 349, in real_download
    establish_connection()
  File "/tmp/ytdl-nightly-2024.07.11/youtube_dl/downloader/http.py", line 116, in establish_connection
    raise err
HTTPError: HTTP Error 403: Forbidden

Overall result is the same.

Of all formats webm seems to be affected. In this run YT formats 278, 133 failed, in the next one all went fine, and the next one 242 failed.
Here, they all are only mp4, and three times all went well.

@dirkf
Copy link
Contributor

dirkf commented Jul 18, 2024

You could apply this patch to determine if the problem is the one from yt-dlp/yt-dlp#10397:

--- old/youtube_dl/extractor/youtube.py
+++ new/youtube_dl/extractor/youtube.py
@@ -1877,6 +1877,14 @@
             player_response = self._extract_yt_initial_variable(
                 webpage, self._YT_INITIAL_PLAYER_RESPONSE_RE,
                 video_id, 'initial player response')
+            POTOKEN_EXPERIMENTS = ('51217476', '51217102')
+            experiments = traverse_obj(self._extract_ytcfg(video_id, webpage), (
+                'WEB_PLAYER_CONTEXT_CONFIGS', Ellipsis, 'serializedExperimentIds',
+                T(lambda x: x.split(',')), Ellipsis))
+            if all(x in experiments for x in POTOKEN_EXPERIMENTS):
+                self.report_warning(
+                    'Webpage contains broken formats (poToken experiment detected).')
+
         if not player_response:
             player_response = self._call_api(
                 'player', {'videoId': video_id}, video_id)

@tansy
Copy link
Author

tansy commented Jul 19, 2024

It doesn't seem to work any better.

Tried to test yt-dlp with it but they disagree on available formats. yt-dlp-2024.07.07 cannot see all formats. yt-dlp-2024.07.16 does.
So, all I did was to test the latter and it seems to work. Sometimes gives the warning (`WARNING: [youtube] Webpage contains broken formats (poToken experiment detected). Ignoring initial player response') but downloads it.
Not sure if they have anything to do with each other.
You can see all logs here.

@dirkf
Copy link
Contributor

dirkf commented Aug 9, 2024

Continued in #32905.

@dirkf dirkf closed this as not planned Won't fix, can't repro, duplicate, stale Aug 9, 2024
@dirkf dirkf changed the title YT: HTTPError: HTTP Error 403: Forbidden Retrying fragment 1 [Youtube] HTTP Error 403: Forbidden Retrying fragment 1 Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
duplicate
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dirkf @tansy