[twitter] Broken Python 2 support

[mia-0]

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2019.04.07. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

• I've verified and I assure that I'm running youtube-dl 2019.04.07

Before submitting an issue make sure you have:

• At least skimmed through the README, most notably the FAQ and BUGS sections
• Searched the bugtracker for similar issues including closed ones
• Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

• Bug report (encountered problems with youtube-dl)
• Site support request (request for adding support for a new site)
• Feature request (request for a new functionality)
• Question
• Other

---

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

---

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

$ youtube-dl -v 'https://twitter.com/Shitty_Future/status/1106486925282684928'
[debug] System config: []
[debug] User config: [u'--netrc', u'--sub-lang', u'deDE', u'--cookies', u'/home/lachs0r/ytdl-cookies.txt', u'--hls-prefer-ffmpeg', u'--fixup', u'warn']
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://twitter.com/Shitty_Future/status/1106486925282684928']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.07
[debug] Python version 2.7.15 (CPython) - Linux-5.0.5-1-default-x86_64-with-SuSE-20151201-x86_64
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3
[debug] Proxy map: {}
[twitter] 1106486925282684928: Downloading webpage
[twitter:card] 1106486925282684928: Downloading webpage
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 474, in main

  File "/usr/bin/youtube-dl/youtube_dl/__init__.py", line 464, in _real_main

  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2008, in download
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 807, in extract_info
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 873, in process_ie_result
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in extract
  File "/usr/bin/youtube-dl/youtube_dl/extractor/twitter.py", line 239, in _real_extract
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 2816, in _get_cookies
  File "/usr/lib64/python2.7/Cookie.py", line 583, in __init__
    if input: self.load(input)
  File "/usr/lib64/python2.7/Cookie.py", line 643, in load
    for k, v in rawdata.items():
AttributeError: 'unicode' object has no attribute 'items'

Description of your issue, suggested solution and other information

Encountered this issue on openSUSE Tumbleweed.

It seems at least the Twitter extractor no longer works with Python 2, which is unfortunate because the youtube-dl executable uses a shebang that will invoke the Python 2 interpreter on most distributions. I suggest changing the shebang so it uses Python 3 in addition to fixing this issue.

[mia-0]

Of course I omitted the crucial fact that this only breaks when using a cookie file.

[remitamine]

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v ), copy the whole output and insert it here.

[mia-0]

Updated. Sorry, should have just followed the instructions carefully. Not that it adds anything that wouldn’t have been obvious from the traceback :P

[remitamine]

this patch fixes the problem, but not sure if it's the right way(str function normally not called directly in the project):

diff --git a/youtube_dl/extractor/common.py b/youtube_dl/extractor/common.py
index 0889288f0..efe0006f6 100644
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -2813,7 +2813,7 @@ class InfoExtractor(object):
         """ Return a compat_cookies.SimpleCookie with the cookies for the url """
         req = sanitized_Request(url)
         self._downloader.cookiejar.add_cookie_header(req)
-        return compat_cookies.SimpleCookie(req.get_header('Cookie'))
+        return compat_cookies.SimpleCookie(str(req.get_header('Cookie')))
 
     def get_testcases(self, include_onlymatching=False):
         t = getattr(self, '_TEST', None)

[kpmiller]

I can confirm that I was seeing this same problem, and switching my shell to my python3 install does fix the problem. (Using the same cookies file.)

[remitamine]

_get_cookies method no longer called from twitter extractor, superseded by #23256.

[seahorse41]

I am seeing this same error output after an unattended-upgrade this morning. It was working earlier this week. My os is debian. The site is not twitter however, it is dailymotion.com

kpmiller mentioned python3. I don't understand the details of that fix, if it applies for dailymotion.
Commenting here does not reopen the issue, maybe I should open a new issue after all.

[TaktakTaktouk]

Any news for Dailymotion.com? we really need a fix for this site. Thank you

[meaz]

why is it closed as it doesn't work with dailymotion?
I had to do this:
python3 /usr/local/bin/youtube-dl dailymotionlink
to make it work

[davinkevin]

Same thing for me, I'll like to have a simpler solution ❤️

[seahorse41]

Easy solution is to define an alias, and add it to the home .bash_aliases file
alias youtube-dl='python3 /usr/local/bin/youtube-dl'

[ldexterldesign]

I fixed this by upgrading from python 2 to 3.8.2 (macos, brew and pyenv)

Hope this helps

Regards

[tophee]

Excellent! The latest commit above has fixed the issue. No need for prepending python3 anymore!