`youtube-dl` python encodings error #144

mfts · 2023-01-26T13:02:46Z

$ youtube-dl -h
installed: ~/.tea/invisible-island.net/ncurses/v6.4.0
installed: ~/.tea/curl.se/ca-certs/v2022.7.19
installed: ~/.tea/gnu.org/readline/v8.1.0
installed: ~/.tea/zlib.net/v1.2.13
installed: ~/.tea/sourceware.org/bzip2/v1.0.8
installed: ~/.tea/openssl.org/v1.1.1s
installed: ~/.tea/sourceware.org/libffi/v3.4.4
installed: ~/.tea/libexpat.github.io/v2.5.0
installed: ~/.tea/bytereef.org/mpdecimal/v2.5.1
installed: ~/.tea/tukaani.org/xz/v5.2.7
installed: ~/.tea/sqlite.org/v3.40.0
installed: ~/.tea/python.org/v3.11.1
installed: ~/.tea/gnu.org/bash/v5.1.16
installed: ~/.tea/youtube-dl.org/v2021.12.17
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/python.org/v3.11.1/lib/python3.11'
  sys._base_executable = '/Users/mfts/.tea/python.org/v3/bin/python3.11'
  sys.base_prefix = '/opt/python.org/v3.11.1'
  sys.base_exec_prefix = '/opt/python.org/v3.11.1'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/mfts/.tea/youtube-dl.org/v2021.12.17/libexec/bin/python'
  sys.prefix = '/opt/python.org/v3.11.1'
  sys.exec_prefix = '/opt/python.org/v3.11.1'
  sys.path = [
    '/opt/python.org/v3.11.1/lib/python311.zip',
    '/opt/python.org/v3.11.1/lib/python3.11',
    '/opt/python.org/v3.11.1/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff8563ed8c0 (most recent call first):
  <no Python frame>

after fresh install of tea v0.21.2

mxcl · 2023-01-26T13:31:47Z

lol, like usual: works for me.

@jonchang is our python venv stuff still screwed up you think?

mfts · 2023-01-26T13:51:05Z

my laptop is cursed -_-

mxcl · 2023-02-02T22:56:08Z

@mfts any python env vars set? What's env say?

nihaals · 2023-02-03T12:03:33Z

This is reproducible in Docker (here using poetry):

$ docker run -it --rm ubuntu
root@528b4cf0d3c0:/# apt update && apt install -y curl && sh <(curl https://tea.xyz) -y
...
root@528b4cf0d3c0:/# tea poetry --version
installed: ~/.tea/invisible-island.net/ncurses/v6.4.0
installed: ~/.tea/curl.se/ca-certs/v2022.7.19
installed: ~/.tea/gnu.org/readline/v8.1.0
installed: ~/.tea/zlib.net/v1.2.13
installed: ~/.tea/sourceware.org/bzip2/v1.0.8
installed: ~/.tea/openssl.org/v1.1.1s
installed: ~/.tea/sourceware.org/libffi/v3.4.4
installed: ~/.tea/libexpat.github.io/v2.5.0
installed: ~/.tea/bytereef.org/mpdecimal/v2.5.1
installed: ~/.tea/tukaani.org/xz/v5.2.7
installed: ~/.tea/sqlite.org/v3.40.0
installed: ~/.tea/python.org/v3.11.1
installed: ~/.tea/gnu.org/bash/v5.1.16
installed: ~/.tea/python-poetry.org/v1.3.2
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/python.org/v3.11.1/lib/python3.11'
  sys._base_executable = '/root/.tea/python.org/v3/bin/python3.11'
  sys.base_prefix = '/opt/python.org/v3.11.1'
  sys.base_exec_prefix = '/opt/python.org/v3.11.1'
  sys.platlibdir = 'lib'
  sys.executable = '/root/.tea/python-poetry.org/v1.3.2/libexec/bin/python'
  sys.prefix = '/opt/python.org/v3.11.1'
  sys.exec_prefix = '/opt/python.org/v3.11.1'
  sys.path = [
    '/opt/python.org/v3.11.1/lib/python311.zip',
    '/opt/python.org/v3.11.1/lib/python3.11',
    '/opt/python.org/v3.11.1/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000ffffb95ca840 (most recent call first):
  <no Python frame>
root@528b4cf0d3c0:/# tea python --version
Python 3.11.1

mxcl · 2023-02-03T14:04:30Z

oh interesting. Let me try

mxcl · 2023-02-03T14:16:34Z

@jonchang you're the expert here, any clue?

jhheider · 2023-02-04T01:10:05Z

See my notes on pkgxdev/pkgx#351 about /opt/ (build path) leaking into the package.

mxcl · 2023-02-05T22:22:08Z

I think the /opt references are false flags as in: they've always been there and these are hard to fix without a post-install step, but I'm pretty sure they don't matter.

I might be wrong.

This was working, and our test step passed. It's weird.

mxcl · 2023-02-05T22:23:21Z

like our test is youtube-dl --simulate https://youtu.be/pOtd1cbOP7k so how could it pass??

jhheider · 2023-02-05T23:26:24Z

I suspect but can't prove that pkgxdev/brewkit#42 might have helped.

jhheider · 2023-02-06T03:46:34Z

Happening with meson too:

+ mkdir -p build
+ cd build
+ meson .. --prefix=/Users/jacob/.tea/freedesktop.org/p11-kit/v0.24.1 --libdir=/Users/jacob/.tea/freedesktop.org/p11-kit/v0.24.1/lib --buildtype=release
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/python.org/v3.11.0/lib/python3.11'
  sys._base_executable = '/Users/jacob/.tea/python.org/v3/bin/python3.11'
  sys.base_prefix = '/opt/python.org/v3.11.0'
  sys.base_exec_prefix = '/opt/python.org/v3.11.0'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/jacob/.tea/mesonbuild.com/v0.64.1/libexec/bin/python'
  sys.prefix = '/opt/python.org/v3.11.0'
  sys.exec_prefix = '/opt/python.org/v3.11.0'
  sys.path = [
    '/opt/python.org/v3.11.0/lib/python311.zip',
    '/opt/python.org/v3.11.0/lib/python3.11',
    '/opt/python.org/v3.11.0/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001ea1c7a80 (most recent call first):
  <no Python frame>

I wonder. We used to set one or more PYTHON variables in useShellenv.ts, no? Any chance that's changed it and, if so, we should move that to a runtime.env.KEY in python.org

mfts · 2023-02-06T19:32:21Z

My env

$ env
__CFBundleIdentifier=com.apple.Terminal
TMPDIR=/var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/
XPC_FLAGS=0x0
TERM=xterm-256color
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.kvfguRiAJd/Listeners
XPC_SERVICE_NAME=0
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=447
TERM_SESSION_ID=8D687CC7-62D0-4E79-A981-F7B6005D4CAE
SHELL=/bin/zsh
HOME=/Users/mfts
LOGNAME=mfts
USER=mfts
PATH=/Users/mfts/.tea/tea.xyz/v*/bin:/Users/mfts/.nvm/versions/node/v18.13.0/bin:/Users/mfts/.pyenv/shims:/Users/mfts/.rbenv/shims:/usr/local/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/Library/Apple/usr/bin:/Users/mfts/Library/Android/sdk/emulator:/Users/mfts/Library/Android/sdk/tools:/Users/mfts/Library/Android/sdk/tools/bin:/Users/mfts/Library/Android/sdk/platform-tools
SHLVL=1
PWD=/Users/mfts
OLDPWD=/Users/mfts
VIRTUALENVWRAPPER_PYTHON=/Users/mfts/.pyenv/shims/python
WORKON_HOME=/Users/mfts/.virtualenvs
VIRTUALENVWRAPPER_PROJECT_FILENAME=.project
VIRTUALENVWRAPPER_WORKON_CD=1
VIRTUALENVWRAPPER_SCRIPT=/usr/local/bin/virtualenvwrapper.sh
VIRTUALENVWRAPPER_HOOK_DIR=/Users/mfts/.virtualenvs
RBENV_SHELL=zsh
PYENV_SHELL=zsh
LE_WORKING_DIR=/Users/mfts/.acme.sh
NVM_DIR=/Users/mfts/.nvm
NVM_CD_FLAGS=-q
NVM_BIN=/Users/mfts/.nvm/versions/node/v18.13.0/bin
NVM_INC=/Users/mfts/.nvm/versions/node/v18.13.0/include/node
ANDROID_HOME=/Users/mfts/Library/Android/sdk
LC_CTYPE=UTF-8
_=/usr/bin/env

p.s.: it's a mess

jhheider · 2023-02-06T19:36:06Z

I think I know what might be up. I believe there's been a regression with companions: no longer being included; in this case, pip, which sets PYTHONPATH. Try this: tea +pip.pypa.io +youtube-dl.org youtube-dl "https://youtube.com/watch?v=dQw4w9WgXcQ"

jhheider · 2023-02-06T19:38:30Z

ref: pkgxdev/pkgx#368 (comment)

mfts · 2023-02-06T20:46:43Z

@jhheider unfortunately that didn't work.

It just installed pip and youtube-dl (already installed) and nothing more

$ tea +pip.pypa.io +youtube-dl.org youtube-dl "https://youtube.com/watch?v=dQw4w9WgXcQ"
installed: ~/.tea/pip.pypa.io/v22.3.1
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/Users/mfts/.tea/pip.pypa.io/v22.3.1'
  program name = 'python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/opt/python.org/v3.11.1/lib/python3.11'
  sys._base_executable = '/Users/mfts/.tea/python.org/v3/bin/python3.11'
  sys.base_prefix = '/opt/python.org/v3.11.1'
  sys.base_exec_prefix = '/opt/python.org/v3.11.1'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/mfts/.tea/youtube-dl.org/v2021.12.17/libexec/bin/python'
  sys.prefix = '/opt/python.org/v3.11.1'
  sys.exec_prefix = '/opt/python.org/v3.11.1'
  sys.path = [
    '/Users/mfts/.tea/pip.pypa.io/v22.3.1',
    '/opt/python.org/v3.11.1/lib/python311.zip',
    '/opt/python.org/v3.11.1/lib/python3.11',
    '/opt/python.org/v3.11.1/lib/python3.11/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007ff8476608c0 (most recent call first):
  <no Python frame>
~ $

jhheider · 2023-02-06T20:47:27Z

Hm. Darn, that solved it for me.

mfts · 2023-02-07T10:19:43Z

Remove .tea from system

rm -rf .tea

and ran the code with the tea one-liner abstraction and it worked...

$ sh <(curl tea.xyz) youtube-dl "https://youtube.com/watch?v=dQw4w9WgXcQ"

installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/invisible-island.net/ncurses/v6.4.0
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/curl.se/ca-certs/v2022.7.19
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/gnu.org/readline/v8.1.0
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/zlib.net/v1.2.13
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/sourceware.org/bzip2/v1.0.8
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/openssl.org/v1.1.1s
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/sourceware.org/libffi/v3.4.4
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/libexpat.github.io/v2.5.0
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/bytereef.org/mpdecimal/v2.5.1
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/tukaani.org/xz/v5.2.7
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/sqlite.org/v3.40.0
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/python.org/v3.11.1
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/gnu.org/bash/v5.1.16
installed: /var/folders/gf/dmvw3mx540l26pc3_2_7b2km0000gn/T/tea-XXXXXX.LJ80owxF/youtube-dl.org/v2021.12.17
[youtube] dQw4w9WgXcQ: Downloading webpage
[youtube] dQw4w9WgXcQ: Downloading player f565d246

sanchitram1 · 2023-02-07T16:20:21Z

@jhheider did not work for me either. I got an identical error as Marc.

@mfts solution did work for me though!

mxcl · 2023-02-13T17:30:32Z

Update to 0.24.2 and reinstall effected packages. Only way is rm -rf ~/.tea/pkg-name then re-run.

mfts · 2023-02-13T19:58:10Z

Confirmed v0.24.2 fixes the issue

* #143 init step for ingesting package details * #143 create reusable workflow for triggering package ingestion/indexing * space * #143 create task after uploading bottles * cleanup --------- Co-authored-by: neil <neil@neils-MacBook-Pro.local>

jonchang mentioned this issue Feb 9, 2023

youtube-dl not executing pkgxdev/pkgx#351

Closed

mxcl closed this as completed in pkgxdev/pkgx@e0fac6d Feb 13, 2023

mxcl added a commit that referenced this issue Mar 15, 2023

getting there (#144)

a5ff99c

mxcl added a commit that referenced this issue Mar 15, 2023

getting there (#144)

adc85d5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`youtube-dl` python encodings error #144

`youtube-dl` python encodings error #144

mfts commented Jan 26, 2023

mxcl commented Jan 26, 2023

mfts commented Jan 26, 2023

mxcl commented Feb 2, 2023

nihaals commented Feb 3, 2023

mxcl commented Feb 3, 2023

mxcl commented Feb 3, 2023

jhheider commented Feb 4, 2023

mxcl commented Feb 5, 2023

mxcl commented Feb 5, 2023

jhheider commented Feb 5, 2023

jhheider commented Feb 6, 2023

mfts commented Feb 6, 2023

jhheider commented Feb 6, 2023 •

edited

Loading

jhheider commented Feb 6, 2023 •

edited

Loading

mfts commented Feb 6, 2023

jhheider commented Feb 6, 2023

mfts commented Feb 7, 2023

sanchitram1 commented Feb 7, 2023

mxcl commented Feb 13, 2023

mfts commented Feb 13, 2023

youtube-dl python encodings error #144

youtube-dl python encodings error #144

Comments

mfts commented Jan 26, 2023

mxcl commented Jan 26, 2023

mfts commented Jan 26, 2023

mxcl commented Feb 2, 2023

nihaals commented Feb 3, 2023

mxcl commented Feb 3, 2023

mxcl commented Feb 3, 2023

jhheider commented Feb 4, 2023

mxcl commented Feb 5, 2023

mxcl commented Feb 5, 2023

jhheider commented Feb 5, 2023

jhheider commented Feb 6, 2023

mfts commented Feb 6, 2023

jhheider commented Feb 6, 2023 • edited Loading

jhheider commented Feb 6, 2023 • edited Loading

mfts commented Feb 6, 2023

jhheider commented Feb 6, 2023

mfts commented Feb 7, 2023

sanchitram1 commented Feb 7, 2023

mxcl commented Feb 13, 2023

mfts commented Feb 13, 2023

`youtube-dl` python encodings error #144

`youtube-dl` python encodings error #144

jhheider commented Feb 6, 2023 •

edited

Loading

jhheider commented Feb 6, 2023 •

edited

Loading