Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MacOS Monterey 12.3 no longer has Python 2.7 installed, finding it very difficult to use sqlite3 #1552

Closed
merrickfox opened this issue Mar 16, 2022 · 38 comments

Comments

@merrickfox
Copy link

merrickfox commented Mar 16, 2022

Output from yarn install

error /myproject/node_modules/sqlite3: Command failed.
Exit code: 1
Command: node-pre-gyp install --fallback-to-build
Arguments:
Directory: /myproject/node_modules/sqlite3
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@0.11.0
node-pre-gyp info using node@16.14.1 | darwin | arm64
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp info check checked for "/myproject/node_modules/sqlite3/lib/binding/napi-v3-darwin-arm64/node_sqlite3.node" (not found)
node-pre-gyp http GET https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-darwin-arm64.tar.gz
node-pre-gyp http 403 https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-darwin-arm64.tar.gz
node-pre-gyp WARN Tried to download(403): https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-darwin-arm64.tar.gz
node-pre-gyp WARN Pre-built binaries not found for sqlite3@5.0.2 and node@16.14.1 (node-v93 ABI, unknown) (falling back to source compile with node-gyp)
node-pre-gyp http 403 status code downloading tarball https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v5.0.2/napi-v3-darwin-arm64.tar.gz
gyp info it worked if it ends with ok
gyp info using node-gyp@3.8.0
gyp info using node@16.14.1 | darwin | arm64
gyp info ok
gyp info it worked if it ends with ok
gyp info using node-gyp@3.8.0
gyp info using node@16.14.1 | darwin | arm64
gyp ERR! configure error
gyp ERR! stack Error: Can't find Python executable "python", you can set the PYTHON env variable.
gyp ERR! stack     at PythonFinder.failNoPython (/myproject/node_modules/node-gyp/lib/configure.js:484:19)
gyp ERR! stack     at PythonFinder.<anonymous> (/myproject/node_modules/node-gyp/lib/configure.js:406:16)
gyp ERR! stack     at F (/myproject/node_modules/node-gyp/node_modules/which/which.js:68:16)
gyp ERR! stack     at E (/myproject/node_modules/node-gyp/node_modules/which/which.js:80:29)
gyp ERR! stack     at /myproject/node_modules/node-gyp/node_modules/which/which.js:89:16
gyp ERR! stack     at /myproject/node_modules/isexe/index.js:42:5
gyp ERR! stack     at /myproject/node_modules/isexe/mode.js:8:5
gyp ERR! stack     at FSReqCallback.oncomplete (node:fs:198:21)
gyp ERR! System Darwin 21.4.0
gyp ERR! command "/Users/merrick/.nvm/versions/node/v16.14.1/bin/node" "/myproject/node_modules/node-gyp/bin/node-gyp.js" "configure" "--fallback-to-build" "--module=/myproject/node_modules/sqlite3/lib/binding/napi-v3-darwin-arm64/node_sqlite3.node" "--module_name=node_sqlite3" "--module_path=/myproject/node_modules/sqlite3/lib/binding/napi-v3-darwin-arm64" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=3" "--node_napi_label=napi-v3"
gyp ERR! cwd /myproject/node_modules/sqlite3
gyp ERR! node -v v16.14.1
gyp ERR! node-gyp -v v3.8.0
gyp ERR! not ok
node-pre-gyp ERR! build error

I've tried pointing it at python3 which does come installed but the code relies on python 2.7 syntax.

I have also been trying to install python 2.7 back on the system with little luck, homebrew seems to have made it hard to install, and pyenv is even having issues on the apple silicon, at least for me, this is with trying workarounds using another x86_64 homebrew install and also using a rosetta terminal.

I know this isn't really an issue with this library, but Monterey 12.3 came out a few days ago at time of writing, I'd expect to see this question asked a lot in the near future.

And I'd also love some help on how I can either 1: Get this working without installing an old version of python or 2: Somehow get an old version of python running on this system.

Cheers!

@Vendll
Copy link

Vendll commented Mar 16, 2022

I had the same problem (M1 Mac, just updated), I've downloaded python 2.7.18 from python.org and duplicated my terminal.app, set it to use rosetta and now it works. It is not optimal but for now the only option I could find.

@merrickfox
Copy link
Author

I had the same problem (M1 Mac, just updated), I've downloaded python 2.7.18 from python.org and duplicated my terminal.app, and set it to use rosetta, now it works. It is not optimal but for now the only option I could find.

Thanks, I didn't actually think of downloading an installer from the official site, almost seems a bit old school, but I tried it and it worked!

@patiwet
Copy link

patiwet commented Mar 17, 2022

I install 2.7.18 through pyenv it work for me.

@DrDavidS
Copy link

Same problem...M1 with MacOS 12.3
😭

@brodock
Copy link

brodock commented Mar 18, 2022

the root of the problem is the bundled sqlite3 tar file sqlite-autoconf-3360000.tar.gz which itself contains calls to a hardcoded python command. If you bypass that file and use an external sqlite3 it should work.

I'm using HomeBrew, and therefore:

brew install sqlite

with this, I can run:

npm install sqlite3 --build-from-source --sqlite=/opt/homebrew/opt/sqlite --save

@bmccorm2
Copy link

Having the same issue. Aside from reinstalling python 2.7 along with the existing installation there is no solution for this now?

@mohd-akram
Copy link
Contributor

This is because of the python invocation in deps/sqlite3.gyp. Luckily, it reads the npm config before falling back to python. To workaround the issue, run this once:

npm config set python python3

@bmccorm2
Copy link

@mohd-akram - thanks that works locally but i tried the same in a dockerfile and it complains about the python executable. Any ideas on how to build a docker image?

@mohd-akram
Copy link
Contributor

mohd-akram commented Mar 20, 2022

That command modifies the config for the current user. It's possible that you're running npm install as a different user. You can try npm config -g set python python3 to set the option globally. See here for more info.

For production it's better to just use the --sqlite flag as mentioned above so you use your distribution's SQLite version which is probably newer.

@bmccorm2
Copy link

Sorry I'm a mac noob here; in my linux box i would just set a symbolic link to python3 and call it python. When i try to do that on the mac it just prompts me to install command line tools which are already installed. Then, I've set the variable globally with npm config set python=python3 and can see it is set via npm config ls -l. However still gives the error.

@rufreakde
Copy link

pm config ls -l

Even if you do a symlink this cant be build with python3. Since it is python2. Here is the alias for your .rc file though (if you installed python3 with homebrew):

# Python 3
alias pip=/usr/local/bin/pip3
alias python=/usr/local/bin/python3

@guidone
Copy link

guidone commented Mar 22, 2022

This worked for me

  1. Install pyenv with brew install pyenv
  2. Install Python 2.7 with pyenv install 2.7.18 and then pyenv global 2.7.18
  3. Then set the default paths typing in the console export PATH="${HOME}/.pyenv/shims:${PATH}"
  4. Finally npm install sqlite3

@olso-nordsec
Copy link

This is because of the python invocation in deps/sqlite3.gyp. Luckily, it reads the npm config before falling back to python. To workaround the issue, run this once:

npm config set python python3

how did this work for you? i'm getting syntax errors

Screenshot 2022-03-24 at 16 51 05

@VsevolodKurochka
Copy link

This worked for me

  1. Install pyenv with brew install pyenv
  2. Install Python 2.7 with pyenv install 2.7.18 and then pyenv global 2.7.18
  3. Then set the default paths export PATH="${HOME}/.pyenv/shims:${PATH}"
  4. Finally npm install sqlite3

Hello! After your steps I also get the error:
stack Error: Can't find Python executable "python2.7.18", you can set the PYTHON env variable.

Maybe I understand you wrong but where I should write the following code: "export PATH="${HOME}/.pyenv/shims:${PATH}"" ? In console?

@guidone
Copy link

guidone commented Mar 25, 2022

Maybe I understand you wrong but where I should write the following code: "export PATH="${HOME}/.pyenv/shims:${PATH}"" ? In console?

Yes in console

@chicunic
Copy link

macos no longer support python@2. can we update the script to python@3?

@wrmack
Copy link

wrmack commented Mar 26, 2022

Where does the requirement for node-gyp@3.8.0 come from? Node-gyp is up to version 9 now. 3.8.0 requires python 2.7. The latest version requires python 3.7 - 3.10. v3.8.0 is not a dependency of node-sqlite3. It depends on version 7.x. in package.json.

I had no problems on Mac M1 using @vscode/sqlite3. Not sure what the difference is with node-sqlite3. Am now trying to use node-sqlite3 in Windows running on qemu virtual machine on a Mac M1 and getting same problems as above. Also get a lot of deprecation warnings.

@wrmack
Copy link

wrmack commented Mar 26, 2022

Discussion here and suggestion to change"

 "sqlite3": "^5.0.2",

to:

"sqlite3": "github:mapbox/node-sqlite3",

At least that upgrades node-gyp to version 7. It requires python v2.7, v3.5, v3.6, v3.7, or v3.8.

@ronaldlangeveld
Copy link
Member

ronaldlangeveld commented Mar 26, 2022

What worked for me on M1 MacOS Monterey 12.3.

Install Python2.7 via Pyenv (Step 1 and 2, as instructed by @guidone above).

Find the directory where Python2 is installed, which -a python2.

In my case it was /Users/username/.pyenv/shims/python2

Then export to your NPM config.

$ export npm_config_python=/path/to/python2

check if that is configured by listing the config:

$ npm config list

It should be listed as follows.

...
python = "/path/to/python2"
...

Run the installation.

@wodka
Copy link

wodka commented Mar 26, 2022

For me it was enough to install node-gyp as a devDependency - now the build is working with python3

@landodesign
Copy link

landodesign commented Mar 28, 2022

What worked for me on M1 MacOS Monterey 12.3.

Install Python2.7 via Pyenv (Step 1 and 2, as instructed by @guidone above).

Find the directory where Python2 is installed, which -a python2.

In my case it was /Users/username/.pyenv/shims/python2

Then export to your NPM config.

$ export npm_config_python=/path/to/python2

check if that is configured by listing the config:

$ npm config list

It should be listed as follows.

...
python = "/path/to/python2"
...

Run the installation.

I tried all this and I'm still getting a bunch of errors like

npm ERR! ERROR:root:code for hash XXX was not found

It looks like it's an issue with node-gyp so I tried to run "node-gyp rebuild" but I'm getting another error:

gyp: binding.gyp not found

Basically I'm looping through errors.

I'm also on a Macbook with MacOS Monterey 12.3. Incidentally, on my Macmini running Monterey 12.1 everything works fine.

@wrmack
Copy link

wrmack commented Mar 28, 2022

@landodesign I have gone back to using:

"@vscode/sqlite3": "latest"

and it works fine. It installs the mapbox sqlite3 but seems to work better. And uses python3. If you try this I suggest make the change in package.json, delete package-lock.json and delete your project's node_modules folder then run npm install to re-install all dependencies.

Not sure if this is relevant, but if I do npm ls node-gyp or npm list --all in my project, node-gyp is not shown as a dependency. If I do npm ls node-gyp -g then node-gyp is shown as a dependency of npm itself and is version 8.4.1. I think this is the only node-gyp on my system.

npm ls node-gyp -g 
/opt/homebrew/lib
└─┬ npm@8.5.2
├─┬ @npmcli/run-script@3.0.0
  │ └── node-gyp@8.4.1 deduped
  └── node-gyp@8.4.1

I am using a MacBook Pro with an M1 chip running Monterey 12.3.

Hope that helps with your trouble-shooting.

@landodesign
Copy link

@landodesign I have gone back to using:

"@vscode/sqlite3": "latest"

and it works fine. It installs the mapbox sqlite3 but seems to work better. And uses python3. If you try this I suggest make the change in package.json, delete package-lock.json and delete your project's node_modules folder then run npm install to re-install all dependencies.

Not sure if this is relevant, but if I do npm ls node-gyp or npm list --all in my project, node-gyp is not shown as a dependency. If I do npm ls node-gyp -g then node-gyp is shown as a dependency of npm itself and is version 8.4.1. I think this is the only node-gyp on my system.

npm ls node-gyp -g 
/opt/homebrew/lib
└─┬ npm@8.5.2
├─┬ @npmcli/run-script@3.0.0
  │ └── node-gyp@8.4.1 deduped
  └── node-gyp@8.4.1

I am using a MacBook Pro with an M1 chip running Monterey 12.3.

Hope that helps with your trouble-shooting.

After trying different solutions, I managed to make it work by downgrading Nodejs version from 17 to 12!

So just to sum it up:

  1. Open iTerm with Rosetta (also, if you're on VSCode like me, follow this )
  2. Follow @guidone 's steps to install python 2
  3. Downgrade Nodejs from 17 to 12

@Pomax
Copy link

Pomax commented Apr 4, 2022

Note that python 2.7 reached EOL over 2 years ago, and has not -nor ever will- received security fixes since. The same goes for Node 12 a few days from now: that reached EOL on the 23rd of this month, and will never receive security updates ever again either. So the real fix here needs to come from mapbox here.

@sandro-pasquali
Copy link

If you need sqlite3 as the package namespace (eg for Sequelize) you can use "sqlite3": "npm:@vscode/sqlite3@^5.0.8".

@Pomax
Copy link

Pomax commented Apr 6, 2022

Also note that the advice above to first explicitly install the current version of node-gyp as devDependency (i.e. npm install -D node-gyp) prior to installing sqlite3 may also yield a working version of this package without needing VScode's fork, as this will cause sqlite3 to get built by a version of node-gyp that explicitly uses python 3.

daniellockyer added a commit that referenced this issue Apr 12, 2022
refs https://github.com/nodejs/node-gyp/blob/f5fa6b86fd2847ca8c1996102f43d44f98780c4a/lib/find-python.js
refs #1552

- Monterey 12.3 removed Python 2, so `node-sqlite3` doesn't build because
  the v3 binary is called `python3` and therefore `python` is missing
- `node-gyp` has logic to find the python binary and stores it in the `PYTHON`
  env variable
- we should be able to switch to using that
@daniellockyer
Copy link
Member

I'm on it 🙂 Fix will be landing in master soon and then it'll be released in due course

daniellockyer added a commit that referenced this issue Apr 12, 2022
refs https://github.com/nodejs/node-gyp/blob/f5fa6b86fd2847ca8c1996102f43d44f98780c4a/lib/find-python.js
refs #1552

- Monterey 12.3 removed Python 2, so `node-sqlite3` doesn't build because
  the v3 binary is called `python3` and therefore `python` is missing
- `node-gyp` has logic to find the python binary and stores it in the `PYTHON`
  env variable
- we should be able to switch to using that
@daniellockyer
Copy link
Member

This should be fixed in sqlite3 v5.0.3 - let me know if you still experience issues 🙂

@Fedeorlandau
Copy link

This worked for me

  1. Install pyenv with brew install pyenv
  2. Install Python 2.7 with pyenv install 2.7.18 and then pyenv global 2.7.18
  3. Then set the default paths typing in the console export PATH="${HOME}/.pyenv/shims:${PATH}"
  4. Finally npm install sqlite3

I had some issues with node-sass and this fixed it. (macos monterey)

Thank you mate

@Pomax
Copy link

Pomax commented May 5, 2022

@Fedeorlandau node-sass has been deprecated for a very long time now, you should be able to switch to the normal sass package without any problems and without needing a dead version of Python.

@oscaredel
Copy link

oscaredel commented May 10, 2022

For those using yarn in a (older) Rails app and a Python installation via pyenv, this finally solved it for me after hours of applying every possible suggested solution:

pyenv install 2.7.18

yarn config set python "/Users/$USER/.pyenv/shims/python2.7""

This will add that config to ~/.yarnrc

Also I made sure to undo as much of the previous given solutions in the other files like: ~/.zsh_profile ~/.zshrc ~/.npmrc

And finally I had to remove the yarn.lock file and update a webpacker version as is suggested here.

Then when running yarn it correctly installed everything.

@christopher-caldwell
Copy link

@olso-nordsec

Did you ever find a solution to the syntax error? Having the same issue

@Pomax
Copy link

Pomax commented Aug 3, 2022

@christopher-caldwell the solution kind of really is "use the current versions of python and node, and first explicitly install the latest version of node-gyp in your project dir if you get errors".

(The oldest still supported versions of python and node here are 3.7, and 14, respectively)

There really is no reason to need python 2.7 for any of this, the only thing that matters is that you're running the latest node-gyp with a python version that it supports (supported versions are always listed in https://github.com/nodejs/node-gyp#on-unix), because it only needs python to run its own compile-specific python code. So the only thing that can go wrong here is that npm picks an ancient/dead version of node-gyp, to which the solution is "make npm not do that, by locally installing the latest node-gyp".

@ryanflowers
Copy link

ryanflowers commented Aug 31, 2022

This should be fixed in sqlite3 v5.0.3 - let me know if you still experience issues 🙂

@daniellockyer I see this same error on 5.0.3. Doesnt seem fixed. Am I missing something?


/Development/developers-experience-frontends rflowers/state * npm install sqlite3@5.0.3

npm ERR! code 1
npm ERR! path /Users/rflowers/Development/developers-experience-frontends/node_modules/sqlite3
npm ERR! command failed
npm ERR! command sh -c node-gyp rebuild
npm ERR! gyp info it worked if it ends with ok
npm ERR! gyp info using node-gyp@8.4.1
npm ERR! gyp info using node@16.0.0 | darwin | x64
npm ERR! gyp info find Python using Python version 3.9.12 found at "/usr/local/opt/python@3.9/bin/python3.9"
npm ERR! gyp info spawn /usr/local/opt/python@3.9/bin/python3.9
npm ERR! gyp info spawn args [
npm ERR! gyp info spawn args   '/Users/rflowers/Development/developers-experience-frontends/node_modules/node-gyp/gyp/gyp_main.py',
npm ERR! gyp info spawn args   'binding.gyp',
npm ERR! gyp info spawn args   '-f',
npm ERR! gyp info spawn args   'make',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/rflowers/Development/developers-experience-frontends/node_modules/sqlite3/build/config.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/rflowers/Development/developers-experience-frontends/node_modules/node-gyp/addon.gypi',
npm ERR! gyp info spawn args   '-I',
npm ERR! gyp info spawn args   '/Users/rflowers/Library/Caches/node-gyp/16.0.0/include/node/common.gypi',
npm ERR! gyp info spawn args   '-Dlibrary=shared_library',
npm ERR! gyp info spawn args   '-Dvisibility=default',
npm ERR! gyp info spawn args   '-Dnode_root_dir=/Users/rflowers/Library/Caches/node-gyp/16.0.0',
npm ERR! gyp info spawn args   '-Dnode_gyp_dir=/Users/rflowers/Development/developers-experience-frontends/node_modules/node-gyp',
npm ERR! gyp info spawn args   '-Dnode_lib_file=/Users/rflowers/Library/Caches/node-gyp/16.0.0/<(target_arch)/node.lib',
npm ERR! gyp info spawn args   '-Dmodule_root_dir=/Users/rflowers/Development/developers-experience-frontends/node_modules/sqlite3',
npm ERR! gyp info spawn args   '-Dnode_engine=v8',
npm ERR! gyp info spawn args   '--depth=.',
npm ERR! gyp info spawn args   '--no-parallel',
npm ERR! gyp info spawn args   '--generator-output',
npm ERR! gyp info spawn args   'build',
npm ERR! gyp info spawn args   '-Goutput_dir=.'
npm ERR! gyp info spawn args ]
npm ERR! gyp: Undefined variable module_name in binding.gyp while trying to load binding.gyp
npm ERR! gyp ERR! configure error 
npm ERR! gyp ERR! stack Error: `gyp` failed with exit code: 1
npm ERR! gyp ERR! stack     at ChildProcess.onCpExit (/Users/rflowers/Development/developers-experience-frontends/node_modules/node-gyp/lib/configure.js:259:16)
npm ERR! gyp ERR! stack     at ChildProcess.emit (node:events:365:28)
npm ERR! gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:290:12)
npm ERR! gyp ERR! System Darwin 21.6.0
npm ERR! gyp ERR! command "/Users/rflowers/.volta/tools/image/node/16.0.0/bin/node" "/Users/rflowers/Development/developers-experience-frontends/node_modules/.bin/node-gyp" "rebuild"
npm ERR! gyp ERR! cwd /Users/rflowers/Development/developers-experience-frontends/node_modules/sqlite3
npm ERR! gyp ERR! node -v v16.0.0
npm ERR! gyp ERR! node-gyp -v v8.4.1
npm ERR! gyp ERR! not ok```

@daniellockyer
Copy link
Member

FYI v5.1.1 now comes with prebuilt Darwin ARM64 binaries 🙂 https://github.com/TryGhost/node-sqlite3/releases/tag/v5.1.1

@vipansh

This comment was marked as resolved.

@VishalBhandare-Apertumo

This comment was marked as outdated.

@Pomax
Copy link

Pomax commented Nov 15, 2022

Yet again: do not install python 2.7, you do not need it and it's a dead, unsupported, security-hole.

  1. update your sqlite3 to the version Daniel just released. Or, if for some otherworldly reason you can't,
  2. first install the most recent node-gyp in your own project dir, then install the version of sqlite3 you're locked into, so that it even though it's going to need to compiling, that happens with a node_gyp version that knows to use python 3

The problem here is not sqlite3 itself, the problem is that you're relying on an ancient version of node-gyp, so you need to tell npm to ignore whatever version a package "wants" to use, and force it to use the modern version of node-gyp instead, by installing that in your project.

@TryGhost TryGhost locked as resolved and limited conversation to collaborators Nov 15, 2022
@TryGhost TryGhost unlocked this conversation Nov 15, 2022
@TryGhost TryGhost locked as resolved and limited conversation to collaborators Nov 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests