Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python312Packages.blis: 0.7.11 -> 1.0.0 #333221

Merged
merged 4 commits into from
Sep 5, 2024

Conversation

dotlambda
Copy link
Member

Description of changes

Diff: explosion/cython-blis@v0.7.11...release-v1.0.0

Changelog: https://github.com/explosion/cython-blis/releases/tag/release-v1.0.0

Version 1.0 depends on numpy 2.0. C.f. #327437.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@doronbehar
Copy link
Contributor

Changes look good. Why is this a draft?

@dotlambda
Copy link
Member Author

Changes look good. Why is this a draft?

It breaks thinc and we can't depend on numpy_2 inside python3Packages.

@doronbehar
Copy link
Contributor

Changes look good. Why is this a draft?

It breaks thinc and we can't depend on numpy_2 inside python3Packages.

And thinc can't depend on numpy_2?

@SuperSandro2000
Copy link
Member

It breaks thinc

Then we need an overlay there

we can't depend on numpy_2 inside python3Packages.

I think we can since we catch conflicts now. :) Please give it a shoot.

@dotlambda
Copy link
Member Author

It breaks thinc

Then we need an overlay there

Thinc is a Python package, that won't work.

@dotlambda
Copy link
Member Author

The package now works with both numpy and numpy_2.

Comment on lines -55 to -56
# Do not update to BLIS 0.9.x until the following issue is resolved:
# https://github.com/explosion/thinc/issues/771#issuecomment-1255825935
Copy link
Member Author

@dotlambda dotlambda Aug 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidak Can thinc's blis pin be removed?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually upstream removed it in explosion/thinc@efda3b4 so this should be fine.

Copy link
Member

@SuperSandro2000 SuperSandro2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, can be merged of war resolve the question above this

Comment on lines +50 to +54
numpy
];

propagatedBuildInputs = [ numpy ];
dependencies = [ numpy ];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't that a bit weird to put it in both places?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what upstream specifies. IIRC numpy is imported in setup.py.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what upstream specifies. IIRC numpy is imported in setup.py.

Hmm if so, the only good reason to still put Numpy also in the native inputs is for cross compilation, which will not work on NixOS with the way upstream uses the native Numpy in their setup.py:

https://github.com/explosion/cython-blis/blob/b33c2ed9cf7ed51d999a926c4950dd8dec4d0286/setup.py#L86C39-L86C58

I think that for cross compilation you'd have to patch that line, and for non-cross compilation, you'd get the host's Numpy available anyway, so there is no need for numpy in the build-system list IMO.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We plan to separate build dependencies and runtime dependencies in the future. #272178
So, we need to add numpy to both build-system and dependencies.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We plan to separate build dependencies and runtime dependencies in the future. #272178 So, we need to add numpy to both build-system and dependencies.

I'm not sure I agree with everything said there. Even if I had agreed, #272179 should have been finished before you'd apply that policy upon other packages.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order is reversed.
Without proper separation, we cannot start #272179.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you'd get the host's Numpy available anyway, so there is no need for numpy in the build-system list IMO.

There is because in the future dependencies might not be available at build time in order to reduce the number of rebuilds cause by updates, overrides, etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you'd get the host's Numpy available anyway, so there is no need for numpy in the build-system list IMO.

There is because in the future dependencies might not be available at build time in order to reduce the number of rebuilds cause by updates, overrides, etc.

I support this motivation and this goal, but in this case, due to the goal of upstream's import numpy in setup.py, it is completely incorrect to add numpy to build-system, but rather you should patch (downstream or upstream) the file to support specifying manually the host machine's Numpy headers.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that cross doesn't work as is. But at least this continues to work if dependencies are unavailable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that cross doesn't work as is. But at least this continues to work if dependencies are unavailable.

OK I understand now. Still it would have been nice if you had put a TODO comment or something. Also the intentions of #272178 should be communicated in the documentation IMO, even though not everything is implemented yet.

pkgs/development/python-modules/blis/default.nix Outdated Show resolved Hide resolved
@dotlambda dotlambda marked this pull request as ready for review September 2, 2024 00:01
@dotlambda
Copy link
Member Author

Result of nixpkgs-review pr 333221 run on x86_64-linux 1

2 packages failed to build:
  • python311Packages.private-gpt
  • python311Packages.private-gpt.dist
88 packages built:
  • aider-chat
  • aider-chat.dist
  • python311Packages.blis
  • python311Packages.blis.dist
  • python311Packages.fastai
  • python311Packages.fastai.dist
  • python311Packages.llama-index
  • python311Packages.llama-index-agent-openai
  • python311Packages.llama-index-agent-openai.dist
  • python311Packages.llama-index-cli
  • python311Packages.llama-index-cli.dist
  • python311Packages.llama-index-core
  • python311Packages.llama-index-core.dist
  • python311Packages.llama-index-embeddings-gemini
  • python311Packages.llama-index-embeddings-gemini.dist
  • python311Packages.llama-index-embeddings-google
  • python311Packages.llama-index-embeddings-google.dist
  • python311Packages.llama-index-embeddings-huggingface
  • python311Packages.llama-index-embeddings-huggingface.dist
  • python311Packages.llama-index-embeddings-ollama
  • python311Packages.llama-index-embeddings-ollama.dist
  • python311Packages.llama-index-embeddings-openai
  • python311Packages.llama-index-embeddings-openai.dist
  • python311Packages.llama-index-graph-stores-nebula
  • python311Packages.llama-index-graph-stores-nebula.dist
  • python311Packages.llama-index-graph-stores-neo4j
  • python311Packages.llama-index-graph-stores-neo4j.dist
  • python311Packages.llama-index-graph-stores-neptune
  • python311Packages.llama-index-graph-stores-neptune.dist
  • python311Packages.llama-index-indices-managed-llama-cloud
  • python311Packages.llama-index-indices-managed-llama-cloud.dist
  • python311Packages.llama-index-legacy
  • python311Packages.llama-index-legacy.dist
  • python311Packages.llama-index-llms-ollama
  • python311Packages.llama-index-llms-ollama.dist
  • python311Packages.llama-index-llms-openai
  • python311Packages.llama-index-llms-openai-like
  • python311Packages.llama-index-llms-openai-like.dist
  • python311Packages.llama-index-llms-openai.dist
  • python311Packages.llama-index-multi-modal-llms-openai
  • python311Packages.llama-index-multi-modal-llms-openai.dist
  • python311Packages.llama-index-program-openai
  • python311Packages.llama-index-program-openai.dist
  • python311Packages.llama-index-question-gen-openai
  • python311Packages.llama-index-question-gen-openai.dist
  • python311Packages.llama-index-readers-database
  • python311Packages.llama-index-readers-database.dist
  • python311Packages.llama-index-readers-file
  • python311Packages.llama-index-readers-file.dist
  • python311Packages.llama-index-readers-json
  • python311Packages.llama-index-readers-json.dist
  • python311Packages.llama-index-readers-llama-parse
  • python311Packages.llama-index-readers-llama-parse.dist
  • python311Packages.llama-index-readers-s3
  • python311Packages.llama-index-readers-s3.dist
  • python311Packages.llama-index-readers-twitter
  • python311Packages.llama-index-readers-twitter.dist
  • python311Packages.llama-index-readers-txtai
  • python311Packages.llama-index-readers-txtai.dist
  • python311Packages.llama-index-readers-weather
  • python311Packages.llama-index-readers-weather.dist
  • python311Packages.llama-index-vector-stores-chroma
  • python311Packages.llama-index-vector-stores-chroma.dist
  • python311Packages.llama-index-vector-stores-google
  • python311Packages.llama-index-vector-stores-google.dist
  • python311Packages.llama-index-vector-stores-postgres
  • python311Packages.llama-index-vector-stores-postgres.dist
  • python311Packages.llama-index-vector-stores-qdrant
  • python311Packages.llama-index-vector-stores-qdrant.dist
  • python311Packages.llama-index.dist
  • python311Packages.llama-parse
  • python311Packages.llama-parse.dist
  • python311Packages.spacy
  • python311Packages.spacy-lookups-data
  • python311Packages.spacy-lookups-data.dist
  • python311Packages.spacy-transformers
  • python311Packages.spacy-transformers.dist
  • python311Packages.spacy.dist
  • python311Packages.textacy
  • python311Packages.textacy.dist
  • python311Packages.textnets
  • python311Packages.textnets.dist
  • python311Packages.thinc
  • python311Packages.thinc.dist
  • python312Packages.blis
  • python312Packages.blis.dist
  • python312Packages.thinc
  • python312Packages.thinc.dist

@dotlambda dotlambda merged commit 70e145e into NixOS:master Sep 5, 2024
25 of 26 checks passed
@dotlambda dotlambda deleted the python3Packages.blis branch September 5, 2024 22:41
@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/several-comments-about-priorities-and-new-policies-in-the-python-ecosystem/51790/1

@nixos-discourse
Copy link

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/several-comments-about-priorities-and-new-policies-in-the-python-ecosystem/51790/9

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants