treewide: fix cargoSha256/cargoHash

[danieldk]

Motivation for this change

Rust 1.50.0 incorporated a Cargo change (rust-lang/cargo#8937) in
which cargo vendor erroneously changed permissions of vendored
crates. This was fixed in Rust
1.51.0 (rust-lang/cargo#9131). Unfortunately, this means that all
cargoSha256/cargoHashes produced during the Rust 1.50.0 cycle are
potentially broken.

This change updates cargoSha256/cargoHash tree-wide.

Fixes #121994.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[danieldk]

/rebase staging

[github-actions]

Rebased, please reopen the pull request to restart CI

[danieldk]

@jonringer is it ok to merge this into staging now?

[dotlambda]

@danieldk Would you mind publishing the script you did this with? And did you also check cargoDeps?

[danieldk]

@danieldk Would you mind publishing the script you did this with? And did you also check cargoDeps?

I’ll make a gist tomorrow. I have rebuilt all cargoDeps after updating the hashes.

[dotlambda]

I have rebuilt all cargoDeps after updating the hashes.

Sorry, I meant whether you tested packages that use fetchCargoTarball and thus don't set cargoHash.

[jonringer]

I would like to run a review, but that might take a while. If you've built a few packages and feel confident about them, then we can merge.

[danieldk]

I would like to run a review, but that might take a while. If you've built a few packages and feel confident about them, then we can merge.

I have built a few packages (and all cargoDeps), but nix-review requires too many rebuilds for my poor 3700X. I'll fix the merge conflicts and then merge this, since there are three conflicts already. Since this affects so many packages, there will probably be more conflicts within no time :/.

[jonringer]

didn't see any hash mismatches

https://github.com/NixOS/nixpkgs/pull/122016

19 packages marked as broken and skipped:
autotrace gimpPlugins.exposureBlend gimpPlugins.texturize git-dit glimpsePlugins.exposureBlend glimpsePlugins.texturize gnome3.gnome-documents hhvm intecture-cli invoice2data rustracerd scribus sit spring springLobby synfigstudio termplay vimiv xmonad_log_applet

126 packages failed to build:
abiword adapta-gtk-theme almanah arc-theme asciidoc-full asciidoc-full-with-plugins asymptote bat bat-extras.batdiff bat-extras.batgrep bat-extras.batman bat-extras.batwatch bat-extras.prettybat beets beetsExternalPlugins.alternatives beetsExternalPlugins.copyartifacts beetsExternalPlugins.extrafiles btrbk calibre calibre-web calligra cargo-geiger ccache ccacheStdenv ccacheWrapper chrome-gnome-shell cinnamon.cinnamon-screensaver clevis crosvm crypto-org-wallet cutter dblatexFull denemo devede digikam disorderfs dvdstyler dwm-status electrum emulsion ethabi evolution evolution-ews evolutionWithPlugins frescobaldi gimp gimp-with-plugins gimpPlugins.farbfeld gimpPlugins.fourier gimpPlugins.gap gimpPlugins.gimplensfun gimpPlugins.gmic gimpPlugins.lightning gimpPlugins.lqrPlugin gimpPlugins.resynthesizer gimpPlugins.waveletSharpen gir-rs glimpse glimpse-with-plugins glimpsePlugins.farbfeld glimpsePlugins.fourier glimpsePlugins.gap glimpsePlugins.gimplensfun glimpsePlugins.lightning glimpsePlugins.lqrPlugin glimpsePlugins.resynthesizer glimpsePlugins.waveletSharpen gmic-qt-krita gnome3.gnome-applets gnome3.gnome-flashback gnome3.gnome-initial-setup gnome3.gnome-panel gnome3.gnome-session gnome3.gnome-shell gnome3.gnome-terminal gnome3.gnome-tweaks gnome3.pomodoro gnomeExtensions.easyScreenCast gnomeExtensions.gsconnect gnomeExtensions.night-theme-switcher gnvim gnvim-unwrapped gopro gscan2pdf haskellPackages.neuron hydra-unstable influxdb2 rep krita ledger-live-desktop lilypond lilypond-with-fonts luksmeta neuron-notes noto-fonts-emoji openshot-qt pantheon.elementary-session-settings perl530Packages.Gtk3ImageView perl532Packages.Gtk3ImageView phosh photoflow playonlinux py-spy python38Packages.diagrams python38Packages.johnnycanencrypt sequoia python38Packages.tumpa python39Packages.diagrams python39Packages.johnnycanencrypt python39Packages.sequoia python39Packages.tumpa rdedup rq rustracer shutter siril skribilo sniffglue solfege sourcetrail tang twitter-color-emoji udiskie vcs wio xprite-editor

467 packages built:
alacritty alass amp angle-grinder arx-libertatis as-tree asc-key-to-qr-code-gif asciinema-scenario asuka async authenticator awesome b3sum bandwhich betterlockscreen bibata-cursors bibata-cursors-translucent bibata-extra-cursors bingrep bitwarden_rs bitwarden_rs-mysql bitwarden_rs-postgresql blflash blockhash bombono boringtun bottom brasero brasero-original break-time bukubrow cached-nix-shell capitaine-cursors cargo-about cargo-asm cargo-binutils cargo-bloat cargo-c cargo-cache cargo-criterion cargo-cross cargo-deps cargo-edit cargo-embed cargo-expand cargo-feature cargo-flamegraph cargo-flash cargo-fund cargo-fuzz cargo-generate cargo-graph cargo-inspect cargo-insta cargo-kcov cargo-license cargo-play cargo-readme cargo-release cargo-sweep cargo-sync-readme cargo-tarpaulin cargo-udeps cargo-update cargo-valgrind cargo-web cargo-whatfeatures cargo-wipe cargo-xbuild castor cataract cataract-unstable catfs cd-hit cfdyndns chafa chars chit choose cicero-tui cinnamon.iso-flags-png-320x420 cinnamon.iso-flags-svg click clipcat clog-cli cntr code-minimap code-server coloursum convco crate2nix cuneiform czkawka delta diesel-cli diffr dijo diskonaut diskus dmtx-utils docbook2odf dogdns doh-proxy-rust dot-http dotenv-linter drill du-dust dvd-slideshow dvdauthor dvdbackup each effitask eidolon elfx86exts emojione emplace enigma espanso eva evscript fac-build far2l fbida fd fend fgallery fido2luks fim finalfrontier finalfusion-utils findomain firmware-manager fishnet fitnesstrax flavours fmbt fontfor fontpreview foxtrotgps fped freedroid freenukum fst g933-utils geeqie genpass gifski git-absorb git-backup git-codeowners git-gone git-ignore git-series git-subset git-trim git-vanity-hash git-workspace gitoxide gnirehtet gnome-keysign gnome3.gdm goattracker goattracker-stereo gping greetd.dlm greetd.greetd greetd.tuigreet greetd.wlgreet grex habitat hacksaw heatseeker hello-wayland hexdino hexyl httplz hydra-cli hyperfine i3-auto-layout i3-ratiosplit i3lock-fancy i3lock-pixeled i3nator i3status-rust ikiwiki imag imagemagick imagemagick6 imagemagick6Big imagemagickBig imgproxy imv inkscape inkscape-with-extensions intecture-agent intecture-auth intermodal ion iso-flags itm-tools ja2-stracciatella joplin journaldriver jwt-cli kak-lsp parinfer-rust kibi kitty kmon kondo kubernix kubie ldgallery libheif libpst libsForQt5.libopenshot libsForQt512.libopenshot libsForQt514.libopenshot libwmf licensor loc loop lscolors lsd mail-notification martin mastodon mate.caja-extensions mate.caja-with-extensions mate.mate-utils matrix-synapse-tools.rust-synapse-compress-state mcfly mdbook mdcat mdctags mdsh mediainfo-gui megapixels meilisearch mgba microserver mmtc mojave-gtk-theme monolith mozwire mq-cli multilockscreen muso natron ncgopher netbeans netease-music-tui newsboat nip2 nix-direnv nix-doc nix-index nix-query-tree-viewer nix-simple-deploy nix-template nix-update nixFlakes nixdoc nixos-icons nixpkgs-fmt nixpkgs-review noaa-apt noteshrink numix-cursor-theme numix-solarized-gtk-theme numworks-epsilon onefetch openspades optar ox oxipng pactorio page panopticon paperless paperwork pastel pax-rs pazi pcb pdf-redact-tools pdf2odt pdfsandwich peek peep perl530Packages.LaTeXML perl530Packages.PerlMagick perl532Packages.LaTeXML perl532Packages.PerlMagick pfstools phash php73Extensions.imagick php74Extensions.imagick php80Extensions.imagick pipr plata-theme pop-gtk-theme portmod powerline-rs pqiv pqrs probe-run procs prometheus-unbound-exporter protonvpn-gui pstoedit psw ptags pueue python38Packages.SQLAlchemy-ImageAttach python38Packages.Wand python38Packages.paperwork-backend python38Packages.paperwork-shell python38Packages.pyocr python38Packages.pyvips pywal python39Packages.SQLAlchemy-ImageAttach python39Packages.Wand python39Packages.paperwork-backend python39Packages.paperwork-shell python39Packages.pyocr python39Packages.pyvips python39Packages.pywal railcar rargs rav1e rebazel recoverjpeg reddsaver resvg ripasso-cursive ripcord ripgrep ripgrep-all rizin rnix-hashes rnix-lsp routinator rs-git-fsmonitor rsclock rshijack rss-glx rtss rubyPackages.rmagick rubyPackages.ruby-vips rubyPackages_2_6.rmagick rubyPackages_2_6.ruby-vips rubyPackages_3_0.rmagick rubyPackages_3_0.ruby-vips ruffle ruplacer rust-bindgen rustscan rx sandboxfs sccache sd shadowenv shell-hist shotgun silicon skim sn0int so soulseekqt sound-juicer sozu spacevim split2flac spotify-tui spotifyd stm32cubemx sub-batch supertag surface-control svgbob svgcleaner svlint svls swapview swaylock-fancy swaywsr synapse-bt system-syzygy system76-firmware tab-rs tagref taizen tango-icon-theme tarssh taskwarrior-tui tdns-cli tealdeer tectonic tensorman terminal-typeracer tex-match texlab the-way tickrs timelapse-deflicker tiny tiv todiff tokei topgrade tox-node tre-command tree-sitter trunk ttygif tunnelto tuxpaint tydra udpt unpfs unused uq urn-timer utsushi uwc uwuify vapoursynth vapoursynth-editor vapoursynth-mvtools variety vimPlugins.minimap-vim vimPlugins.skim vimPlugins.skim-vim vimPlugins.vim-markdown-composer vips viu vivid void vscode-extensions.vadimcn.vscode-lldb wagyu wasm-bindgen-cli wasm-pack wasmer wasmtime webmetro wg-bond whitebox-tools windowmaker wishbone-tool wmfocus wpa_supplicant_gui wpgtk wv xastir xfce.xfce4icontheme xfce.xfce4-windowck-plugin xh xidlehook xscast xsv xxv yubikey-personalization-gui zbar zenith zktree zz

[danieldk]

@dotlambda: https://gist.github.com/danieldk/d8c7791d0074ce2227420e3754bf4a05

(unfortunately still to hacky + requires some manual work)

[dotlambda]

So I take it that you didn't check expressions using cargoDeps = rustPlatform.fetchCargoTarball { hash/sha256 = ... }.

[danieldk]

So I take it that you didn't check expressions using cargoDeps = rustPlatform.fetchCargoTarball { hash/sha256 = ... }.

No, that's still a todo, didn't have time to check these as well.

[danieldk]

Also, I fear that a lot of hashes will break again over time. Maybe I am wrong, but I find it hard to believe that we had ~320 incorrect hashes as a result of the relatively short window where Cargo modified permissions while vendoring. Some of the derivations that I maintain weren't updated during that window and still had cargoSha256s that are now incorrect. There may have been other drifts that we didn't catch.

Maybe some day it will be acceptable to include Cargo.lock files in nixpkgs, so that we can have proper fixed-output derivations through #122158 and do not need cargoSha256/cargoHash anymore.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/status-of-lang2nix-approaches/14477/20