Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an immutable tarball link to archive download headers for Nix #31139

Merged
merged 1 commit into from
May 28, 2024

Conversation

Mic92
Copy link
Contributor

@Mic92 Mic92 commented May 28, 2024

This allows nix flake metadata and nix in general to lock a branch tarball link in a manner that causes it to fetch the correct commit even if the branch is updated with a newer version.

For further context, Nix flakes are a feature that, among other things, allows for "inputs" that are "github:someuser/somerepo", "https://some-tarball-service/some-tarball.tar.gz", "sourcehut:~meow/nya" or similar. This feature allows our users to fetch tarballs of git-based inputs to their builds rather than using git to fetch them, saving significant download time.

There is presently no gitea or forgejo specific fetcher in Nix, and we don't particularly wish to have one. Ideally (as a developer on a Nix implementation myself) we could just use the generic tarball fetcher and not add specific forgejo support, but to do so, we need additional metadata to know which commit a given branch tarball represents, which is the purpose of the Link header added here.

The result of this patch is that a Nix user can specify inputs.something.url = "https://forgejo-host/some/project/archive/main.tar.gz" in flake.nix and get a link to some concrete tarball for the actual commit in the lock file, then when they run nix flake update in the future, they will get the latest commit in that branch.

Example of it working locally:

» nix flake metadata --refresh 'http://localhost:3000/api/v1/repos/jade/cats/archive/main.tar.gz?dir=configs/nix'
Resolved URL: http://localhost:3000/api/v1/repos/jade/cats/archive/main.tar.gz?dir=configs/nix
Locked URL: http://localhost:3000/api/v1/repos/jade/cats/archive/804ede182b6b66469b23ea4d21eece52766b7a06.tar.gz?dir=configs
/nix&narHash=sha256-yP7KkDVfuixZzs0fsqhSETXFC0y8m6nmPLw2GrAMxKQ%3D
Description: Computers with the nixos
Path: /nix/store/s856c6yqghyan4v0zy6jj19ksv0q22nx-source
Revision: 804ede182b6b66469b23ea4d21eece52766b7a06
Last modified: 2024-05-02 00:48:32

For details on the header value, see:
https://github.com/nixos/nix/blob/56763ff918eb308db23080e560ed2ea3e00c80a7/doc/manual/src/protocols/tarball-fetcher.md

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label May 28, 2024
@pull-request-size pull-request-size bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label May 28, 2024
@github-actions github-actions bot added modifies/api This PR adds API routes or modifies them modifies/go Pull requests that update Go code labels May 28, 2024
@Mic92
Copy link
Contributor Author

Mic92 commented May 28, 2024

This is a backport of https://codeberg.org/forgejo/forgejo/pulls/3615

Copy link
Contributor

@wxiaoguang wxiaoguang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR, it overall looks good to me. Some non-blocking suggestions (simplify the code and make the check more strict)

routers/api/v1/repo/file.go Outdated Show resolved Hide resolved
routers/web/repo/repo.go Outdated Show resolved Hide resolved
tests/integration/api_repo_archive_test.go Outdated Show resolved Hide resolved
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels May 28, 2024
@Mic92
Copy link
Contributor Author

Mic92 commented May 28, 2024

cc @6543

This allows `nix flake metadata` and nix in general to lock a *branch*
tarball link in a manner that causes it to fetch the correct commit even
if the branch is updated with a newer version.

For further context, Nix flakes are a feature that, among other things,
allows for "inputs" that are "github:someuser/somerepo",
"https://some-tarball-service/some-tarball.tar.gz",
"sourcehut:~meow/nya" or similar. This feature allows our users to fetch
tarballs of git-based inputs to their builds rather than using git to
fetch them, saving significant download time.

There is presently no gitea or forgejo specific fetcher in Nix, and we
don't particularly wish to have one. Ideally (as a developer on a Nix
implementation myself) we could just use the generic tarball fetcher and
not add specific forgejo support, but to do so, we need additional
metadata to know which commit a given *branch* tarball represents, which
is the purpose of the Link header added here.

The result of this patch is that a Nix user can specify `inputs.something.url =
"https://forgejo-host/some/project/archive/main.tar.gz"` in flake.nix
and get a link to some concrete tarball for the actual commit in the
lock file, then when they run `nix flake update` in the future, they
will get the latest commit in that branch.

Example of it working locally:

 » nix flake metadata --refresh 'http://localhost:3000/api/v1/repos/jade/cats/archive/main.tar.gz?dir=configs/nix'
Resolved URL:  http://localhost:3000/api/v1/repos/jade/cats/archive/main.tar.gz?dir=configs/nix
Locked URL:    http://localhost:3000/api/v1/repos/jade/cats/archive/804ede182b6b66469b23ea4d21eece52766b7a06.tar.gz?dir=configs
/nix&narHash=sha256-yP7KkDVfuixZzs0fsqhSETXFC0y8m6nmPLw2GrAMxKQ%3D
Description:   Computers with the nixos
Path:          /nix/store/s856c6yqghyan4v0zy6jj19ksv0q22nx-source
Revision:      804ede182b6b66469b23ea4d21eece52766b7a06
Last modified: 2024-05-02 00:48:32

For details on the header value, see:
https://github.com/nixos/nix/blob/56763ff918eb308db23080e560ed2ea3e00c80a7/doc/manual/src/protocols/tarball-fetcher.md

Update routers/api/v1/repo/file.go

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

Update routers/web/repo/repo.go

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

Update tests/integration/api_repo_archive_test.go

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
@Mic92 Mic92 marked this pull request as ready for review May 28, 2024 15:05
@Mic92
Copy link
Contributor Author

Mic92 commented May 28, 2024

Tested here: https://git.clan.lol/clan/clan-infra/pulls/185

@wxiaoguang wxiaoguang enabled auto-merge (squash) May 28, 2024 15:18
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels May 28, 2024
@wxiaoguang wxiaoguang merged commit 4fe4156 into go-gitea:main May 28, 2024
26 checks passed
@GiteaBot GiteaBot added this to the 1.23.0 milestone May 28, 2024
@wxiaoguang wxiaoguang added the backport/v1.22 This PR should be backported to Gitea 1.22 label May 28, 2024
@Mic92 Mic92 deleted the nix-tarball-link branch May 28, 2024 15:33
Copy link
Member

@6543 6543 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm ;)

GiteaBot pushed a commit to GiteaBot/gitea that referenced this pull request May 29, 2024
…-gitea#31139)

This allows `nix flake metadata` and nix in general to lock a *branch*
tarball link in a manner that causes it to fetch the correct commit even
if the branch is updated with a newer version.

Co-authored-by: Jade Lovelace <software@lfcode.ca>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
@GiteaBot GiteaBot added the backport/done All backports for this PR have been created label May 29, 2024
lunny pushed a commit that referenced this pull request May 29, 2024
…1139) (#31145)

Backport #31139 by @Mic92

This allows `nix flake metadata` and nix in general to lock a *branch*
tarball link in a manner that causes it to fetch the correct commit even
if the branch is updated with a newer version.

For further context, Nix flakes are a feature that, among other things,
allows for "inputs" that are "github:someuser/somerepo",
"https://some-tarball-service/some-tarball.tar.gz",
"sourcehut:~meow/nya" or similar. This feature allows our users to fetch
tarballs of git-based inputs to their builds rather than using git to
fetch them, saving significant download time.

There is presently no gitea or forgejo specific fetcher in Nix, and we
don't particularly wish to have one. Ideally (as a developer on a Nix
implementation myself) we could just use the generic tarball fetcher and
not add specific forgejo support, but to do so, we need additional
metadata to know which commit a given *branch* tarball represents, which
is the purpose of the Link header added here.

The result of this patch is that a Nix user can specify
`inputs.something.url =
"https://forgejo-host/some/project/archive/main.tar.gz"` in flake.nix
and get a link to some concrete tarball for the actual commit in the
lock file, then when they run `nix flake update` in the future, they
will get the latest commit in that branch.

Example of it working locally:

» nix flake metadata --refresh
'http://localhost:3000/api/v1/repos/jade/cats/archive/main.tar.gz?dir=configs/nix'
Resolved URL:
http://localhost:3000/api/v1/repos/jade/cats/archive/main.tar.gz?dir=configs/nix
Locked URL:
http://localhost:3000/api/v1/repos/jade/cats/archive/804ede182b6b66469b23ea4d21eece52766b7a06.tar.gz?dir=configs
/nix&narHash=sha256-yP7KkDVfuixZzs0fsqhSETXFC0y8m6nmPLw2GrAMxKQ%3D
Description:   Computers with the nixos
Path:          /nix/store/s856c6yqghyan4v0zy6jj19ksv0q22nx-source
Revision:      804ede182b6b66469b23ea4d21eece52766b7a06
Last modified: 2024-05-02 00:48:32

For details on the header value, see:

https://github.com/nixos/nix/blob/56763ff918eb308db23080e560ed2ea3e00c80a7/doc/manual/src/protocols/tarball-fetcher.md

Co-authored-by: Jörg Thalheim <Mic92@users.noreply.github.com>
Co-authored-by: Jade Lovelace <software@lfcode.ca>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
zjjhot added a commit to zjjhot/gitea that referenced this pull request May 30, 2024
* giteaofficial/main: (30 commits)
  Azure blob storage support (go-gitea#30995)
  Use repo as of renderctx's member rather than a repoPath on metas (go-gitea#29222)
  Ignore FindRecentlyPushedNewBranches err (go-gitea#31164)
  [skip ci] Updated translations via Crowdin
  Fix markup preview (go-gitea#31158)
  Swap word order in Comment and Close (go-gitea#31148)
  Fix push multiple branches error with tests (go-gitea#31151)
  Use vertical layout for multiple code expander buttons (go-gitea#31122)
  Remove duplicate `ProxyPreserveHost` in Apache httpd doc (go-gitea#31143)
  [skip ci] Updated translations via Crowdin
  Add an immutable tarball link to archive download headers for Nix (go-gitea#31139)
  Improve mobile review ui (go-gitea#31091)
  Add topics for repository API (go-gitea#31127)
  Add missed return after `ctx.ServerError` (go-gitea#31130)
  Fix API repository object format missed (go-gitea#31118)
  Fix DashboardRepoList margin (go-gitea#31121)
  Update JS dependencies (go-gitea#31120)
  [skip ci] Updated translations via Crowdin
  Prevent simultaneous editing of comments and issues (go-gitea#31053)
  Update demo site location from try.gitea.io -> demo.gitea.com (go-gitea#31054)
  ...
@go-gitea go-gitea locked as resolved and limited conversation to collaborators Aug 27, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/done All backports for this PR have been created backport/v1.22 This PR should be backported to Gitea 1.22 lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. modifies/api This PR adds API routes or modifies them modifies/go Pull requests that update Go code size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants