Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect submodule update=none strategy in .gitmodules #10717

Merged
merged 1 commit into from
Jun 7, 2022

Conversation

jsitnicki
Copy link
Contributor

@jsitnicki jsitnicki commented May 31, 2022

Git lets users define the default update/checkout strategy for a submodule
by setting the submodule.<name>.update key in .gitmodules file.

If the update strategy is none, the submodule will be skipped during
update. It will not be fetched and checked out:

  1. foo is a big git repo
/tmp $ git init foo
Initialized empty Git repository in /tmp/foo/.git/
/tmp $ dd if=/dev/zero of=foo/big bs=1000M count=1
1+0 records in
1+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.482087 s, 2.2 GB/s
/tmp $ git -C foo add big
/tmp $ git -C foo commit -m 'I am big'
[main (root-commit) 84fb533] I am big
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 big
  1. bar is a repo with a big submodule with update=none
/tmp $ git init bar
Initialized empty Git repository in /tmp/bar/.git/
/tmp $ git -C bar submodule add file:///tmp/foo foo
Cloning into '/tmp/bar/foo'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), 995.50 KiB | 338.00 KiB/s, done.
/tmp $ git -C bar config --file .gitmodules submodule.foo.update none
/tmp $ cat bar/.gitmodules
[submodule "foo"]
        path = foo
        url = file:///tmp/foo
        update = none
/tmp $ git -C bar commit --all -m 'I have a big submodule with update=none'
[main (root-commit) 6c355ea] I have a big submodule not updated by default
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 foo
  1. baz is a clone of bar, notice foo submodule gets skipped
/tmp $ git clone --recurse-submodules file:///tmp/bar baz
Cloning into 'baz'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
Submodule 'foo' (file:///tmp/foo) registered for path 'foo'
Skipping submodule 'foo'
/tmp $ git -C baz submodule update --init
Skipping submodule 'foo'
/tmp $

Cargo, on the other hand, ignores the submodule update strategy set in
.gitmodules properties when updating dependencies. Such behavior can
be considered against the wish of the crate publisher.

  1. bar is now a lib with a big submodule with update disabled
/tmp $ cargo init --lib bar
     Created library package
/tmp $ git -C bar add .
/tmp $ git -C bar commit -m 'I am a lib with a big submodule but update=none'
[main eb07cf7] I am a lib with a big submodule but update=none
 3 files changed, 18 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Cargo.toml
 create mode 100644 src/lib.rs
/tmp $
  1. qux depends on bar, notice bar's submodules are fetched
/tmp $ cargo init qux && cd qux
     Created binary (application) package
/tmp/qux $ echo -e '[dependencies.bar]\ngit = "file:///tmp/bar"' >> Cargo.toml
/tmp/qux $ time cargo update
    Updating git repository `file:///tmp/bar`
    Updating git submodule `file:///tmp/foo`

real    0m22.182s
user    0m20.402s
sys     0m1.714s
/tmp/qux $

Fix it by checking if a Git repository submodule should be updated when
cargo processes dependencies.

  1. With the change applied, submodules with update=none are skipped
/tmp/qux $ cargo cache -a > /dev/null
/tmp/qux $ time ~/src/cargo/target/debug/cargo update
    Updating git repository `file:///tmp/bar`
    Skipping git submodule `file:///tmp/foo`

real    0m0.029s
user    0m0.021s
sys     0m0.008s
/tmp/qux $

Fixes #4247.

@rust-highfive
Copy link

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @ehuss (or someone else) soon.

Please see the contribution instructions for more information.

@rust-highfive rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 31, 2022
@@ -370,6 +370,14 @@ impl<'a> GitCheckout<'a> {
anyhow::format_err!("non-utf8 url for submodule {:?}?", child.path())
})?;

// Skip the submodule if the config says not to update it.
Copy link
Member

@Muscraft Muscraft May 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be a good idea to include a test for this change. That way future changes wouldn't revert this one. I'd look at this test or this test for inspiration

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the pointers @Muscraft. Let me do that.

@weihanglo weihanglo added S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 6, 2022
@jsitnicki jsitnicki force-pushed the respect-git-submod-update-none branch 3 times, most recently from 7de931d to a05107a Compare June 6, 2022 19:03
@jsitnicki
Copy link
Contributor Author

v2:

  • added test for skipping a submodule of a dependency, when its update strategy is none
  • fix up sign off in the commit message

PTAL

Git lets users define the default update/checkout strategy for a submodule
by setting the `submodule.<name>.update` key in `.gitmodules` file.

If the update strategy is `none`, the submodule will be skipped during
update. It will not be fetched and checked out:

1. *foo* is a big git repo

```
/tmp $ git init foo
Initialized empty Git repository in /tmp/foo/.git/
/tmp $ dd if=/dev/zero of=foo/big bs=1000M count=1
1+0 records in
1+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 0.482087 s, 2.2 GB/s
/tmp $ git -C foo add big
/tmp $ git -C foo commit -m 'I am big'
[main (root-commit) 84fb533] I am big
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 big
```

2. *bar* is a repo with a big submodule with `update=none`

```
/tmp $ git init bar
Initialized empty Git repository in /tmp/bar/.git/
/tmp $ git -C bar submodule add file:///tmp/foo foo
Cloning into '/tmp/bar/foo'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 1 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), 995.50 KiB | 338.00 KiB/s, done.
/tmp $ git -C bar config --file .gitmodules submodule.foo.update none
/tmp $ cat bar/.gitmodules
[submodule "foo"]
        path = foo
        url = file:///tmp/foo
        update = none
/tmp $ git -C bar commit --all -m 'I have a big submodule with update=none'
[main (root-commit) 6c355ea] I have a big submodule not updated by default
 2 files changed, 4 insertions(+)
 create mode 100644 .gitmodules
 create mode 160000 foo
```

3. *baz* is a clone of *bar*, notice *foo* submodule gets skipped

```
/tmp $ git clone --recurse-submodules file:///tmp/bar baz
Cloning into 'baz'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (3/3), done.
Submodule 'foo' (file:///tmp/foo) registered for path 'foo'
Skipping submodule 'foo'
/tmp $ git -C baz submodule update --init
Skipping submodule 'foo'
/tmp $
```

Cargo, on the other hand, ignores the submodule update strategy set in
`.gitmodules` properties when updating dependencies. Such behavior can
be considered against the wish of the crate publisher.

4. *bar* is now a lib with a big submodule with update disabled

```
/tmp $ cargo init --lib bar
     Created library package
/tmp $ git -C bar add .
/tmp $ git -C bar commit -m 'I am a lib with a big submodule but update=none'
[main eb07cf7] I am a lib with a big submodule but update=none
 3 files changed, 18 insertions(+)
 create mode 100644 .gitignore
 create mode 100644 Cargo.toml
 create mode 100644 src/lib.rs
/tmp $
```

5. *qux* depends on *bar*, notice *bar*'s submodules are fetched

```
/tmp $ cargo init qux && cd qux
     Created binary (application) package
/tmp/qux $ echo -e '[dependencies.bar]\ngit = "file:///tmp/bar"' >> Cargo.toml
/tmp/qux $ time cargo update
    Updating git repository `file:///tmp/bar`
    Updating git submodule `file:///tmp/foo`

real    0m22.182s
user    0m20.402s
sys     0m1.714s
/tmp/qux $
```

Fix it by checking if a Git repository submodule should be updated when
cargo processes dependencies.

6. With the change applied, submodules with `update=none` are skipped

```
/tmp/qux $ cargo cache -a > /dev/null
/tmp/qux $ time ~/src/cargo/target/debug/cargo update
    Updating git repository `file:///tmp/bar`
    Skipping git submodule `file:///tmp/foo`

real    0m0.029s
user    0m0.021s
sys     0m0.008s
/tmp/qux $
```

Fixes rust-lang#4247.

Signed-off-by: Jakub Sitnicki <jakub@cloudflare.com>
@jsitnicki jsitnicki force-pushed the respect-git-submod-update-none branch from a05107a to 26031b9 Compare June 7, 2022 12:52
@jsitnicki
Copy link
Contributor Author

v3:

  • explain to the user why the git submodule was skipped

@epage
Copy link
Contributor

epage commented Jun 7, 2022

Thanks!

@bors r+

@bors
Copy link
Collaborator

bors commented Jun 7, 2022

📌 Commit 26031b9 has been approved by epage

@bors bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-author Status: The marked PR is awaiting some action (such as code changes) from the PR author. labels Jun 7, 2022
@bors
Copy link
Collaborator

bors commented Jun 7, 2022

⌛ Testing commit 26031b9 with merge bc3b99d...

@bors
Copy link
Collaborator

bors commented Jun 7, 2022

☀️ Test successful - checks-actions
Approved by: epage
Pushing bc3b99d to master...

@bors bors merged commit bc3b99d into rust-lang:master Jun 7, 2022
bors added a commit to rust-lang-ci/rust that referenced this pull request Jun 8, 2022
Update cargo

7 commits in 38472bc19f2f76e245eba54a6e97ee6821b3c1db..85e457e158db216a2938d51bc3b617a5a7fe6015
2022-05-31 02:03:24 +0000 to 2022-06-07 21:57:52 +0000
- Make -Z http-registry use index.crates.io when accessing crates-io (rust-lang/cargo#10725)
- Respect submodule update=none strategy in .gitmodules (rust-lang/cargo#10717)
- Expose rust-version through env var (rust-lang/cargo#10713)
- add validation for string "true"/"false" in lto profile (rust-lang/cargo#10676)
- Enhance documentation of testing (rust-lang/cargo#10726)
- Clear disk space on CI. (rust-lang/cargo#10724)
- Enforce to use tar v0.4.38 (rust-lang/cargo#10720)
@ehuss ehuss added this to the 1.63.0 milestone Jun 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Is there a way to skip some submodules?
7 participants