Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sparse: Update functional tests to check sparse mode #77

Closed
derrickstolee opened this issue Aug 20, 2019 · 1 comment · Fixed by #84
Closed

Sparse: Update functional tests to check sparse mode #77

derrickstolee opened this issue Aug 20, 2019 · 1 comment · Fixed by #84
Assignees
@derrickstolee
Labels
engineering
Milestone
Demo

Comments

@derrickstolee
Copy link
Contributor

After #76, update the sparse-mode functional tests to work with a sparse scalar clone.
@derrickstolee derrickstolee added this to the Demo milestone Aug 20, 2019
@derrickstolee derrickstolee mentioned this issue Aug 20, 2019
Merged
@derrickstolee derrickstolee self-assigned this Aug 22, 2019
@derrickstolee
Copy link
Contributor Author

After #54 merges, I'll make this my next task. Having a larger set of functional tests for the sparse mode will be helpful in safely making changes like microsoft/git#185.

derrickstolee added a commit that referenced this issue Aug 22, 2019
@derrickstolee
Merge pull request #54: Initial sparse implementation
19d745f 
Uses the code currently at microsoft/git#180. Covers the basics of #8.

* The default `scalar clone` runs `git sparse-checkout init` so the working directory only has files at root.

* Run `git sparse-checkout add <folders.txt` to pipe in a list of folders, and it will expand those files.

Example workflow:

```sh
$ scalar clone https://dev.azure.com/gvfs/ci/_git/ForTests
Clone parameters:
  Repo URL:     https://dev.azure.com/gvfs/ci/_git/ForTests
  Branch:       Default
  Cache Server: Default
  Local Cache:  C:\.scalarCache
  Destination:  C:\_git\test2\ForTests
Authenticating...Succeeded
Querying remote for config...Succeeded
Using cache server: None (https://dev.azure.com/gvfs/ci/_git/ForTests)

WARNING: Unable to validate your Scalar version
Server not configured to provide supported Scalar versions

Cloning...Succeeded
Fetching commits and trees from origin (no cache server)...Succeeded
Validating repo...Succeeded
Mounting...Succeeded

$ cd ForTests/src/
$ ls
AuthoringTests.md  GvFlt_EULA.md  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ echo GVFS/GVFS.Common >>../folders.txt
$ echo GVFS/GVFS.UnitTests >>../folders.txt
$ echo GitHooksLoader >>../folders.txt
$ git sparse-checkout add <../folders.txt

$ ls
AuthoringTests.md  GitHooksLoader/  GvFlt_EULA.md  GVFS/  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ echo GVFS/GVFS >>../folders2.txt
$ echo GVFS/GVFS.FunctionalTests >>../folders2.txt
$ git sparse-checkout add <../folders2.txt

$ ls
AuthoringTests.md  GitHooksLoader/  GvFlt_EULA.md  GVFS/  GVFS.sln  License.md  nuget.config  Protocol.md  Readme.md  Settings.StyleCop

$ ls GVFS
GVFS/  GVFS.Common/  GVFS.FunctionalTests/  GVFS.UnitTests/  LibGit2Sharp.NativeBinaries.props  ProjectedFSLib.NativeBinaries.props

$ git sparse-checkout list
/
/GVFS/
/GVFS/GVFS/*
/GVFS/GVFS.Common/*
/GVFS/GVFS.FunctionalTests/*
/GVFS/GVFS.UnitTests/*
/GitHooksLoader/*

```

I spun up a few remaining issues for follow-up work: #76, #77, #78.
@derrickstolee derrickstolee mentioned this issue Aug 23, 2019
Merged
derrickstolee added a commit that referenced this issue Aug 27, 2019
@derrickstolee
Merge pull request #84: Run GitRepoTests in sparse mode
38070a1 
Resolves #77.

Update the functional tests to initialize a sparse enlistment with `scalar sparse --add-stdin` and to test lots of Git commands in that mode versus a vanilla Git repo. Uses the SparseMode checks from microsoft/vfsforgit to verify the contents _inside the cone_ match, but ignores all changes outside the cone. 

This took a lot longer than anticipated. Here are a few things that happened along the way:

* The untracked cache doesn't work correctly on Windows. The functional tests were failing due to quickly running status after every call. This _was_ working correctly on Mac, so maybe we re-enable it on a platform-specific basis later. Reverts #86 until we can revisit.

* The clone verb was causing problems. This is due to some timing concerns, mostly: VFS for Git needs a checkout to construct an index before mounting, but we can't checkout before mounting. Since the sparse verb doesn't prefetch before checkout (it doesn't need _everything_, but needs _some_ things), we needed to change this order. However, the `CloneVerb` code is very convoluted. It required a lot of refactoring to split out that checkout code. Hopefully the new code is cleaner and easier to modify.

* The input for the cone matches is slightly different for Git: we need `/` as the path separator, but the logic for interpreting those paths in the test code uses the platform-specific separator.

* The parallel test scripts were causing new issues in the `LooseObjectStepTests`. Turns out that in #68, I messed up the base constructor parameters and didn't give that enlistment a local object cache. That meant that other clones had prefetches and loose object downloads causing different numbers in these tests. That's fixed.

* `CloneWithDefaultLocalCacheLocation()` was failing on the build machine but not on my laptop. The error appeared to be due to failing to register the mount with the service. Turns out, this test used to use the `--no-mount` option, but that was removed in #80. Oops! Maybe we'll put that back as an `--internal_use_only` parameter. OR we can figure out why the service isn't running during that test.
@derrickstolee derrickstolee mentioned this issue Aug 27, 2019
Merged
derrickstolee added a commit that referenced this issue Aug 27, 2019
@derrickstolee
Merge pull request #89: UntrackedCache: enable based on platform
1dabdcc 
While adding functional tests in #77, I noticed that functional tests on Windows were failing due to the untracked cache. I was unable to repro them when paused in the debugger, so it must be something wrong with however Git detects changes to the filesystem using timestamps.

For now, re-enable the untracked cache on non-Windows platforms. Since it is related to the filesystem, I put the constant there.
@derrickstolee derrickstolee mentioned this issue Aug 28, 2019
Merged
derrickstolee added a commit that referenced this issue Aug 28, 2019
@derrickstolee
Merge pull request #92: Always use read-object hook during checkout
bdd2a24 
While refactoring the `CloneVerb` in #77, I moved the `git checkout -f` command to the end of clone, as needed. This is because we will actually fill out blobs during this initial checkout.

However, there was one slight problem: we need the read-object hook. There are two ways to do a checkout in the `GitProcess`, and one was not allowing the read-object hook.

Before #77, this checkout was called before the mount was ready, but that's no longer important.

Thanks @jamill for finding this bug!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@derrickstolee derrickstolee

Labels
engineering
Projects
None yet
Milestone
Demo
Development

Successfully merging a pull request may close this issue.

Run GitRepoTests in sparse mode
1 participant
@derrickstolee