Unable to Unzip Test Assets -- gzip: stdin: invalid compressed data--format violated

[mdh1418]

Libraries Test Run release coreclr Linux arm Release

stderr: 
gzip: stdin: invalid compressed data--format violated
/bin/tar: Unexpected EOF in archive
/bin/tar: Unexpected EOF in archive
/bin/tar: Error is not recoverable: exiting now
 
error: undefined;
##[error]Extraction failed for file: /mnt/vss/_work/1/s/__download__/libraries_test_assets_Linux_arm_Release/libraries_test_assets_Linux_arm_Release.tar.gz 

https://dev.azure.com/dnceng/public/_build/results?buildId=1636124&view=logs&j=3af70de0-9753-58b8-b476-56d8238660f2&t=ebc957cc-2054-5b9c-76a0-1c61cc6bbc2e

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Libraries Test Run release coreclr Linux arm Release

stderr: 
gzip: stdin: invalid compressed data--format violated
/bin/tar: Unexpected EOF in archive
/bin/tar: Unexpected EOF in archive
/bin/tar: Error is not recoverable: exiting now
 
error: undefined;
##[error]Extraction failed for file: /mnt/vss/_work/1/s/__download__/libraries_test_assets_Linux_arm_Release/libraries_test_assets_Linux_arm_Release.tar.gz 

https://dev.azure.com/dnceng/public/_build/results?buildId=1636124&view=logs&j=3af70de0-9753-58b8-b476-56d8238660f2&t=ebc957cc-2054-5b9c-76a0-1c61cc6bbc2e

Author:	mdh1418
Assignees:	-
Labels:	area-Infrastructure-libraries, untriaged
Milestone:	-

[carlossanlop]

How often does this happen? Do you know where the code is that calls the tar tool? I'd like to see the arguments we are passing to it.

[mdh1418]

I saw this happen twice on a seemingly unrelated PR (successive attempts). I don't know where the code is, but I can try to look what that step in the lane is doing.

[ViktorHofer]

Sounds like an AzDO issue which isn't actionable on our side. If this is still happening, please talk with core-eng to understand how to reach out to their team to get an IcM ticket opened.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[ghost]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[AaronRobinsonMSFT]

Perhaps another instance of AzDO issue: https://dev.azure.com/dnceng/public/_build/results?buildId=1783143&view=logs&jobId=2662b7d9-28ff-5dca-b58f-3053d0bc5578&j=2662b7d9-28ff-5dca-b58f-3053d0bc5578&t=4d4a5b9c-0a42-5d4d-71ab-d0e293a111d8

This is on Windows x64

ERROR: D:\a\_work\1\s\__download__\CoreCLRProduct___windows_x64_checked\CoreCLRProduct___windows_x64_checked.zip
Can not open the file as archive


7-Zip 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21
Error: Extraction failed for file: D:\a\_work\1\s\__download__\CoreCLRProduct___windows_x64_checked\CoreCLRProduct___windows_x64_checked.zip 

code: 2 
Scanning the drive for archives:
stdout: 
1 file, 0 bytes

7-Zip 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21
Extracting archive: D:\a\_work\1\s\__download__\CoreCLRProduct___windows_x64_checked\CoreCLRProduct___windows_x64_checked.zip

Can't open as archive: 1
Files: 0
Size:       0
Compressed: 0
##[error]Extraction failed for file: D:\a\_work\1\s\__download__\CoreCLRProduct___windows_x64_checked\CoreCLRProduct___windows_x64_checked.zip 
code: 2 
stdout: 
7-Zip 19.00 (x64) : Copyright (c) 1999-2018 Igor Pavlov : 2019-02-21

Scanning the drive for archives:
1 file, 0 bytes

[ViktorHofer]

@MattGal @dotnet/dnceng are you aware of this issue? Need to gather some data on how often this is happening.

[MattGal]

@MattGal @dotnet/dnceng are you aware of this issue? Need to gather some data on how often this is happening.

First I've seen this (recently). From the symptoms, it's either a partial artifact download or a malformed archive which should be detect-able from downloading the same artifact locally and unzipping via 7-zip.

[MattGal]

Yup it's what I guessed:

From the logs:

Downloaded  to D:\a\_work\1\s\__download__\CoreCLRProduct___windows_x64_checked.zip
Total Files: 1, Processed: 1, Skipped: 0, Failed: 0, Download time: 32.249 secs, Download size: 402Bytes
Starting artifact items integrity check
Artifact items integrity check successfully finished
Successfully downloaded artifacts to D:\a\_work\1\s\__download__

Note this file is 301 MB, not 402 bytes. There's a bug with the download task here. @ilyas1974 FYI.

[MattGal]

After checking timestamps and convincing myself the artifact was finished uploading many minutes before the download occurred, I filed https://portal.microsofticm.com/imp/v3/incidents/details/309716763/home to ask Azure DevOps to investigate. Note from the logging (Artifact items integrity check successfully finished) this isn't the first time we've asked them to do better here.

[ViktorHofer]

Thanks for opening the IcM.

[MattGal]

@ViktorHofer I synched with an engineer from the Azure DevOps team who is investigating the IcM linked above. He suggests that updating the task version used to latest might buy you some validation checking (it's cheap to try, at least)
As seen in places like
https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/download-artifact-step.yml#L10-L17
https://github.com/dotnet/runtime/blob/main/eng/pipelines/common/download-specific-artifact-step.yml#L10-L21,
it should literally boil down "change to DownloadBuildArtifacts@0 --> DownloadBuildArtifacts@1 and verify"

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Libraries Test Run release coreclr Linux arm Release

stderr: 
gzip: stdin: invalid compressed data--format violated
/bin/tar: Unexpected EOF in archive
/bin/tar: Unexpected EOF in archive
/bin/tar: Error is not recoverable: exiting now
 
error: undefined;
##[error]Extraction failed for file: /mnt/vss/_work/1/s/__download__/libraries_test_assets_Linux_arm_Release/libraries_test_assets_Linux_arm_Release.tar.gz 

https://dev.azure.com/dnceng/public/_build/results?buildId=1636124&view=logs&j=3af70de0-9753-58b8-b476-56d8238660f2&t=ebc957cc-2054-5b9c-76a0-1c61cc6bbc2e

Author:	mdh1418
Assignees:	-
Labels:	tracking-external-issue, area-Infrastructure
Milestone:	Future

[ViktorHofer]

He suggests that updating the task version used to latest might buy you some validation checking (it's cheap to try, at least)
As seen in places like

@dotnet/runtime-infrastructure we might want to do that as we are still on version "@0".

[akoeplinger]

So I looked at the sources and from what I can tell the v1 and v0 tasks are identical (I think that happened in microsoft/azure-pipelines-tasks@714ebff) looks like that commit was reverted.

There is an option on the task called checkDownloadedFiles that we could enable, though that seems to also only fail the task if the validation failed.

[akoeplinger]

Maybe updating to DownloadPipelineArtifact@2 would help, it is recommended by the docs: https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/download-build-artifacts?view=azure-devops