Dataset Upload Failed with Multiple YAML's found #89

PanterSoft · 2022-08-28T09:06:21Z

Search before asking

I have searched the HUB issues and found no similar bug report.

HUB Component

No response

Bug

I am trying to upload a Dataset wich has the exact same structure as the tutorial suggests the yaml file has the correct format and the labels only consists of numbers.

After that was not working I tried the example dataset coco6 and got the same result shown in the image below.

Environment

I am using a 2021 MacBook Pro 14 inch M1 Pro running Safari.

Minimal Reproducible Example

Login
Create Dataset
Name Dataset and upload dataset.zip
Let it upload and fail while processing

Additional

No response

github-actions · 2022-08-28T09:06:55Z

👋 Hello @PanterSoft, thank you for raising an issue about Ultralytics HUB 🚀! Please visit https://ultralytics.com/hub to learn more, and see our ⭐️ HUB Guidelines to quickly get started uploading datasets and training YOLOv5 models.

If this is a 🐛 Bug Report, please provide screenshots and steps to recreate your problem to help us get started working on a fix.

If this is a ❓ Question, please provide as much information as possible, including dataset, model, environment details etc. so that we might provide the most helpful response.

We try to respond to all issues as promptly as possible. Thank you for your patience!

glenn-jocher · 2022-08-29T00:10:05Z

@PanterSoft hi I saw this issue was resolved on your side? Should we close the issue?

ghost · 2022-08-29T04:42:40Z

@PanterSoft hi I saw this issue was resolved on your side? Should we close the issue?

Sorry was a mistake I still have the same problem.

ghost · 2022-08-29T15:44:25Z

I found the issue when you create a zip file on MacOS it creates a hidden folder in the file which contains the same files which then gives the error multiple yaml files. solved it by uploading dataset from Windows

kalenmike · 2022-09-01T14:10:58Z

@nicomattes Thanks for letting us know. We will look into improving this for zips created with MacOS.

mehlkelm · 2022-10-17T08:31:32Z

I have the same problem.
Workaround: zip the folder from the command line like this:
https://apple.stackexchange.com/a/239587

glenn-jocher · 2022-10-17T12:02:28Z

@mehlkelm hi! Could you explain in more detail how the issue arises or show us screenshots of what your directory structure looks like that is causing problems?

mehlkelm · 2022-10-17T13:29:22Z

The issue arises (I think) because macOS creates a hidden (for mac users) directory "__MACOSX" inside the zip file that contains additional information about all the files. To a non-mac OS it looks like everything is duplicated.

Maybe the dataset upload tool should just ignore the __MACOSX folder in there.

glenn-jocher · 2022-10-17T13:32:03Z

@mehlkelm yeah good point. We definitely want a macOS-robust tool. Could you upload a small zip that's crashing HUB here or email it to glenn.jocher@ultralytics.com? Thanks!

mehlkelm · 2022-10-18T08:39:33Z

This is a completely made up example but it shows the problem.
(It doesn't crash HUB, it just shows the "Multiple YAML's found" problem)

macos_zip_problem_demo.zip

glenn-jocher · 2022-10-18T12:13:44Z

@mehlkelm thanks! We'll use the zip to debug and improve the dataset upload process.

glenn-jocher · 2022-10-18T12:35:55Z

When unzipping I see the __MACOSX/ directory containing a duplicate YAML:

(venv39) glennjocher@Glenns-MacBook-Air Downloads % unzip macos_zip_problem_demo.zip
Archive:  macos_zip_problem_demo.zip
   creating: macos_zip_problem_demo/
  inflating: macos_zip_problem_demo/.DS_Store  
  inflating: __MACOSX/macos_zip_problem_demo/._.DS_Store  
   creating: macos_zip_problem_demo/images/
  inflating: macos_zip_problem_demo/macos_zip_problem_demo.yaml  
  inflating: __MACOSX/macos_zip_problem_demo/._macos_zip_problem_demo.yaml  
   creating: macos_zip_problem_demo/labels/
  inflating: macos_zip_problem_demo/images/.DS_Store  
  inflating: __MACOSX/macos_zip_problem_demo/images/._.DS_Store  
   creating: macos_zip_problem_demo/images/train/
   creating: macos_zip_problem_demo/images/val/
  inflating: macos_zip_problem_demo/labels/.DS_Store  
  inflating: __MACOSX/macos_zip_problem_demo/labels/._.DS_Store  
   creating: macos_zip_problem_demo/labels/train/
   creating: macos_zip_problem_demo/labels/val/
  inflating: macos_zip_problem_demo/images/train/IMG_1931.jpeg  
  inflating: __MACOSX/macos_zip_problem_demo/images/train/._IMG_1931.jpeg  
  inflating: macos_zip_problem_demo/images/train/IMG_1925.jpeg  
  inflating: __MACOSX/macos_zip_problem_demo/images/train/._IMG_1925.jpeg  
  inflating: macos_zip_problem_demo/images/train/IMG_1923.jpeg  
  inflating: __MACOSX/macos_zip_problem_demo/images/train/._IMG_1923.jpeg  
  inflating: macos_zip_problem_demo/images/train/IMG_1915.jpeg  
  inflating: __MACOSX/macos_zip_problem_demo/images/train/._IMG_1915.jpeg  
  inflating: macos_zip_problem_demo/images/val/IMG_1927.jpeg  
  inflating: __MACOSX/macos_zip_problem_demo/images/val/._IMG_1927.jpeg  
  inflating: macos_zip_problem_demo/images/val/IMG_1914.jpeg  
  inflating: __MACOSX/macos_zip_problem_demo/images/val/._IMG_1914.jpeg  
  inflating: macos_zip_problem_demo/labels/train/IMG_1915.txt  
  inflating: __MACOSX/macos_zip_problem_demo/labels/train/._IMG_1915.txt  
  inflating: macos_zip_problem_demo/labels/train/IMG_1923.txt  
  inflating: __MACOSX/macos_zip_problem_demo/labels/train/._IMG_1923.txt  
  inflating: macos_zip_problem_demo/labels/train/IMG_1925.txt  
  inflating: __MACOSX/macos_zip_problem_demo/labels/train/._IMG_1925.txt  
  inflating: macos_zip_problem_demo/labels/train/IMG_1931.txt  
  inflating: __MACOSX/macos_zip_problem_demo/labels/train/._IMG_1931.txt  
  inflating: macos_zip_problem_demo/labels/val/IMG_1914.txt  
  inflating: __MACOSX/macos_zip_problem_demo/labels/val/._IMG_1914.txt  
  inflating: macos_zip_problem_demo/labels/val/IMG_1927.txt  
  inflating: __MACOSX/macos_zip_problem_demo/labels/val/._IMG_1927.txt

We need to update to be robust to this. I'll look into it.

mehlkelm · 2022-10-18T12:40:49Z

All the files in __MACOSX are hidden, as are the .DS_Store files, which are also some sort of OS specific info files. Maybe ignoring hidden files in general would be enough…

mehlkelm · 2022-10-18T12:42:28Z

(according to the unix way of treating files with names starting with a dot as hidden)

glenn-jocher · 2022-10-18T12:43:43Z

@mehlkelm actually this is really strange, on my mac if I unzip using the GUI/mouse commands everything unzips correctly into 1 directory. If I unzip using the terminal I get two top level directories:

(venv39) glennjocher@Glenns-MacBook-Air sandbox % unzip macos_zip_problem_demo.zip

mehlkelm · 2022-10-18T12:49:32Z

I guess the regular mac unzip processes the information in __MACOSX (technically they are resource forks) and doesn't interpret them as files, while all the non Mac tools/OS (even unzip in the terminal on mac?) think they are files.

mehlkelm · 2022-10-18T12:50:11Z

but yes, it's silly that the mac puts them there

glenn-jocher · 2022-10-18T13:01:46Z

Its ok, we can't change macOS but we can change HUB. I'll let you know when this is resolved.

glenn-jocher · 2022-10-18T13:20:00Z

@kalenmike I'm not able to produce any errors with HUBDatasetStats() on this zip. To reproduce I downloaded https://github.com/ultralytics/hub/files/9808500/macos_zip_problem_demo.zip, placed in a sandbox/ dir, and ran the code below.

        from utils.dataloaders import HUBDatasetStats
        stats = HUBDatasetStats('/Users/glennjocher/Downloads/sandbox/macos_zip_problem_demo.zip')
        stats.get_json()
        stats.process_images()

Initially directory contained only zip:

After unzip contained the unzipped dir plus an extra __MAXOSX dir with duplicated data:

But the actual dataset processing seemed to work without any duplicate YAML errors. Could you check HUB to see if the error is produced downstream of these steps?

glenn-jocher · 2022-10-18T14:02:40Z

@kalenmike @mehlkelm I've opened ultralytics/yolov5#9843 in YOLOv5 to better handle unzipping while rejecting files in an exclude list, i.e. .DS_Store and __MACOSX instances in file paths.

kalenmike · 2022-10-18T14:05:26Z

@glenn-jocher There is independent validation on the server that is not excluding the MACOSX folder by the looks of it.

glenn-jocher · 2022-10-18T19:56:27Z

@kalenmike ok got it, I'll let you try to debug on the server side.

kalenmike · 2022-10-20T11:53:03Z

This is now resolved and zip files created on Mac can now be processed successfully without special zipping.

glenn-jocher · 2022-10-20T15:03:18Z

@kalenmike thanks for the fix!!

@mehlkelm tested today and everything works now. Removing TODO. Let us know if you find any other issues or think of features you'd like to see!

PanterSoft added the bug Something isn't working label Aug 28, 2022

kalenmike closed this as completed Sep 1, 2022

kalenmike self-assigned this Sep 1, 2022

glenn-jocher added the todo Further action is needed by Ultralytics label Oct 18, 2022

glenn-jocher removed the todo Further action is needed by Ultralytics label Oct 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dataset Upload Failed with Multiple YAML's found #89

Dataset Upload Failed with Multiple YAML's found #89

PanterSoft commented Aug 28, 2022

github-actions bot commented Aug 28, 2022

glenn-jocher commented Aug 29, 2022

ghost commented Aug 29, 2022

ghost commented Aug 29, 2022 •

edited by ghost

Loading

kalenmike commented Sep 1, 2022

mehlkelm commented Oct 17, 2022

glenn-jocher commented Oct 17, 2022

mehlkelm commented Oct 17, 2022

glenn-jocher commented Oct 17, 2022

mehlkelm commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022 •

edited

Loading

mehlkelm commented Oct 18, 2022

mehlkelm commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

mehlkelm commented Oct 18, 2022

mehlkelm commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022 •

edited

Loading

glenn-jocher commented Oct 18, 2022

kalenmike commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

kalenmike commented Oct 20, 2022

glenn-jocher commented Oct 20, 2022

Dataset Upload Failed with Multiple YAML's found #89

Dataset Upload Failed with Multiple YAML's found #89

Comments

PanterSoft commented Aug 28, 2022

Search before asking

HUB Component

Bug

Environment

Minimal Reproducible Example

Additional

github-actions bot commented Aug 28, 2022

glenn-jocher commented Aug 29, 2022

ghost commented Aug 29, 2022

ghost commented Aug 29, 2022 • edited by ghost Loading

kalenmike commented Sep 1, 2022

mehlkelm commented Oct 17, 2022

glenn-jocher commented Oct 17, 2022

mehlkelm commented Oct 17, 2022

glenn-jocher commented Oct 17, 2022

mehlkelm commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022 • edited Loading

mehlkelm commented Oct 18, 2022

mehlkelm commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

mehlkelm commented Oct 18, 2022

mehlkelm commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022 • edited Loading

glenn-jocher commented Oct 18, 2022

kalenmike commented Oct 18, 2022

glenn-jocher commented Oct 18, 2022

kalenmike commented Oct 20, 2022

glenn-jocher commented Oct 20, 2022

ghost commented Aug 29, 2022 •

edited by ghost

Loading

glenn-jocher commented Oct 18, 2022 •

edited

Loading

glenn-jocher commented Oct 18, 2022 •

edited

Loading