Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve organization of the data, labels and models needed in tutorials and readme. #667

Closed
elboyran opened this issue Nov 24, 2023 · 4 comments

Comments

@elboyran
Copy link
Contributor

elboyran commented Nov 24, 2023

There are three places for data, models, and labels in our repo:

  1. https://github.com/dianna-ai/dianna/tree/main/tutorials/data (data and labels)
  2. https://github.com/dianna-ai/dianna/tree/main/tutorials/models (models)
  3. https://github.com/dianna-ai/dianna/tree/main/dianna/data (as visible in our updated schema, see Update the architecture schema #639: data, lables and models, some repetition of those in locations 1 and 2 above )

This is messy and we should improve on organizing the repo and remove the repetitoins!

@elboyran elboyran changed the title Improve the data and models needed in tutorials/readme. Improve organization of the data, labels and models needed in tutorials and readme. Nov 24, 2023
@elboyran elboyran self-assigned this Jan 26, 2024
@elboyran elboyran moved this from To do to In progress in SS Sprint 4 - Complete WP1 Jan 26, 2024
elboyran pushed a commit that referenced this issue Jan 26, 2024
elboyran added a commit that referenced this issue Jan 26, 2024
elboyran added a commit that referenced this issue Jan 26, 2024
@elboyran elboyran added the standup Temp label- for disscussion with the team next standup label Jan 29, 2024
@cwmeijer cwmeijer removed the standup Temp label- for disscussion with the team next standup label Jan 29, 2024
@elboyran
Copy link
Contributor Author

elboyran commented Jan 29, 2024

PR #699

@elboyran elboyran moved this from In progress to Ready for review in SS Sprint 4 - Complete WP1 Jan 31, 2024
@elboyran elboyran moved this from Ready for review to In progress in SS Sprint 4 - Complete WP1 Feb 2, 2024
@elboyran
Copy link
Contributor Author

elboyran commented Feb 2, 2024

Relocated all data, models, and labels used by the dashboard
Relocated all tutorials and the data, models, and labels used by them
TODO:

  • restore bee.jpg
  • Update README with correct links, mention the new folders including the ONNX conversions
  • Fix all broken paths in the notebooks.
  • Fix broken links in documentation

@cwmeijer cwmeijer added this to In progress in SS Sprint 5 Feb 7, 2024
@cwmeijer cwmeijer added this to In progress in SS Sprint 6 Feb 28, 2024
elboyran added a commit that referenced this issue Mar 1, 2024
elboyran added a commit that referenced this issue Mar 1, 2024
#667 Fixing tutorial paths in tutorial README
elboyran added a commit that referenced this issue Mar 1, 2024
elboyran added a commit that referenced this issue Mar 1, 2024
@elboyran elboyran moved this from In progress to Ready for review in SS Sprint 6 Mar 6, 2024
@elboyran elboyran added standup Temp label- for disscussion with the team next standup and removed standup Temp label- for disscussion with the team next standup labels Mar 6, 2024
@elboyran
Copy link
Contributor Author

elboyran commented Mar 7, 2024

Old and proposed folder structures of dianna/dianna and dianna/tutorials :

main branch

dianna (6 directories, 56 files)

├── init.py
├── ...
├── dashboard
│   ├── ...
├── data
│   ├── bee_2.png
│   ├── binary-mnist.npz
│   ├── digit0.jpeg
│   ├── digit0.jpg
│   ├── digit0.png
│   ├── digit1.png
│   ├── labels_mnist.txt
│   ├── labels_resnet50.txt
│   ├── labels_text.txt
│   ├── leafsnap_model.onnx
│   ├── logo.png
│   ├── mnist_model_tf.onnx
│   ├── movie_review_model.onnx
│   ├── movie_reviews_word_vectors.txt
│   ├── season_prediction_model_temp_max_binary.onnx
│   ├── weather_data.npy
│   └── weather_data_labels.txt
├── methods
│   ├── ...
├── utils
│   ├── ...
└── visualization
│   ├── ...

tutorials (4 directories, 38 files)

├── README.md
├── conversion_onnx
│   ├── ...
├── data
│   ├── binary-mnist.npz
│   ├── coffee_test.csv
│   ├── coffee_train.csv
│   ├── leafsnap_classes.csv
│   ├── leafsnap_example_acer_rubrum.jpg
│   ├── movie_reviews_word_vectors.txt
│   └── shapes.npz
├── demo.ipynb
├── img
│   ├── bee.jpg
│   └── digit0.jpg
├── kernelshap_geometric_shapes.ipynb
├── kernelshap_mnist.ipynb
├── kernelshap_tabular_penguin.ipynb
├── kernelshap_tabular_weather.ipynb
├── lime_images.ipynb
├── lime_tabular_penguin.ipynb
├── lime_tabular_weather.ipynb
├── lime_text.ipynb
├── lime_timeseries_coffee.ipynb
├── lime_timeseries_weather.ipynb
├── models
│   ├── coffee.onnx
│   ├── geometric_shapes_model.onnx
│   ├── leafsnap_model.onnx
│   ├── mnist_model.onnx
│   ├── mnist_model_tf.onnx
│   ├── movie_review_model.onnx
│   ├── penguin_model.onnx
│   ├── season_prediction_model_temp_max_binary.onnx
│   └── sunshine_hours_regression_model.onnx
├── rise_imagenet.ipynb
├── rise_mnist.ipynb
├── rise_text.ipynb
└── rise_timeseries_weather.ipynb


667-reorganize branch

dianna (8 directories, 66 files)

├── init.py
├── ...
├── dashboard
│   ├── ...
├── data
│   ├── bee.jpg
│   ├── binary-mnist.npz
│   ├── digit0.jpeg
│   ├── digit0.jpg
│   ├── digit0.png
│   ├── digit1.png
│   ├── leafsnap_example_acer_rubrum.jpg
│   ├── logo.png
│   ├── shapes.npz
│   └── weather_data.npy
├── labels
│   ├── coffee_test.csv
│   ├── coffee_train.csv
│   ├── labels_mnist.txt
│   ├── labels_resnet50.txt
│   ├── labels_text.txt
│   ├── leafsnap_classes.csv
│   ├── movie_reviews_word_vectors.txt
│   └── weather_data_labels.txt
├── methods
│   ├── ...
├── models
│   ├── coffee.onnx
│   ├── geometric_shapes_model.onnx
│   ├── leafsnap_model.onnx
│   ├── mnist_model.onnx
│   ├── mnist_model_tf.onnx
│   ├── movie_review_model.onnx
│   ├── penguin_model.onnx
│   ├── season_prediction_model_temp_max_binary.onnx
│   └── sunshine_hours_regression_model.onnx
├── utils
│   ├── ...
└── visualization
├── ...

tutorials (5 directories, 20 files)

├── README.md
├── conversion_onnx
│   ├── ...
├── demo.ipynb
└── explainers
├── KernelSHAP
│   ├── kernelshap_geometric_shapes.ipynb
│   ├── kernelshap_mnist.ipynb
│   ├── kernelshap_tabular_penguin.ipynb
│   └── kernelshap_tabular_weather.ipynb
├── LIME
│   ├── lime_images.ipynb
│   ├── lime_tabular_penguin.ipynb
│   ├── lime_tabular_weather.ipynb
│   ├── lime_text.ipynb
│   ├── lime_timeseries_coffee.ipynb
│   └── lime_timeseries_weather.ipynb
└── RISE
├── rise_imagenet.ipynb
├── rise_mnist.ipynb
├── rise_text.ipynb
└── rise_timeseries_weather.ipynb

@elboyran
Copy link
Contributor Author

@cwmeijer do you mind merging the PR #699 after the release?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
No open projects
Development

No branches or pull requests

2 participants