Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update README #87

Merged
merged 1 commit into from
Jan 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 53 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,54 +8,52 @@ Ianvs is a distributed synergy AI benchmarking project incubated in KubeEdge SIG

## Scope

The distributed synergy AI benchmarking Ianvs aims to test the performance of distributed synergy AI solutions following recognized standards, in order to facilitate more efficient and effective development.
The distributed synergy AI benchmarking Ianvs aims to test the performance of distributed synergy AI solutions following recognized standards, in order to facilitate more efficient and effective development.

The scope of Ianvs includes

- Providing end-to-end benchmark toolkits across devices, edge nodes, and cloud nodes based on typical distributed-synergy AI paradigms and applications.
- Tools to manage test environment. For example, it would be necessary to support the CRUD (Create, Read, Update, and Delete) actions in test environments. Elements of such test environments include algorithm-wise and system-wise configuration.
- Tools to control test cases. Typical examples include paradigm templates, simulation tools, and hyper-parameter-based assistant tools.
- Tools to manage benchmark presentation, e.g., leaderboard and test report generation.
- Tools to manage test environment. For example, it would be necessary to support the CRUD (Create, Read, Update, and Delete) actions in test environments. Elements of such test environments include algorithm-wise and system-wise configuration.
- Tools to control test cases. Typical examples include paradigm templates, simulation tools, and hyper-parameter-based assistant tools.
- Tools to manage benchmark presentation, e.g., leaderboard and test report generation.
- Cooperation with other organizations or communities, e.g., in KubeEdge SIG AI, to establish comprehensive benchmarks and developed related applications, which can include but are not limited to
- Dataset collection, re-organization, and publication
- Formalized specifications, e.g., standards
- Holding competitions or coding events, e.g., open source promotion plan
- Maintaining solution leaderboards or certifications for commercial usage

- Dataset collection, re-organization, and publication
- Formalized specifications, e.g., standards
- Holding competitions or coding events, e.g., open source promotion plan
- Maintaining solution leaderboards or certifications for commercial usage

## Architecture

The architectures and related concepts are shown in the below figure. The ianvs is designed to run **within a single node**. Critical components include

- Test Environment Manager: the CRUD of test environments serving for global usage
- Test Case Controller: control the runtime behavior of test cases like instance generation and vanish
- Generation Assistant: assist users to generate test cases based on certain rules or constraints, e.g., the range of parameters
- Simulation Controller: control the simulation process of edge-cloud synergy AI, including the instance generation and vanishment of simulation containers
- Test Case Controller: control the runtime behavior of test cases like instance generation and vanish
- Generation Assistant: assist users to generate test cases based on certain rules or constraints, e.g., the range of parameters
- Simulation Controller: control the simulation process of edge-cloud synergy AI, including the instance generation and vanishment of simulation containers
- Story Manager: the output management and presentation of the test case, e.g., leaderboards

![](docs/guides/images/ianvs_arch.png)

More details on Ianvs components:

1. Test-Environment Manager supports the CRUD of Test environments, which basically includes
- Algorithm-wise configuration
- Public datasets
- Pre-processing algorithms
- Feature engineering algorithms
- Post-processing algorithms like metric computation
- System-wise configuration
- Overall architecture
- System constraints or budgets
- End-to-end cross-node
- Per node
1. Test-case Controller, which includes but is not limited to the following components
- Templates of common distributed-synergy-AI paradigms, which can help the developer to prepare their test case without too much effort. Such paradigms include edge-cloud synergy joint inference, incremental learning, federated learning, and lifelong learning.
- Simulation tools. Develop simulated test environments for test cases
- Other tools to assist test-case generation. For instance, prepare test cases based on a given range of hyper-parameters.
1. Story Manager, which includes but is not limited to the following components
- Leaderboard generation
- Test report generation

- Algorithm-wise configuration
- Public datasets
- Pre-processing algorithms
- Feature engineering algorithms
- Post-processing algorithms like metric computation
- System-wise configuration
- Overall architecture
- System constraints or budgets
- End-to-end cross-node
- Per node
2. Test-case Controller, which includes but is not limited to the following components
- Templates of common distributed-synergy-AI paradigms, which can help the developer to prepare their test case without too much effort. Such paradigms include edge-cloud synergy joint inference, incremental learning, federated learning, and lifelong learning.
- Simulation tools. Develop simulated test environments for test cases
- Other tools to assist test-case generation. For instance, prepare test cases based on a given range of hyper-parameters.
3. Story Manager, which includes but is not limited to the following components
- Leaderboard generation
- Test report generation

## Guides

Expand All @@ -69,9 +67,29 @@ Follow the [Ianvs installation document](docs/guides/how-to-install-ianvs.md) to

### Examples

Scenario PCB-AoI:[Industrial Defect Detection on the PCB-AoI Dataset](/docs/proposals/scenarios/industrial-defect-detection/pcb-aoi.md).
Example PCB-AoI-1:[Testing single task learning in industrial defect detection](/docs/proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi.md).
Example PCB-AoI-2:[Testing incremental learning in industrial defect detection](/docs/proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi.md).
- Scenario PCB-AoI:[Industrial Defect Detection on the PCB-AoI Dataset](docs/proposals/scenarios/industrial-defect-detection/pcb-aoi.md).

- Example PCB-AoI-1:[Testing single task learning in industrial defect detection](docs/proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi.md).

- Example PCB-AoI-2:[Testing incremental learning in industrial defect detection](docs/proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi.md).

- Scenario Cityscapes-Synthia: [Curb Detetion on Cityscapes-Synthia Dataset](docs/proposals/algorithms/lifelong-learning/Additional-documentation/curb_detetion_datasets.md)

- Example Cityscapes-Synthia-1: [Lifelong learning in semantic segmentation](examples/cityscapes-synthia/lifelong_learning_bench/semantic-segmentation/README.md)

- Example Cityscapes-Synthia-2: [Lifelong learning in curb detetion](examples/cityscapes-synthia/lifelong_learning_bench/curb-detection/README.md)

- Example Cityscapes-Synthia-3: [Scene based unknown task recognition in curb detetion](examples/cityscapes-synthia/scene-based-unknown-task-recognition/curb-detection/README.md)

- Example Cityscapes-Synthia-4: [Integrating GAN and Self-taught Learning into Ianvs Lifelong Learning](examples/cityscapes/lifelong_learning_bench/unseen_task_processing-GANwithSelfTaughtLearning/README.md)

- Scenario Cloud-Robotics: [Semantic Segmentation on Cloud-Robotics Dataset](docs/proposals/scenarios/Cloud-Robotics/Cloud-Robotics_zh.md)

- Example Cloud-Robotics-1: [Lifelong learning in semantic segmentation](examples/robot/lifelong_learning_bench/semantic-segmentation/README.md)

- Example Cloud-Robotics-2: [Class increment learning in semantic segmentation](examples/robot-cityscapes-synthia/lifelong_learning_bench/semantic-segmentation/README.md)

- Example Cloud-Robotics-3: [Lifelong learning in sam annotation](examples/robot/lifelong_learning_bench/sam_annotation/tutorial.md)

## Roadmap

Expand All @@ -82,7 +100,7 @@ Example PCB-AoI-2:[Testing incremental learning in industrial defect detection
Routine Community Meeting for KubeEdge SIG AI runs weekly:

- Europe Time: **Thursdays at 16:30-17:30 Beijing Time**.
([Convert to your timezone.](https://www.thetimezoneconverter.com/?t=16%3A30&tz=GMT%2B8&))
([Convert to your timezone.](https://www.thetimezoneconverter.com/?t=16%3A30&tz=GMT%2B8&))

Resources:

Expand All @@ -107,4 +125,4 @@ If you're interested in being a contributor and want to get involved in developi

## License

Ianvs is under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.
Ianvs is under the Apache 2.0 license. See the [LICENSE](LICENSE) file for details.
2 changes: 1 addition & 1 deletion docs/guides/how-to-build-simulation-env.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Using simulation controller build the edge-cloud AI simulation environment with just one host
# How to build simulation env

This document introduces how to build a edge-cloud AI simulation environment(e.g. kubeedge sedna) with just one host.

Expand Down
6 changes: 5 additions & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Start your journey on Ianvs with the following links:

guides/how-to-install-ianvs
guides/how-to-test-algorithms
guides/how-to-build-simulation-env


.. toctree::
Expand All @@ -56,7 +57,8 @@ Start your journey on Ianvs with the following links:

Single Task Learning: FPN <proposals/algorithms/single-task-learning/fpn>
Incremental Learning: BasicIL-FPN <proposals/algorithms/incremental-learning/basicIL-fpn>

Lifelong Learning: RFNet <proposals/algorithms/lifelong-learning/Cloud-Robotic AI Benchmarking for Edge-cloud Collaborative Lifelong Learning>

.. toctree::
:maxdepth: 1
:titlesonly:
Expand All @@ -76,6 +78,8 @@ Start your journey on Ianvs with the following links:

Leaderboard: Single Task Learning on PCB-AoI <leaderboards/leaderboard-in-industrial-defect-detection-of-PCB-AoI/leaderboard-of-single-task-learning>
Leaderboard: Incremental Learning on PCB-AoI <leaderboards/leaderboard-in-industrial-defect-detection-of-PCB-AoI/leaderboard-of-incremental-learning>
Leaderboard: Lifelong Learning on Cloud-Robotics <leaderboards/leaderboard-in-semantic-segmentation-of-Cloud-Robotics/leaderboard-of-lifelong-learning>
Leaderboard: Edge-Cloud Collaboration on Cloud-Robotics <leaderboards/leaderboard-in-semantic-segmentation-of-Cloud-Robotics/leaderboard-of-SAM-based-Edge-Cloud-Collaboration>
Test Report: Single Task Learning on PCB-AoI <proposals/test-reports/testing-single-task-learning-in-industrial-defect-detection-with-pcb-aoi>
Test Report: Incremental Learning on PCB-AoI <proposals/test-reports/testing-incremental-learning-in-industrial-defect-detection-with-pcb-aoi>

Expand Down
Loading