Support aml #2615

SparkSnail · 2020-06-30T05:18:40Z

No description provided.

Merge master

1. rename storage file name 2. add more log on status changes 3. change isEnd to isAlive for better naming

add internal prefix for internal storage methods for clear usage. fix pylint errors minor fixes

rename methods of storageService move trial to a seperated file fix some bugs.

fix openPAI breaking changes

fix minor bugs

to router training service for better understanding.

merge master

trialService is used to support different submission types like AML.

merge master

TrialDispatcher is easier to understand it's purpose.

…to dev-aml

src/nni_manager/training_service/reusable/trialDispatcher.ts

merge master

Some small fix and merge logic

ultmaster · 2020-06-30T15:19:29Z

src/nni_manager/training_service/reusable/aml/amlConfig.ts

+}
+
+export class AMLEnvironmentInformation extends EnvironmentInformation {
+        public amlClient?: AMLClient;


bad indentation

ultmaster · 2020-06-30T15:22:59Z

docs/en_US/TrainingService/AMLMode.md

+trial:
+  command: python3 mnist.py
+  codeDir: .
+  computeTarget: ussc40rscl


Replace it with a placeholder

ultmaster · 2020-06-30T15:24:47Z

src/nni_manager/training_service/reusable/environments/amlEnvironmentService.ts

+        }
+        const amlEnvironment: AMLEnvironmentInformation = environment as AMLEnvironmentInformation;
+        const environmentLocalTempFolder = path.join(this.experimentRootDir, this.experimentId, "environment-temp");
+        environment.command = `import os\nos.system('${amlEnvironment.command}')`;


Needs to escape special characters like ' here.

don't need to process this command, this is environment command, not trial's command here.

ultmaster · 2020-06-30T15:25:39Z

examples/trials/mnist-pytorch/config_aml.yml

+  command: python3 mnist.py
+  codeDir: .
+  computeTarget: ussc40rscl
+  nodeCount: 1


Each trial will use one node, i.e., all 8 GPUs?

ultmaster · 2020-06-30T15:26:39Z

docs/en_US/TrainingService/AMLMode.md

+Compared with [LocalMode](LocalMode.md) trial configuration in aml mode have these additional keys:
+* computeTarget
+    * required key. The computer cluster name you want to use in your AML workspace.
+* nodeCount


I think nodeCount can default to 1 because multi-machine runs are seldom used.

yes, perhaps hide this variable is better, has removed.

ultmaster · 2020-06-30T15:29:11Z

docs/en_US/TrainingService/AMLMode.md

+  command: python3 mnist.py
+  codeDir: .
+  computeTarget: ussc40rscl
+  nodeCount: 1


Where is docker image?

fixed, missed this variable in doc.

tools/nni_trial_tool/trial_runner.py

squirrelsc · 2020-06-30T12:28:42Z

src/nni_manager/training_service/reusable/trialDispatcher.ts

@@ -58,6 +59,8 @@ class TrialDispatcher implements TrainingService {
        this.environments = new Map<string, EnvironmentInformation>();
        this.metricsEmitter = new EventEmitter();
        this.experimentId = getExperimentId();
+        this.experimentRootDir = getExperimentRootDir();


It can be changed to a local variable, as it's used only once in run.

squirrelsc · 2020-07-01T00:08:09Z

docs/en_US/TrainingService/AMLMode.md

-  computeTarget: ussc40rscl
-  nodeCount: 1
+  computeTarget: ${replace_to_your_computeTarget}
+  image: msranni/nni


Is aml installed in this image?

SparkSnail and others added 30 commits May 29, 2020 17:02

Merge pull request #251 from microsoft/master

dcd2ffd

Merge master

init changes

a738331

Merge remote-tracking branch 'official/master' into 2391-reuse-job

3177aeb

refactors

2aafac1

1. rename storage file name 2. add more log on status changes 3. change isEnd to isAlive for better naming

refactoring

0435b7f

add internal prefix for internal storage methods for clear usage. fix pylint errors minor fixes

minor fix, and take some review comments.

2e5ef51

move reuse to upper level

6d7bc62

support multi nodes

c67b162

rename methods of storageService move trial to a seperated file fix some bugs.

fix eslint errors

e13a620

support multi environments better

59d4a71

fix openPAI breaking changes

Merge remote-tracking branch 'official/master' into 2391-reuse-job

eae0540

code refactor

81c49cf

fix minor bugs

fix openpai yaml format

92cab3a

fix k8s yaml schema

0674d88

rename forward training service

e5b9665

to router training service for better understanding.

Merge remote-tracking branch 'official/master' into 2391-reuse-job

67ef648

Merge pull request #252 from microsoft/master

3b8b6fb

merge master

add trialService

1e626fd

trialService is used to support different submission types like AML.

not send stop for single node

c6b6061

Merge pull request #253 from microsoft/master

916e444

merge master

rename environmentManager to trialDispatcher

b8e47be

TrialDispatcher is easier to understand it's purpose.

support no central storage service

0ee933a

init

c7973be

improve delopment support

c094057

Merge remote-tracking branch 'official/master' into 2391-reuse-job

c2735d3

use latest storage component

d0b2504

add gpu info

c8d4696

work version

1a9f19f

separate channel and add gpu collector in runner

3f4c177

Merge branch '2391-reuse-job' of https://github.com/squirrelsc/nni in…

648e0bb

…to dev-aml

squirrelsc reviewed Jun 30, 2020

View reviewed changes

src/nni_manager/training_service/reusable/trialDispatcher.ts Outdated Show resolved Hide resolved

SparkSnail and others added 12 commits June 30, 2020 16:10

fix comments

b341dce

Merge remote-tracking branch 'snail/dev-aml' into 2391-improve

ce81c51

fix comments

e66dc23

merge code logic

9cf6744

Merge remote-tracking branch 'snail/dev-aml' into 2391-improve

bd77f5c

Merge branch 'master' of https://github.com/microsoft/nni into dev-aml

ddfb0cc

Merge pull request #257 from microsoft/master

65660e6

merge master

Merge branch 'master' of https://github.com/SparkSnail/nni into dev-aml

fec8a67

Merge remote-tracking branch 'snail/dev-aml' into 2391-improve

5200a3a

fix eslint errors

51befa5

add run fo messages

478629f

Merge pull request #256 from squirrelsc/2391-improve

c299ce1

Some small fix and merge logic

chicm-ms approved these changes Jun 30, 2020

View reviewed changes

SparkSnail added 5 commits June 30, 2020 18:26

fix comments

0517e13

sort class

fc4b978

fix eslint

e527743

fix eslint

b047681

fix annotation

4acc7e8

ultmaster reviewed Jun 30, 2020

View reviewed changes

SparkSnail added 5 commits July 1, 2020 07:46

fix import aml

ec1475a

fix comments

8eaeebf

fix doc build

56b6818

fix trial_runner import

e09ff79

fix doc

ecf615d

squirrelsc approved these changes Jul 1, 2020

View reviewed changes

fix pylint

a7a3baf

SparkSnail merged commit 93f96d4 into microsoft:master Jul 1, 2020

liuzhe-lz mentioned this pull request Jul 13, 2020

Fix yarn.lock #2682

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support aml #2615

Support aml #2615

SparkSnail commented Jun 30, 2020 •

edited

Loading

ultmaster Jun 30, 2020

SparkSnail Jun 30, 2020

ultmaster Jun 30, 2020

SparkSnail Jun 30, 2020

ultmaster Jun 30, 2020

SparkSnail Jul 1, 2020

ultmaster Jun 30, 2020

SparkSnail Jun 30, 2020

ultmaster Jun 30, 2020

SparkSnail Jun 30, 2020

ultmaster Jun 30, 2020

SparkSnail Jun 30, 2020

squirrelsc Jun 30, 2020

squirrelsc Jul 1, 2020

Support aml #2615

Support aml #2615

Conversation

SparkSnail commented Jun 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkSnail commented Jun 30, 2020 •

edited

Loading