add nnictl command to list trial results with highest/lowest metric #2747

tabVersion · 2020-07-30T07:13:45Z

nnictl experiment head
nnictl experiment tail

SparkSnail · 2020-07-31T02:36:34Z

What is the user scenario of this command? How do you handle dict metric?

chicm-ms · 2020-07-31T02:49:54Z

tools/nni_cmd/nnictl_utils.py

+    else:
+        print_error('Restful server is not Running')
+
+def experiment_tail(args):


experiment_tail and experiment_head can be implemented with one function, use the sort order as parameter.

chicm-ms · 2020-07-31T02:50:40Z

tools/nni_cmd/nnictl_utils.py

+            if int(args.num) > len(content):
+                print_error('Required number is too large.')
+                return
+            content = sorted(content, key=lambda x: float(x['data'][2: -2]), reverse=True)


If the metric is of dict type, use 'default' key to sort.

tabVersion · 2020-07-31T02:51:56Z

It is for users who can not access webui on the server. I saw the demand in the NNI group in WeChat.
The Dict metric is not handled currently. I use the metric-data-latest API, which only contains data of default metric.

chicm-ms · 2020-07-31T02:53:17Z

tools/nni_cmd/nnictl.py

+                                                                                  'n trials with the highest metric')
+    parser_experiment_head.add_argument('id', nargs='?', help='the id of experiment')
+    parser_experiment_head.add_argument('--num', default=10, help='the number of items to be listed')
+    parser_experiment_head.set_defaults(func=experiment_head)


Can nnictl trial ls with sorting do the same work?

Users usually want to focus on top maximal/minimal trials. I think nnictl trials ls offer too much info and users cannot clearly see the overall situation of the experiment.
I want to provide an interface to the general situation of the experiment just like ↓.

OK, got it, suggest to reuse nnictl trial command since we are listing the trials.
Consider following options :

add options to nnictl trial ls, such as nnictl trial ls -n 10 --sort asc --detail

nnictl trial head 10 , nnictl trial tail 10

The latter looks good to me. How about nnictl trial head 10 --reverse True instead of nnictl trial tail 10.

head 10 --reverse maybe misunderstood as reverse sorting the the top 10, not bottom 10. The meaning of tail 10 is clear.

tabVersion · 2020-07-31T10:50:01Z

It can handle dict metric by keeping only the default part.
For example, if give {'acc': 0.5, 'default': 0.7}, this command will sort on key default and only show default part.
Here is an instance:

$nnictl trial head Xhc1YImB -n 5 -r
INFO:  ----------------------------------------------------------------------------------------
                Experiment Top Minimal Trials
Trial id: eZ5ci         Default metric: 93.64
Trial id: bRVNu         Default metric: 94.24
Trial id: MkYEz         Default metric: 95.36
Trial id: EiL7P         Default metric: 95.62
Trial id: FtMIJ         Default metric: 96.49

----------------------------------------------------------------------------------------

tabVersion · 2020-08-05T06:18:30Z

I have a concern to use eval when dealing with serialized data because eval is not safe.

tools/nni_cmd/nnictl_utils.py

QuanluZhang · 2020-08-10T11:27:44Z

docs/en_US/Tutorial/Nnictl.md

+
+  Note that if `--num` exceeds the number of trials, all trial results are listed.
+
+* __nnictl trial tail__


suggest to merge tail and head into nnictl trial ls. what do you think? @chicm-ms @SparkSnail

agree, nnictl trial ls show all of trial results, and this nnictl trial tail filter trial results, we can merge them to nnictl trial ls --tail {number} to simplify nnictl.

My concern is that nnictl trial ls shows too much information and it is displayed in JSON format. Users cannot see the desired results at first glance even if trial results are sorted. Can we show the information in tables?

@tabVersion i don't get. what do you mean?

tools/nni_cmd/nnictl.py

tools/nni_cmd/nnictl_utils.py

docs/en_US/Tutorial/Nnictl.md

QuanluZhang · 2020-08-12T06:46:42Z

tools/nni_cmd/nnictl.py

@@ -102,6 +102,8 @@ def parse_args():
    parser_trial_subparsers = parser_trial.add_subparsers()
    parser_trial_ls = parser_trial_subparsers.add_parser('ls', help='list trial jobs')
    parser_trial_ls.add_argument('id', nargs='?', help='the id of experiment')
+    parser_trial_ls.add_argument('--head', type=int)
+    parser_trial_ls.add_argument('--tail', type=int)


please follow other commands' coding style

QuanluZhang · 2020-08-12T07:35:19Z

tools/nni_cmd/nnictl_utils.py

@@ -259,6 +274,15 @@ def trial_ls(args):
        response = rest_get(trial_jobs_url(rest_port), REST_TIME_OUT)
        if response and check_response(response):
            content = json.loads(response.text)
+            if args.head:
+                assert int(args.head) > 0, 'The number of requested data must be greater than 0.'
+                args.head = min(int(args.head), len(content))


why use int(args.head)?

Sorry, I don't know if specify the arg's type as int, it will be converted to int.

QuanluZhang · 2020-08-12T07:35:34Z

tools/nni_cmd/nnictl_utils.py

+                args.head = min(int(args.head), len(content))
+                content = sorted(filter(lambda x: 'finalMetricData' in x, content),
+                                 key=cmp_to_key(final_metric_data_cmp), reverse=True)[:args.head]


this logic is not correct

QuanluZhang · 2020-08-12T07:35:53Z

tools/nni_cmd/nnictl_utils.py

+            elif args.tail:
+                assert int(args.tail) > 0, 'The number of requested data must be greater than 0.'
+                args.tail = min(int(args.tail), len(content))
+                content = sorted(content, key=cmp_to_key(final_metric_data_cmp))[:args.tail]


no filter here

tools/nni_cmd/nnictl_utils.py

…icrosoft#2747) (cherry picked from commit 44954e0)

tabVersion added 2 commits July 30, 2020 15:12

update

1e920f1

code format

4f13233

tabVersion closed this Jul 30, 2020

tabVersion reopened this Jul 30, 2020

QuanluZhang requested review from SparkSnail and ultmaster July 30, 2020 15:33

chicm-ms reviewed Jul 31, 2020

View reviewed changes

interface move to -> nnictl trial head

d658a93

tabVersion added 2 commits July 31, 2020 18:58

patch

5fb456d

add nnictl trial tail

822189f

chicm-ms approved these changes Aug 5, 2020

View reviewed changes

SparkSnail approved these changes Aug 6, 2020

View reviewed changes

SparkSnail reviewed Aug 6, 2020

View reviewed changes

tools/nni_cmd/nnictl_utils.py Outdated Show resolved Hide resolved

Update nnictl_utils.py

ed3d826

tabVersion closed this Aug 6, 2020

tabVersion reopened this Aug 6, 2020

tabVersion closed this Aug 7, 2020

tabVersion reopened this Aug 7, 2020

QuanluZhang reviewed Aug 10, 2020

View reviewed changes

move to -> nnictl trial ls

d447e3b

QuanluZhang reviewed Aug 11, 2020

View reviewed changes

tools/nni_cmd/nnictl.py Outdated Show resolved Hide resolved

QuanluZhang reviewed Aug 11, 2020

View reviewed changes

tools/nni_cmd/nnictl_utils.py Show resolved Hide resolved

QuanluZhang reviewed Aug 11, 2020

View reviewed changes

tools/nni_cmd/nnictl_utils.py Show resolved Hide resolved

ultmaster mentioned this pull request Aug 12, 2020

July iteration plan #2608

Closed

66 tasks

tabVersion added 2 commits August 12, 2020 11:51

Update Nnictl.md

39879c8

Update nnictl_utils.py

f87334a

QuanluZhang reviewed Aug 12, 2020

View reviewed changes

docs/en_US/Tutorial/Nnictl.md Show resolved Hide resolved

QuanluZhang reviewed Aug 12, 2020

View reviewed changes

tabVersion added 3 commits August 12, 2020 15:25

Merge branch 'master' into exp-head-tail2

34f576c

Update Nnictl.md

799b6d3

Update nnictl.py

d7011ac

QuanluZhang reviewed Aug 12, 2020

View reviewed changes

tabVersion added 3 commits August 12, 2020 16:34

Update nnictl_utils.py

33a7c1f

Update Nnictl.md

eecde16

Update nnictl_utils.py

b1e7575

ultmaster approved these changes Aug 12, 2020

View reviewed changes

QuanluZhang reviewed Aug 12, 2020

View reviewed changes

tools/nni_cmd/nnictl_utils.py Show resolved Hide resolved

QuanluZhang approved these changes Aug 12, 2020

View reviewed changes

QuanluZhang merged commit 44954e0 into microsoft:master Aug 12, 2020

LovPe pushed a commit to LovPe/nni that referenced this pull request Aug 17, 2020

add nnictl command to list trial results with highest/lowest metric (m…

c313248

…icrosoft#2747) (cherry picked from commit 44954e0)

autodevreviewdevelopment bot mentioned this pull request Nov 3, 2021

Update helloworld.py SahilPatnayakuni/Test2#3

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add nnictl command to list trial results with highest/lowest metric #2747

add nnictl command to list trial results with highest/lowest metric #2747

tabVersion commented Jul 30, 2020

SparkSnail commented Jul 31, 2020

chicm-ms Jul 31, 2020

chicm-ms Jul 31, 2020

tabVersion commented Jul 31, 2020

chicm-ms Jul 31, 2020

tabVersion Jul 31, 2020

chicm-ms Jul 31, 2020 •

edited

Loading

tabVersion Jul 31, 2020

chicm-ms Aug 5, 2020

tabVersion commented Jul 31, 2020 •

edited

Loading

tabVersion commented Aug 5, 2020

QuanluZhang Aug 10, 2020

SparkSnail Aug 11, 2020

tabVersion Aug 11, 2020

QuanluZhang Aug 11, 2020

QuanluZhang Aug 12, 2020

tabVersion Aug 12, 2020

QuanluZhang Aug 12, 2020

tabVersion Aug 12, 2020

QuanluZhang Aug 12, 2020

tabVersion Aug 12, 2020

QuanluZhang Aug 12, 2020

tabVersion Aug 12, 2020


		Note that if `--num` exceeds the number of trials, all trial results are listed.

		* __nnictl trial tail__

add nnictl command to list trial results with highest/lowest metric #2747

add nnictl command to list trial results with highest/lowest metric #2747

Conversation

tabVersion commented Jul 30, 2020

SparkSnail commented Jul 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tabVersion commented Jul 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chicm-ms Jul 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tabVersion commented Jul 31, 2020 • edited Loading

tabVersion commented Aug 5, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chicm-ms Jul 31, 2020 •

edited

Loading

tabVersion commented Jul 31, 2020 •

edited

Loading