Implement enas-mode and oneshot-mode for NAS interface

[Crysple]

Related issue #1159, used to run ENAS

API Changes

In the new version of NAS interface, users need to specify the NAS mode in their Config.yml for NNI under the trial section. Like this:

trial:
  command: sh ./macro_cifar10.sh
  codeDir: ./
  gpuNum: 1
+ #choice: classic_mode, enas_mode, oneshot_mode
+ nasMode: oneshot_mode

Currently there are three modes available:

• classic_mode: The same as the original version of NAS interface
• enas_mode: An efficient mode developed for tensorflow to run ENAS (introduced in the following)
• oneshot_mode: A mode for users to train a full graph of a search space, where all the inputs and operations will be chosen and summed together

Different modes

enas_mode

Overview

In tensorflow, users need to build graph first and then create a session to run that graph. In the previous version, the graph of a trial will be determined (to be a sub-graph) once it receives a parameter configuration from the tuner. That is to say, the graph of this trial will not change even if it receives other parameter configurations in the future.

So in this mode we will create and use tensorflow variable as signals, and tensorflow conditional functions to control the search space (full-graph) to be more flexible, which means it can be changed into different sub-graphs (multiple times) depending on these signals.

API Changes

Users need to add nni.get_next_parameter(session) before they invoke the session.run function:

Note that they need to pass their tensorflow session as an arg into this function. An example might be:

for _ in range(num):
    """@nni.get_next_parameter(self.session)"""
    loss, _ = self.session.run([loss_op, train_op])
    """@nni.report_final_result(loss)"""

oneshot-mode

We provide a mode for users to train oneshot model.

In this mode, every inputs and operations will be chosen. As suggested in the paper, a dropout method is implemented to the inputs for every layer. The dropout rate is set to r^(1/k), where 0 < r < 1 is a hyper-parameter of the model (default to 0.01) and k is number of optional inputs for a specific layer. The higher the fan-in, the more likely each possible input is to be dropped out. However, the probability of dropping out all optional_inputs of a layer is kept constant regardless of its fan-in. Suppose r = 0.05. If a layer has k = 2 optional_inputs then each one will independently be dropped out with probability 0.051/2 ≈ 0.22 and will be retained with probability 0.78. If a layer has k = 7 optional_inputs then each one will independently be dropped out with probability 0.051/7 ≈ 0.65 and will be retained with probability 0.35. In both cases, the probability of dropping out all of the layer's optional_inputs is 5%.

[QuanluZhang]

please pass pylint