Skip to content

YAML settings file reference

Jump to bottom
Szymon Maszke edited this page Apr 13, 2020 · 6 revisions

This page contains description for each field of YAML generated by subcommand settings:

$ torchlambda settings

Below is a generated template settings with every field with some example values:

---
grad: false
validate_json: true
model: /opt/model.ptc
input:
  name: data
  validate: true
  type: base64
  shape: [1, 3, width, height]
  validate_shape: true
  cast: float
  divide: 255
normalize:
  means: [0.485, 0.456, 0.406]
  stddevs: [0.229, 0.224, 0.225]
return:
  output:
    type: double
    name: output
    item: false
  result:
    type: int
    name: result
    operations: argmax
    arguments: 1
    item: true

Please note above are not default values (e.g. when those are left unspecified). Whether those are required or have some defaults is described in section about each field separately!

Fields descriptions below have their specific properties at the end:

  • type: type of value
  • required: whether this field and value is required in order to create C++ code. If false, the whole option will be turned off. Mutually exclusive with default
  • default: whether there is a default value for this field. If this property exists user doesn't have to provide any value. Mutually exclusive with required

For exact description of each field see below:

Table Of Fields

grad

If false disables PyTorch's autograd engine. It is almost always safe to set this value to false as it is not needed for inference. If you need it for some reason, set field below to true.

Properties:

  • type: bool
  • default: false

validate_json

If true check whether JSON passed as request was parsed successfully. If it wasn't, InvalidJSON will be sent as a response containing string: Failed to parse request JSON file.

Properties:

  • type: bool
  • default: true

optimize

torchscript compiled networks, during initial pass, use graph optimization unless instructed otherwise. This may lead to large slowdowns (of 100x or even more in case of AWS Lambda) and is turned off by default. Specify True to optimize initial pass anyway. See this issue for discussion.

Properties:

  • type: bool
  • default: false

validate_json

If true check whether JSON passed as request was parsed successfully. If it wasn't, InvalidJSON will be sent as a response containing string: Failed to parse request JSON file.

Properties:

  • type: bool
  • default: bool

model

Path to uploaded model within Lambda's filesystem. If model is uploaded as AWS Lambda Layer it will be unpacked to /opt and you should take it into consideration.

Properties:

  • type: str
  • default: /opt/model.ptc

input

This nested field defines how input to your model will look like. Example can be seen below:

input:
  type: float
  validate: true
  shape: [1, 50, timesteps]
  validate_shape: false

Properties:

  • type: nested field
  • required: true

For exact description of each field see below

name

Name of the JSON field which is necessary to pass during request and contains data (for example image).

Properties:

  • type: non-empty str
  • default: data

type

What is the type of input. It is assumed input is always an array (or base64 encoded string) of one of the types:

  • byte (C++ unsigned char)
  • char
  • short
  • int
  • long
  • float
  • double
  • base64 (special type)

If base64 is specified as type, request should containg a JSON file with base64 encoded list as field named like name containing byte-like values (or unsigned char [0-255] range). This mode should be used for any image tensors (which should be flattened before sending).

Also tensor of specified type will be created. If you wish to input neural network with different type (e.g. float16 a.k.a. half) use cast property which allows user to change tensor type to any supported by libtorch (at least for version `1.4.0, post a PR if new types emerge).

Properties:

  • type: str (one of specified above)
  • required: true

validate

If true check whether field name was passed in JSON request. If it wasn't InvalidJSON will be sent as a response containing string: Required field <name_of_field> was not provided.

Also type of data will be verified (whether it's an arrayorstrin case ofbase64` type)

Properties:

  • type: bool
  • default: true

shape

Shape of tensor passed in name JSON field. Should be provided as list containing either int values or str values or both. For integers, shape will be hardcoded (for example 3 for three input channels). If str provided field with this name will be searched for in request and used as one of tensor dimensions.

Properties:

  • type: List[str|int]
  • required: true

validate_shape

If true check whether all str values from shape are passed in the request as JSON fields.

  • If any field is missing InvalidJSON will be sent as a response containing string: Required input shape field: '<name_of_field>' was not provided
  • If field exists but isn't of int type InvalidJSON will be sent as a response containing string: Required input shape field: '<name_of_field>' is not of integer type.

If all shape is static (e.g. only int values are specified) this check will not be performed.

Properties:

  • type: bool
  • default: true

cast

Type to which provided tensor can be casted after it is created from array with specified type. Usually floating point types (half, float, double) should be used.

This field allows you to use half-precision inference if your network uses float16

One of those values allowed:

  • byte
  • char
  • short
  • int
  • long
  • half
  • float
  • double

Properties:

  • type: string
  • required: false (tensor will not be casted)

divide

Value by which tensor will be divided after being casted.

255 is set as default because it will put [0, 255] range tensor after encoding into [0, 1] range (which is usually used for image predictions).

Properties:

  • type: number
  • required: false (tensor will not be divided)

normalize

If this nested field is defined tensor obtained from data will be normalized across channels using specified means and standard deviations.

⚠️ Under the hood uses torch::data::transforms::Normalize which requires means and stddevs to be any shape broadcastable to input tensor.

Example field (with ImageNet means and standard deviations):

normalize:
  means: [0.485, 0.456, 0.406]
  stddevs: [0.229, 0.224, 0.225]

Properties:

  • type: nested field
  • required: false (no normalization will be applied)

For exact description of each field see below

means

Mean values used for normalization.

Properties:

  • required: true (if normalize specified)
  • type: list[number]

stddevs

Standard deviations used for normalization.

Properties:

  • required: true (if normalize specified)
  • type: list[number]

return

This nested fields specifies what and how should be returned by AWS Lambda

It is required for user to specify at least one of the fields below:

  • output
  • result

⚠️ if output and result specified both will be returned in JSON response!

Properties:

  • type: nested field
  • required: true

For exact description of each field see below

output

If this field is specified output taken directly from neural network will be returned as JSON field. This field has additional properties user can specify in order to define what is returned.

Example field (including top level return):

return:
  output:
    name: output
    type: double
    item: true

For exact description of each field see below

type

Data type to be returned.

One of those values allowed:

  • int
  • long
  • double

Properties:

  • required: true
  • type: str

name

Key (name) of field where the output will be placed in returned JSON.

Properties:

  • type: str
  • default: output

item

If true, output will be returned as a single item.

⚠️ output HAS TO BE SINGLE ITEM otherwise you will get silent error

Tensor will be flattened and .item<type>() will be applied on it. If your output is not a single value leave this with default false value.

Properties:

  • type: bool
  • default: false

result

result is created by some tensor operations applied on neural network output. For example, you can apply argmax operation on outputted logits along first dimension to get corresponding labels. You can describe those steps via .yaml setting like this:

return:
  result:
    operations: argmax
    arguments: 1
    name: labels
    type: long
    item: true

⚠️ you don't have to specify output for result to work!

For exact description of each field see below

operations

Name of operation described with str. To see available options please refer to libtorch documentation. All functions in at:: namespace are available but they have to be specified without at:: namespace.

Some available options could be: softmax, argmax, sigmoid.

Multiple consecutive operations can be applied if list is used.

In the example below first sigmoid will be applied on output and after that argmax to produce result:

return:
  result:
    operations: [sigmoid, argmax]
	...

Properties:

  • required: true
  • type: str or list[str]

arguments

Additional arguments beside output to be passed to operations (one or more). Can be specified as single argument of any type that will be passed as argument to first operation.

Example below will pass 1 as a dimension to softmax and nothing to argmax:

return:
  result:
    operations: [softmax, argmax]
    arguments: 1
	...

Example below will pass 1 as a dimension to second argument argmax and nothing to sigmoid:

return:
  result:
    operations: [sigmoid, argmax]
    arguments: [[], 1]
	...

You could also pass multiple arguments to each function, just place them in an appropriate list

Properties:

  • default: None (no additional arguments beside output will be passed to operations
  • type: any or list[any] or list[list[any] | any]

type

Data type to be returned.

One of those values allowed:

  • int
  • long
  • double

Properties:

  • required: true
  • type: str

name

Key (name) of field where the output will be placed in returned JSON.

Properties:

  • type: str
  • default: output

item

If true, output will be returned as a single item.

⚠️ output HAS TO BE SINGLE ITEM otherwise you will get silent error

Tensor will be flattened and .item<type>() will be applied on it. If your output is not a single value leave this with default false value.

Properties:

  • type: bool
  • default: false
Clone this wiki locally