[V2 Loggers] config file

[horheynm]

Config for validation

Description

Config yaml file has five top level properties: version, logging, system, performance and metric.
version is always 2, since its for v2 pipeline
logging is the name of the defined logging (class LoggingFoo: here the name is LoggingFoo) which are singletons (one instance per any Name).
Sys, Perf and Met are the three entry points to use the Logger.
System is for default python (import logging) logging
Performance is for logging any hardware, benchmark,..etc metric-related log (thing that are already defined).
Metric is for input related logging (output value based on inputs).

If logging has any args for instantiation, one can define as

loggers:
  logger_id:
    name: FooLogger
    arg1: 1
    arg2: 2

For the three entry points, the top level after the root logger (system, metric, performance) is called the tag. Tags are used to filter out and log in any matching tag by regex.
Under tags, we have three (four in metrics) properties,
func is the function to apply to any value of interest to log.
freq is the rate to log once a matching entry is encountered.
uses: is a list to specify which logger to use, selected by logger_id

capture (metric only): key of the dict, property name to extract from the output of an operator, ran in logging middleware

system: # Python/system level logging
  ".*": # reges tag to capture
  - func: identity # func name or path/to/file.py:func_name
    freq: 1 # rate to log
    uses: # which logger to use, point to the log id above
    - default 
metric:
  ".*":
  - func: max
    freq: 1
    uses:
    - list
    capture:  # The key, property to log. Ex rtn: Operator = AddOneOperator(...), if capture is set to value, it would record rtn.capture, if tag above is set to any regex that recognizes AddOneOperator (currently set to ".*")
    - ".*" # log every key, property that the dict, class has

In other words, for each root logger, what tags are we logging. With respect to tags, what function are we applying if any, at what rate and with respect to what loggers are we using

Root logger is a function of tags, tags are a fucntion of frequency, func and loggers.
Root logger(tag(freq, func, loggers, capture))

Examples

version: 2 # v2 uses 2

loggers:
  list: # Logger ID
    name: ListLogger # class name or path/to/file.py:ClassName
    handler: None # Handler config, default set to default logging 
  default: 
    name: PythonLogger 
    handler: None
system: # Python/system level logging
  ".*": # reges tag to capture
  - func: identity # func name or path/to/file.py:func_name
    freq: 1 # rate to log
    uses: # which logger to use, point to the log id above
    - default 
performance:
  cpu:
  - func: identity
    freq: 1
    uses:
    - default
metric:
  ".*":
  - func: max
    freq: 1
    uses:
    - list
    capture:  # The key, property to log. Ex rtn: Operator = AddOneOperator(...), if capture is set to value, it would record rtn.capture, if tag above is set to any regex that recognizes AddOneOperator (currently set to ".*")
    - ".*"

Default config without any inputs

version: 2
loggers:
  list:
    name: ListLogger
    handler: None
  default:
    name: PythonLogger
    handler: None
system:
  ".*":
  - func: identity
    freq: 1
    uses:
    - default
performance:
  cpu:
  - func: identity
    freq: 1
    uses:
    - default
metric:
  ".*":
  - func: max
    freq: 1
    uses:
    - list
    capture:
    - ".*"

TODO: Handlers

[bfineran]

lgtm pending comments