Convert ListConfig objects with no structure to list type within DictConfig?

[LarsHill]

Hi,

Using omegaconf as a wrapper around config dicts led to a few problems with respect to the ListConfig type. Some well known external libraries, i.e. huggingface's transformers check specifically for the inbuild list type for certain function inputs.

For example, when adding additional special tokens to a Tokenizer object, these special tokens have to be of type list. Since I register these in a config, the resulting type within the parsed DictConfig object is ListConfig (does not inherit from list) and the transformer library raises an error.

I saw a related issue wrt PyTorch Lighting and the answer was that it is not recommended to inherit from primitive containers and that it is the responsibility of external libraries to check for e.g. the MutualSequence type instead of the inbuild list type. Unfortunately many libraries don't do that.

An other problem I encountered was the following:
In my config I register a list of hidden layer dimensions for a simple MLP model, e.g.

hidden_dims: [512, 256]

In my model I have to add the input and output dimensions (data and task dependent) to this object. However a simple
hidden_dims = [128] + hidden_dims + [2]
does not work since hidden_dims is of type ListConfig and it cannot be added to a list type object.

from omegaconf import DictConfig
d = {'a': 1, 'b': [1,2,3]}
d_cfg = DictConfig(d)
hidden_dims = [10] + d_cfg.b + [1]
=> TypeError: can only concatenate list (not "ListConfig") to list

Since I don't want to manually convert normal list type hyper params to the inbuild list types every time some error is raised, I look for a way to convert all simple ListConfig objects that don't contain a DictConfig structure to the normal list type.

Hence, I tried this simple test, but got:

from omegaconf import DictConfig

d = {'a': 1, 'b': [1, 2, 3]}
d_cfg = DictConfig(d)
print(type(d_cfg.b))
=> ListConfig

d_cfg.b = list(d_cfg.b)
print(type(d_cfg.b))
=> ListConfig

In the second print statement I expected to see type list, but apperantly the list object is wrapped dynamically to a ListConfig object when it is newly assigned.

So is there any other way of avoiding the ListConfig type for e.g. lists that don't have any further nesting?

I think an option like this would be great because it would allow the user to use all the benefits of DictConfig but he can expect that his list hyperparameters don't cause any problems in external libraries or with operations that work with the list type but not with the ListConfig type.

Thanks for your help and answers :)

[Jasha10]

The OmegaConf container types (ListConfig and DictConfig) quack like lists and dicts, respectively. However, as you pointed out, many libraries rely on python-native lists and dicts, and do not play as well with MutableSequences and MutableMappings.

For these situations, I'd recommend calling OmegaConf.to_object to recursively convert your OmegaConf object to a plain python object.

>>> d = {'a': 1, 'b': [1, 2, 3]}
>>> d_cfg = DictConfig(d)
>>> d_obj = OmegaConf.to_object(d_cfg)
>>> print(type(d_obj))
<class 'dict'>
>>> print(type(d_obj["b"]))
<class 'list'>

---

In my model I have to add the input and output dimensions (data and task dependent) to this object. However a simple hidden_dims = [128] + hidden_dims + [2] does not work since hidden_dims is of type ListConfig and it cannot be added to a list type object.

I have just opened up a feature request (#849) to address this.
Once implemented, the sum [128] + hidden_dims + [2] will result in a ListConfig (not a list).

---

In the second print statement I expected to see type list, but apperantly the list object is wrapped dynamically to a ListConfig object when it is newly assigned.
So is there any other way of avoiding the ListConfig type for e.g. lists that don't have any further nesting?

It's not possible to do this with OmegaConf currently. If you want a list (not a ListConfig), just use OmegaConf.to_object.

The motivation for converting lists to ListConfigs is so that extra "metadata" can be tracked (beyond what a plain list would be able to do). Examples of this metadata include typing information (enabling runtime type-checking for structured configs) and keeping track of the parent container (which is necessary so that interpolations to a parent node will work).

In practice, I find that OmegaConf is most useful during the "configuration" phase; often, practitioners end up converting "back to plain python" when it's time to run the business logic.

[Jasha10]

You overwrite the default items() in OmegaConf, probably due to some metadata handling. Is it really necessary though?

It might be possible to use MutableMapping's items method, but it would require major refactoring. It is something that probably would not happen for OmegaConf v2 due to backwards compatibility issues.

[Jasha10]

OmegaConf enables many use-cases (e.g. Hydra, which uses OmegaConf as a backend) besides just the __getattr__ lookup for attributes.

If you need perfect compatibility with dict, it will probably be easier for you to define your own subclass of MutableMapping.

[LarsHill]

In case others are reading this discussion:

I indeed created my own subclass which behaves exactly as a dict with the addition of enabling (nested) attribute-style key access/assignment (dot notation) and autocompletion.

I named it metadict and it is available via pip.

[Jasha10]

Cool!

[Jasha10]

FYI in PR #863 we changed the DictConfig.keys method to return an object of type KeysView instance, and similary DictConfig.items now returns an ItemsView.