Eliminating restrictions on builds(..., zen_partial=<...>, builds_bases=(<...>))

[rsokl]

There are currently two restrictions that hydra-zen imposes on builds(..., zen_partial=<...>, builds_bases=(<...>)):

1. Inheriting from a partial-builds parent

Inheriting from a parent where zen_partial=True requires that all children explicitly specify zen_partial=True.

parent_with_partial = builds(int, zen_partial=True)

child = builds(int, builds_bases=(parent_with_partial,)) # Not allowed
child = builds(int, zen_partial=False, builds_bases=(parent_with_partial,)) # Not allowed

child = builds(int, zen_partial=True, builds_bases=(parent_with_partial,)) # Okay; zen_partial=True for child

This restriction is due to the fact that the default argument for zen_partial is False. Thus there is presently no way for us to distinguish between a user leaving zen_partial unspecified and explicitly specifying that they want zen_partial=False for the child.

Solution

We will update the default of zen_partial to be None so that we can distinguish between zen_partial being left un-specified and it being set to False explicitly. With this change in place, the following behaviors will manifest:

parent_with_partial = builds(int, zen_partial=True)

child = builds(int, builds_bases=(parent_with_partial,)) # zen_partial=True for child (via inheritance)

child = builds(int, zen_partial=None, builds_bases=(parent_with_partial,)) # zen_partial=True for child (via inheritance)

child = builds(int, zen_partial=False, builds_bases=(parent_with_partial,)) # zen_partial=False for child

child = builds(int, zen_partial=True, builds_bases=(parent_with_partial,)) # zen_partial=True for child

2. Mixing _partial_=True with zen-processing features

Hydra's added support for partial instantiation made possible the formation of configs like:

_target_: hydra_zen.funcs.zen_processing
_partial_: true
_zen_target: SomeTarget

where instantiating this config will produce:

functools.partial(<function zen_processing at 0x0000024DA072FD30>, _zen_target=SomeTarget)

this is not desirable as the .func attribute of this partial'd object should return SomeTarget, it will instead return hydra_zen.funcs.zen_processing

Even more insidious is the case where we have:

_target_: hydra_zen.funcs.zen_processing
_partial_: true
_zen_target: SomeTarget
_zen_partial: true

Here, instantiating this config will produce:

>>> out = instantiate(conf)
>>> out
functools.partial(<function zen_processing at 0x0000024DA072FD30>, _zen_target=SomeTarget, zen_partial=True)
>>> out()  # uh oh!
functools.partial(SomeTarget)
>>> out()()  # it was double partial'd!
SomeTarget

I.e. the presence of _partial_=true and _zen_partial=true means that the target will be double partial'd. For these reasons hydra-zen currently prohibits inheriting from a parent that uses zen-processing features into a child that is: a partial-builds and does not itself use zen-processing features.

Specifically this leads to the following restrictions:

P = builds(int, zen_partial=True)  # has _partial_ = True
builds(int, zen_partial=true_or_false, builds_bases=(P,), zen_meta=dict(a=1))  # Not allowed

This is quite unintuitive for users, and leads to issues like #288 where a natural inheritance hierarchy with a partial'd base gets blocked.

Solution

We can detect when a child is both inheriting _partial_=True and is using zen_processing, and we can overwrite _partial_=False and _zen_partial=True so that only zen_processing is responsible for performing the partial instantiation.

Thus we would have:

P = builds(int, zen_partial=True)  # has _partial_ = True
builds(int, zen_partial=true_or_false, builds_bases=(P,), zen_meta=dict(a=1))  # has _partial_ = False, zen_partial=True

Some sharp edges:

If we end up with configs like:

_target_: hydra_zen.funcs.zen_processing
_zen_target: builtins.int
_partial_: false
_zen_partial: true

users may coerce their builds into being partial-builds by checking for / setting _partial_=True. This would be a mistake if _zen_partial=True -- it can lead to the "double partial" scenario laid out above. There are a few things that we might do to reduce the risks of this:

• Provide a public is_partial_builds function that will check both _partial_ and zen_partial for users. This could either warn or raise in the case when both are set to True
• Provide a validate_partial_builds function that will raise when it sees bad combinations of _partial_ and zen-processing