Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds @zen decorator for task functions #310

Merged
merged 35 commits into from
Oct 19, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
116c267
Improve zen tests
rsokl Sep 28, 2022
d84539b
Ensure interpolated fields are resolved
rsokl Sep 28, 2022
7e936a8
improve instantiate annotations to include list and dict
rsokl Sep 28, 2022
2b991ff
Support dict, list, and yaml-string inputs
rsokl Sep 28, 2022
19fab5a
patch coverage and add type-tests
rsokl Sep 28, 2022
57524c9
patch coverage
rsokl Sep 28, 2022
4e30888
add hydra_main capability
rsokl Sep 28, 2022
01af089
add missing file
rsokl Sep 28, 2022
448ebf8
skip cli test on windows
rsokl Sep 28, 2022
b8c2ce9
include interpolation in example
rsokl Sep 28, 2022
b907b92
fix env variable
rsokl Sep 28, 2022
70c2f54
fix env variable
rsokl Sep 28, 2022
001de5e
Update .github/workflows/nightly.yml
rsokl Sep 29, 2022
0bb8564
Enable zen_cfg passthrough and custom wrapping
rsokl Sep 29, 2022
42d77ec
improve ZenWrapper test
rsokl Sep 29, 2022
373f85f
improve annotation tests
rsokl Sep 29, 2022
5d07554
add pre-call validation
rsokl Sep 29, 2022
b53422d
coverage
rsokl Sep 29, 2022
340c1c2
move pre-call validation to init phase
rsokl Sep 29, 2022
6dd8128
add to-do notes
rsokl Sep 29, 2022
9863e7b
test zen with pickling launcher
jgbos Oct 5, 2022
c6fa404
fixes ignore glob
jgbos Oct 5, 2022
66c0dd2
fix type completeness
rsokl Oct 5, 2022
9b6f731
Fix submitit-launcher test being skipped always
rsokl Oct 5, 2022
abac302
include pre-call in hydra_main test
rsokl Oct 5, 2022
519462f
progress on docs
rsokl Oct 8, 2022
0ed67ef
progress on docs
rsokl Oct 8, 2022
167a51a
more docs
rsokl Oct 11, 2022
f28cb25
merge
Oct 14, 2022
bbf2876
fix windows ci regression caused by merge
Oct 14, 2022
cc72443
add exclude argument and improve docs
rsokl Oct 18, 2022
edefdf6
add Zen to ref api docs
rsokl Oct 18, 2022
eb329a4
test nested interpolation
rsokl Oct 19, 2022
2313218
add kw-extraction and update changelog
rsokl Oct 19, 2022
a5d8b8f
Merge branch 'main' into add-zen
rsokl Oct 19, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions docs/source/api_reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,9 @@ Encyclopedia Hydrazennica.
All reference documentation includes detailed Examples sections. Please scroll to the
bottom of any given reference page to see the examples.

*************************
Launching Jobs with Hydra
*************************
**************************************
Creating and Launching Jobs with Hydra
**************************************

hydra-zen provides users the ability to launch a Hydra job via a
Python function instead of from a commandline interface.
Expand All @@ -25,7 +25,8 @@ Python function instead of from a commandline interface.
:toctree: generated/

launch

zen
wrapper.Zen

*********************************
Creating and Working with Configs
Expand Down
60 changes: 60 additions & 0 deletions docs/source/changes.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,66 @@ chronological order. All previous releases should still be available on pip.

.. _v0.8.0:

---------------------
0.9.0rc2 - 2022-10-19
---------------------

.. note:: This is documentation for an unreleased version of hydra-zen. You can try out this version pre-release version using `pip install --pre hydra-zen`


Release Highlights
------------------
This release adds the :func:`~hydra_zen.zen` decorator, which enables users to use Hydra-agnostic task functions for their Hydra app; the decorator will automatically extract, resolve, and instantiate fields from an input config based on the function's signature.

This encourages users to eliminate Hydra-specific boilerplate code from their projects and to instead opt for task functions with explicit signatures, including functions from third parties.

E.g., :func:`~hydra_zen.zen` enables us to replace the following Hydra-specific task function:

.. code-block:: python
:caption: The "old school" way of designing a task function for a Hydra app

import hydra
from hydra.utils import instantiate

@hydra.main(config_name="my_app", config_path=None, version_base="1.2")
def trainer_task_fn(cfg):
model = instantiate(cfg.model)
data = instantiate(cfg.data)
partial_optim = instantiate(cfg.partial_optim)
trainer = instantiate(cfg.trainer)

optim = partial_optim(model.parameters())
trainer(model, optim, data).fit(cfg.num_epochs)

if __name__ == "__main__":
trainer_task_fn()

with a Hydra-agnostic task function that has an explicit signature:

.. code-block:: python
:caption: Using `zen` to design a Hydra-agnostic task function

from hydra_zen import zen

@zen
def trainer_task_fn(model, data, partial_optim, trainer, num_epochs: int):
# All config-field extraction & instantiation is automated/mediated by zen
optim = partial_optim(model.parameters())
trainer(model, optim, data).fit(num_epochs)

if __name__ == "__main__":
trainer_task_fn.hydra_main(config_name="my_app", config_path=None)


There are plenty more bells and whistles to :func:`~hydra_zen.zen`, refer to :pull:`310` and its reference documentation for more details.

New Features
------------
- Adds the :func:`~hydra_zen.zen` decorator (see :pull:`310`)
- Adds the :func:`~hydra_zen.wrapper.Zen` decorator-class (see :pull:`310`)

.. _v0.8.0:

------------------
0.8.0 - 2022-09-13
------------------
Expand Down
22 changes: 22 additions & 0 deletions docs/source/generated/hydra_zen.wrapper.Zen.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
hydra\_zen.wrapper.Zen
======================

.. currentmodule:: hydra_zen.wrapper

.. autoclass:: Zen


.. automethod:: __init__
.. automethod:: __call__
.. automethod:: validate
.. automethod:: hydra_main


.. rubric:: Methods

.. autosummary::

~Zen.__init__
~Zen.__call__
~Zen.validate
~Zen.hydra_main
6 changes: 6 additions & 0 deletions docs/source/generated/hydra_zen.zen.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
hydra\_zen.zen
===============

.. currentmodule:: hydra_zen

.. autofunction:: zen
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,7 @@ python =


[testenv]
passenv = *
deps = pytest
hypothesis
commands = pytest \
Expand Down
2 changes: 2 additions & 0 deletions src/hydra_zen/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
)
from .structured_configs._implementations import get_target
from .structured_configs._type_guards import is_partial_builds, uses_zen_processing
from .wrapper import zen

__all__ = [
"builds",
Expand All @@ -39,6 +40,7 @@
"launch",
"is_partial_builds",
"uses_zen_processing",
"zen",
]

if not TYPE_CHECKING:
Expand Down
3 changes: 3 additions & 0 deletions src/hydra_zen/_compatibility.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ def _get_version(ver_str: str) -> Version:
OMEGACONF_VERSION: Final = _get_version(omegaconf.__version__)
HYDRA_VERSION: Final = _get_version(hydra.__version__)

SUPPORTS_VERSION_BASE = HYDRA_VERSION >= (1, 2, 0)

# OmegaConf issue 830 describes a bug associated with structured configs
# composed via inheritance, where the child's attribute is a default-factory
Expand Down Expand Up @@ -70,6 +71,8 @@ def _get_version(ver_str: str) -> Version:
range,
}

HYDRA_SUPPORTS_LIST_INSTANTIATION = HYDRA_VERSION >= Version(1, 1, 2)


if HYDRA_SUPPORTS_BYTES: # pragma: no cover
HYDRA_SUPPORTED_PRIMITIVES.add(bytes)
Expand Down
12 changes: 10 additions & 2 deletions src/hydra_zen/_hydra_overloads.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
import pathlib
from dataclasses import is_dataclass
from functools import wraps
from typing import IO, Any, Callable, Type, TypeVar, Union, cast, overload
from typing import IO, Any, Callable, Dict, List, Type, TypeVar, Union, cast, overload

from hydra.utils import instantiate as hydra_instantiate
from omegaconf import MISSING, DictConfig, ListConfig, OmegaConf
Expand Down Expand Up @@ -88,7 +88,15 @@ def instantiate(

@overload
def instantiate(
config: Union[HasTarget, ListConfig, DictConfig, DataClass_, Type[DataClass_]],
config: Union[
HasTarget,
ListConfig,
DictConfig,
DataClass_,
Type[DataClass_],
Dict[Any, Any],
List[Any],
],
*args: Any,
**kwargs: Any
) -> Any: # pragma: no cover
Expand Down
4 changes: 2 additions & 2 deletions src/hydra_zen/_launch.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
from hydra.types import HydraContext, RunMode
from omegaconf import DictConfig, ListConfig, OmegaConf

from hydra_zen._compatibility import HYDRA_VERSION
from hydra_zen._compatibility import SUPPORTS_VERSION_BASE
from hydra_zen._hydra_overloads import instantiate
from hydra_zen.typing._implementations import DataClass, InstOrType

Expand Down Expand Up @@ -238,7 +238,7 @@ def launch(
job_name=job_name,
**(
{}
if (HYDRA_VERSION < (1, 2, 0) or version_base is _NotSet)
if (not SUPPORTS_VERSION_BASE or version_base is _NotSet)
else {"version_base": version_base}
)
):
Expand Down
14 changes: 12 additions & 2 deletions src/hydra_zen/structured_configs/_type_guards.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,14 @@
# SPDX-License-Identifier: MIT
# pyright: strict
from functools import partial
from typing import Any
from typing import TYPE_CHECKING, Any, Type, Union

from typing_extensions import TypeGuard

from hydra_zen._compatibility import HYDRA_SUPPORTS_PARTIAL
from hydra_zen.funcs import get_obj, zen_processing
from hydra_zen.typing import Builds, Just, PartialBuilds
from hydra_zen.typing._implementations import DataClass_

from ._globals import (
JUST_FIELD_NAME,
Expand All @@ -19,7 +20,7 @@
ZEN_TARGET_FIELD_NAME,
)

__all__ = ["is_partial_builds", "uses_zen_processing"]
__all__ = ["is_partial_builds", "uses_zen_processing", "is_dataclass"]

# We need to check if things are Builds, Just, PartialBuilds to a higher
# fidelity than is provided by `isinstance(..., <Protocol>)`. I.e. we want to
Expand Down Expand Up @@ -53,6 +54,15 @@ def is_just(x: Any) -> TypeGuard[Just[Any]]:
return False


if TYPE_CHECKING: # pragma: no cover

def is_dataclass(obj: Any) -> TypeGuard[Union[DataClass_, Type[DataClass_]]]:
...

else:
from dataclasses import is_dataclass


def is_old_partial_builds(x: Any) -> bool: # pragma: no cover
# We don't care about coverage here.
# This will only be used in `get_target` and we'll be sure to cover that branch
Expand Down
8 changes: 8 additions & 0 deletions src/hydra_zen/wrapper/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Copyright (c) 2022 Massachusetts Institute of Technology
# SPDX-License-Identifier: MIT

# pyright: strict

from ._implementations import Zen, zen

__all__ = ["zen", "Zen"]
Loading