-
Notifications
You must be signed in to change notification settings - Fork 421
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for custom translators #412
Comments
You can register new translators easily enough. Notice at the bottom of that file we have a bunch of registration calls. https://papermill.readthedocs.io/en/latest/extending-overview.html describes how one can register new engines and IO plugins. But I noticed we're missing papermill/papermill/engines.py Lines 32 to 38 in 718f39e
from setuptools import setup, find_packages
setup(
# all the normal setup.py arguments...
entry_points={"papermill.translators": ["python=translators:PandasPythonTranslator"]},
) in your project to register the translations. |
@MSeal I started to take a stab at this but I hadn't realized that we also inject all of the parameter values into the notebook metadata: https://github.com/nteract/papermill/blob/master/papermill/parameterize.py#L104
Why exactly is there a need to store this extra copy of the parameters if they appear in the |
It was intended to give programmatic access to what parameters were set. You can't read what the user input was from the cell itself if there was manipulations. To make this work, we'd likely want to have a placeholder object for non-json fields being added or catch the invalid json input and skip saving the parameters to the metadata. |
Along the lines of #215, it seems like there are quite a few parameter types that would be desirable to pass as inputs (most notably
pd.DataFrame
s) that are simple enough to translate to/from JSON. Doing the transformation manually every time is a bit of a headache; is there any current method to register a custom translator that would handle this conversion automatically? Monkey-patchingtranslate
papermill/papermill/translators.py
Lines 80 to 99 in 718f39e
To be clear, this is not a proposal to automatically convert pandas objects (though I could certainly see an argument for that as well 😉 ), just to expose a method for users to add their own serialization methods.
dask
's method for allowing custom serializers is a nice, straightforward example of something similar: https://distributed.dask.org/en/latest/serialization.html#id3The text was updated successfully, but these errors were encountered: