Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add examples explaining advanced applications of sample_posterior_predictive #7014

Merged
merged 2 commits into from
Mar 1, 2024

Conversation

ricardoV94
Copy link
Member

@ricardoV94 ricardoV94 commented Nov 17, 2023

This is still a draft. I want to add two more examples first about the default ppc and predictions use-cases

Related to #7069

TODO

  • Add example about common use for in-sample posterior_predictive
  • Add example about common use for predictions
  • Add examples on using new models
  • Change conclusion about non-dependence on deterministics or fix behavior
  • Explain volatility when MutableData changes I think this is itself a mine-field and I would rather change the behavior first

Link to function docs preview: https://pymcio--7014.org.readthedocs.build/projects/docs/en/7014/api/generated/pymc.sampling.forward.sample_posterior_predictive.html


📚 Documentation preview 📚: https://pymc--7014.org.readthedocs.build/en/7014/

@ricardoV94 ricardoV94 changed the title Add explanation about the non-custom behavior of var_names in sample_posterior_predictive Add example about the behavior of var_names in sample_posterior_predictive Nov 17, 2023
Copy link

codecov bot commented Nov 17, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.24%. Comparing base (6c6fd13) to head (08fca4e).
Report is 1 commits behind head on main.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #7014   +/-   ##
=======================================
  Coverage   92.24%   92.24%           
=======================================
  Files         100      100           
  Lines       16887    16887           
=======================================
  Hits        15577    15577           
  Misses       1310     1310           
Files Coverage Δ
pymc/sampling/forward.py 95.90% <ø> (ø)

@ricardoV94 ricardoV94 force-pushed the posterior_pred_example branch 3 times, most recently from f0dccc7 to 3d0f869 Compare November 17, 2023 10:22
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
@ricardoV94 ricardoV94 changed the title Add example about the behavior of var_names in sample_posterior_predictive Explain the behavior of var_names in sample_posterior_predictive Nov 17, 2023
@jessegrabowski
Copy link
Member

jessegrabowski commented Nov 17, 2023

Last suggestion, getting into the weeds of personal preferences -- I think the API docs would look nicer if you added a title (in bold or italics or w/e) to each of the two examples.

I also think the code in the new section would be easier to follow if you split the model definition plus each function call into a separate code block and wrote your commentary as text instead of code comments.

@ricardoV94
Copy link
Member Author

ricardoV94 commented Nov 17, 2023

Last suggestion, getting into the weeds of personal preferences -- I think the API docs would look nicer if you added a title (in bold or italics or w/e) to each of the two examples.

I also think the code in the new section would be easier to follow if you split the model definition plus each function call into a separate code block and wrote your commentary as text instead of code comments.

I agree, even better would be a markdown header, but I don't know if that's supported :D

@jessegrabowski
Copy link
Member

I think you can use restructuredtext subsubsection headings, like:

Example 1: Exploring the effect of var_names on outputs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

See here. It's pretty hideous for viewing in an IDE, though.

@ricardoV94
Copy link
Member Author

That worked @jessegrabowski

pymc/sampling/forward.py Outdated Show resolved Hide resolved

.. code:: python

pm.sample_posterior_predictive(idata, var_names=["obs"], **kwargs)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this is called default behaviour, I would make it pm.sample_posterior_predictive(idata) nothing else.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this helps understanding how the behavior changes when you start playing with it the kwarg

pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
pymc/sampling/forward.py Show resolved Hide resolved
@AlexAndorra
Copy link
Contributor

Thanks @ricardoV94 , these clarifications are very much welcome!

I want to add two more examples first about the default ppc and predictions use-cases

Seems like default ppc is done and you only need predictions use-case?

@ricardoV94 ricardoV94 changed the title Explain the behavior of var_names in sample_posterior_predictive Add examples explaining advanced applications of sample_posterior_predictive Feb 27, 2024
@ricardoV94
Copy link
Member Author

ricardoV94 commented Feb 27, 2024

I have addressed several review comments:

  • Do not call new draws posterior predictive draws, but add a note for the first case where it is equivalent
  • Fix the Deterministics example

I also expanded with two new example sections:

  • Posterior predictive checks and predictions: Shows the common use case
  • Using new models: Shows how you can use a new model with new variables

It's ready for review again. It is also likely I messed up the myst formatting so any help appreciated.

@ricardoV94 ricardoV94 marked this pull request as ready for review February 27, 2024 11:45
Copy link
Contributor

@AlexAndorra AlexAndorra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Love it @ricardoV94 ! Just suggested a few changes, then good to merge

pymc/sampling/forward.py Show resolved Hide resolved
pymc/sampling/forward.py Show resolved Hide resolved
pymc/sampling/forward.py Show resolved Hide resolved
pymc/sampling/forward.py Outdated Show resolved Hide resolved
@ricardoV94
Copy link
Member Author

@AlexAndorra thanks for the review and suggestions. I've addressed them

Copy link
Contributor

@AlexAndorra AlexAndorra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good, thanks @ricardoV94 🥳
Just waiting for an answer to solve link formatting -- and took the opportunity to add a nit 😜

pymc/sampling/forward.py Outdated Show resolved Hide resolved
Comment on lines 670 to 672
Note that "sampling" a :func:`~pymc.Deterministic` does not force random variables
that depend on this quantity to be sampled too. In the following example ``z`` will not
be resampled even though it depends on ``det_xy``:
Copy link
Contributor

@AlexAndorra AlexAndorra Feb 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we have to wait for Oriol's answer on the extrernal link before merging, let me add a nit 🙈
I don't think it'll be super clear to all users that z depends on det_xy, because it looks like it only depends on x and y .
If you did this: z = pm.Normal("z", det_xy), that'd probably be clearer. Not a big deal, just my 2 cents

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I actually wanted to do! Wonder if the bug comes bug up... xD

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah ok! Then that makes more sense

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shows the bug again! I added a separate commit that includes a danger section

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opened an issue: #7183

@ricardoV94
Copy link
Member Author

Things seem to be working. @AlexAndorra wanna do one last pass?

Copy link
Contributor

@AlexAndorra AlexAndorra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good now @ricardoV94 🍾

@AlexAndorra AlexAndorra merged commit 46f1675 into pymc-devs:main Mar 1, 2024
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants