-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Assertions FAQ #434
Comments
One thing to note is once an DSPyAssertionError/DSPySuggestionError is triggered at any point, this is considered as an "attempt." This can have some implications with multiple Assertion statements as if a particular statement continually produces an invalid response, the result will error out on that corresponding statement(if using
These are great FAQs for Assertions! Feel free to drop any follow-ups or additional questions! |
Hey @arnavsinghvi11 thanks for answering so fast these questions. I want to comment about 3 & 5.
DSPyAssertionError(
id=self.id,
msg=self.msg,
state=dsp.settings.trace,
is_metric=self.is_metric,
) this causes
|
Hey @Neoxelox For #3, I believe this was patched in a recent commit. Could you pull or maybe rebuild from the repo? Great point regarding #5. I think in the case of multiple modules with the same signature, it would be advised to keep the corresponding Assertion statement after the relevant module, since by default, it will trace back to the most recent one. (However, naturally, there are edge cases where the multiple modules have unrelated outputs but the same signature so we'll have to integrate a stronger comparison in that case). leaving this open for now. |
Gathering questions about assertions and suggestions from different users.
This will be most relevant for @Shangyint @manishshettym @arnavsinghvi11.
What's changing between different versions? What will be the final stable API for everything below?
Do users need to manually set
trace=[]
. Can this be automated?Does
dspy.Assert
retry? If not, shouldn't it?What happens if I don't set
target_module
? Does it directly re-run the last module? What's confusing people here currently.Is it true that you can't really pass a module to the target_module , it must be it's signature?
I saved my compiled program with assertions. When I load it (like code below), I get
DSPySuggestionError
.The text was updated successfully, but these errors were encountered: