You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The extend generation logic in dsp/primitives/predict.py is producing the behaviour illustrated below.
The example uses an artificially low max_tokens to illustrate the bug with real LM calls.
importdspyllm=dspy.OpenAI(model='gpt-3.5-turbo', max_tokens=10)
dspy.settings.configure(lm=llm)
classSciGen(dspy.Signature):
"""context and answer on science questions"""question=dspy.InputField(desc="Input question")
sun_context=dspy.OutputField(desc="sun_context")
moon_context=dspy.OutputField(desc="moon_context")
earth_context=dspy.OutputField(desc="earth_context")
relative_distances_context=dspy.OutputField(desc="relative_distances_context")
answer=dspy.OutputField(desc="Answer only when you have all the context fields.")
context_generator=dspy.Predict(SciGen, n=1)
response=context_generator(question="Why do we have solar eclipse?")
response# Prediction(# sun_context='During a solar eclipse, the Moon passes between the',# moon_context='', # earth_context="Earth and the Sun, blocking the Sun's light",# relative_distances_context='',# answer='Answer: A solar eclipse occurs when the Moon'.# )
Explanation
When the model does not generate all the correct field prefixes in the correct order, logic in dsp/primitives/predict.py is used to continue the generation until the signature is completed.
And a further call to the LM is made with the updated prompt - with the "last field" as the trailing prefix - ie
"""...Moon Context: """
Whilst the prompt is rendered correctly,template.extract compares the field value to None to decide where the new completion from the LM is meant to go in the signature:
Since the value of the last field is "" the new completion is instead attached to a field which is still None. This logic stands in contrast to how the prompt was built - where the field value is compared to "" to determine if it is "complete".
updating the extend generation logic to assign None to the partial field
updating the TemplateV2.extract logic to check for "" when assigning outputs
updating the TemplateV2.query logic to exclude fields with empty-string values as completed when determining which field to leave as the trailing prefix.
Speculation
I think the bug makes Dspy fail more quietly when the LM does not generate the fields as specified in the "Follow the following format" section of the prompt - as reported by #1169 (comment) after a now reverted change to the extend generation logic (#920). The current logic means that at least one extra OutputField is added to the results for every recursion in generate - independent of what the LM produces. I think this means complex signatures will be reliably completed (up to the user-provided recursion depth) even if the LM is not actually generating the required field prefixes.
I think a "fix" to this bug might cause some examples to start to "fail" more loudly when the extend generation logic reaches its maximum recursion depth.
A more atomised fix to this bug would ensure that the last incomplete field is skipped correctly when the generation is extended, with appropriate documentation.
Cheers
The text was updated successfully, but these errors were encountered:
The extend generation logic in
dsp/primitives/predict.py
is producing the behaviour illustrated below.The example uses an artificially low max_tokens to illustrate the bug with real LM calls.
Explanation
When the model does not generate all the correct field prefixes in the correct order, logic in
dsp/primitives/predict.py
is used to continue the generation until the signature is completed.dspy/dsp/primitives/predict.py
Line 95 in 55510ee
The "Last field"'s value is set to
""
- presumably to throw away a partial completion?dspy/dsp/primitives/predict.py
Line 98 in 55510ee
And a further call to the LM is made with the updated prompt - with the "last field" as the trailing prefix - ie
Whilst the prompt is rendered correctly,
template.extract
compares the field value toNone
to decide where the new completion from the LM is meant to go in the signature:dspy/dsp/templates/template_v2.py
Line 156 in 55510ee
Since the value of the last field is
""
the new completion is instead attached to a field which is stillNone
. This logic stands in contrast to how the prompt was built - where the field value is compared to""
to determine if it is "complete".dspy/dsp/templates/template_v2.py
Line 81 in 55510ee
The bug can be fixed by either
None
to the partial fieldTemplateV2.extract
logic to check for""
when assigning outputsTemplateV2.query
logic to exclude fields with empty-string values as completed when determining which field to leave as the trailing prefix.Speculation
I think the bug makes Dspy fail more quietly when the LM does not generate the fields as specified in the
"Follow the following format"
section of the prompt - as reported by #1169 (comment) after a now reverted change to the extend generation logic (#920). The current logic means that at least one extra OutputField is added to the results for every recursion ingenerate
- independent of what the LM produces. I think this means complex signatures will be reliably completed (up to the user-provided recursion depth) even if the LM is not actually generating the required field prefixes.I think a "fix" to this bug might cause some examples to start to "fail" more loudly when the extend generation logic reaches its maximum recursion depth.
A more atomised fix to this bug would ensure that the last incomplete field is skipped correctly when the generation is extended, with appropriate documentation.
Cheers
The text was updated successfully, but these errors were encountered: