-
Background LLMs are commonly used for both simple and complex tasks but face issues like hallucinations, brittleness to prompt changes, and limited user interactions. Complex tasks exacerbate these deficits as they require multiple reasoning steps, such as text shortening which involves deciding which parts to edit and how to revise each one.
-
Existing Work Existing solutions involve LLM chaining to decompose tasks into sub-tasks. However, these solutions are still not perfect. Researchers are thus looking into crowdsourcing workflows to help design LLM chains to address quality challenges.
- Constructed a design space
-
Challenge 1: How to systematically develop LLM chains for complex tasks? Researchers reviewed 107 crowdsourcing and chaining literature to create a three-tiered design space to help LLM chain designers from high-level objectives to strategies and tactics. This design space connects a designer's objectives to strategies and tactics to implement those strategies.
-
Challenge 2: How to assess the impact of crowdsourcing workflow techniques adapted to LLM chain design? Researchers conducted three case studies adapting crowdsourcing workflows for LLM use. They adjusted several crowdsourcing workflows for tasks varying in degrees of creativity, like Cascade for controlling the precision of a generated taxonomy, Soylent for text shortening, and Mechanical Novel for controlling narrative elements of a story.
-
In the conducted case studies, chain-generated stories were preferred by crowdsourced raters compared to zero-shot prompting. These case studies evaluated the chains in terms of the quality and controllability of their outputs and discussed the transferable techniques from crowdsourcing to LLM chains, as well as implications for future research and development.
The paper introduces a design space framework and three case studies adapting crowdsourcing workflows to LLM chains, providing practical guidance and theoretical insights for the future design and development of LLM chains.