-
Background The paper presents the context of human capability to intuitively understand postures from an image or a brief description, highlighting the disconnection between traditional image-based or text-based pose estimation methods and holistic scene comprehension and nuanced reasoning.
-
Existing Work Current methods typically identify individuals in an image, segment them, and then use a neural network to predict 3D pose and shape. However, these approaches lack a comprehensive understanding of the scene, such as how to estimate 3D human pose from 2D keypoints, which lack contextual information. Likewise, text-driven pose generation methods have evolved rapidly but are constrained by literal instructions and limited by the specific actions described in their training data, like motion capture datasets, which have a narrow expression of human behavior, emotion, and interaction. Consequently, existing specialized systems for 3D pose estimation and generation are bound to narrow tasks as opposed to the general-purpose reasoning showcased by Large Language Models (LLMs).
- Introduced a framework named PoseGPT
- Challenge 1: Addressing novel pose generation and reasoning issues Existing systems are limited by training data and explicit instructions. The PoseGPT framework counters this by embedding SMPL poses as a unique signal token within a multimodal LLM, thereby enabling direct generation of 3D body poses from both textual and visual inputs, simplifying pose prediction, and empowering LLMs to reason about human poses with their world knowledge to support tasks like speculative pose generation (SPG) and reasoning about pose estimation.
- Challenge 2: Fine-tuning LLM through interactive Q&A LLMs must be capable of understanding and interpreting 3D human pose, which PoseGPT achieves by fine-tuning multimodal Large Language Models to predict human pose as SMPL pose parameters. The approach uses a unique token to prompt the LLM to output pose information when queried about SMPL pose-related questions. A novel training strategy was implemented by constructing Q&A pairs derived from pose estimation and text-driven pose generation tasks.
PoseGPT was evaluated on a diverse set of tasks, including traditional 3D human pose estimation from a single image and pose generation from text descriptions. Even though the classical task accuracy metrics do not yet match specialized methods, the authors consider this as proof of concept. More importantly, once LLMs can understand SMPL poses, they can leverage their inherent world knowledge for associating and reasoning about human poses without extensive additional data or training. This capability led to the creation of innovative tasks within the field of human pose analysis.
PoseGPT is a novel framework that enables models to directly generate 3D human poses from textual and visual inputs by embedding SMPL pose tokens within LLMs, achieving some innovation in interpreting 3D human pose.