Abstract:This study focuses on the problem of procedure planning in instructional videos. Given the start and end observations, the task is to plan an action sequence that transforms the start state into the end state within the action space provided by the instructional videos. Instructional videos record and demonstrate the operational processes of various events. Each event includes a specific set of actions, forming the action space for that event. Therefore, the action space is composed of various subspaces corresponding to different events in instructional videos. Previous methods fail to effectively utilize the semantic information of events and overly rely on techniques such as reinforcement learning, resulting in complex training schemes and poorly explainable approaches. In contrast, this study considers the characteristics of instructional videos and proposes the classification-retrieval-based procedure planner (CPP), a pipeline that addresses procedure planning from coarse to fine. Specifically, the planner first identifies the event category based on the given observations, narrowing the action space to a smaller subspace. Then, action planning is performed within the selected subspace, which is significantly easier than planning in the entire action space. Moreover, this study introduces a hybrid planning method that combines retrieval and prediction approaches to generate the action sequence. The proposed method achieves competitive results on three popular procedure planning datasets of varying scales, establishing itself as a simple yet robust baseline for procedure planning.