Community-based Question Answering (CQA) platforms can provide rich experience and suggestions for people who seek to construct Activity Plans (AP), such as bodybuilding or sightseeing. However, answer posts in CQA platforms could be too unstructured and overwhelming to be easily applied to AP construction, as validated by our formative study for understanding relevant user challenges. We therefore proposed an answer-post processing pipeline, based on which we built PlanHelper, a tool assisting users in processing the CQA information and constructing AP interactively. We conducted a within-subject study (N=24) with a Quora-like interface as the baseline. Results suggested that when creating AP with PlanHelper, users were significantly more satisfied with the informational support and more engaged during the interaction. Moreover, we performed an in-depth analysis on the user behaviors with PlanHelper and summarized the design considerations for such supporting tools.