There has been an increasing interest in uncovering the mechanisms underpinning how people decide which task to perform at a given time. Many studies suggest that task representations are crucial in guiding such voluntary task selection behavior, which is primarily reflected in a bias to select task repetitions over task switches. However, it is not yet clear whether the task-specific motor effectors are also a crucial component of task representations when deciding to switch tasks. Across three experiments using different voluntary task switching (VTS) procedures, we show that a greater overlap in task representations with a task-to-finger mapping than task-to-hand mapping increases participants’ switching behavior (Exp. 1 and Exp. 2), but not when they were instructed to randomly select tasks (Exp. 3). Thus, task-specific stimulus-response associations can change the way people mentally represent tasks and influence switching behavior, suggesting that motor effectors should be considered as a component of task representations in biasing cognitive flexibility.