An important finding in the cognitive effort literature has been that sensitivity to the costs of effort varies between individuals, suggesting that some people find effort more aversive than others. It has been suggested this may explain individual differences in other aspects of cognition; in particular that greater effort sensitivity may underlie some of the symptoms of conditions such as depression and schizophrenia. In this paper, we highlight a major problem with existing measures of cognitive effort that hampers this line of research, specifically the confounding of effort and difficulty. This means that behaviour thought to reveal effort costs could equally be explained by cognitive capacity, which influences the frequency of success and thereby the chance of obtaining reward. To address this shortcoming, we introduce a new test, the Number Switching Task (NST), specially designed such that difficulty will be unaffected by the effort manipulation and can easily be standardised across participants. In a large, online sample, we show that these criteria are met successfully and reproduce classic effort discounting results with the NST. We also demonstrate the use of Bayesian modelling with this task, producing behavioural parameters which can be associated with other measures, and report a preliminary association with the Need for Cognition scale.