There are a number of well-accepted ways to measure risk sensitivity, with researchers often making conclusions about individual differences based on a single task. Even though long-standing observations suggest that how risky outcomes are presented changes peoples behavior, it is unclear whether risk sensitivity is a unitary trait that can be measured by any one of these instruments. To directly answer this question, we administered three tasks commonly used to elicit risk sensitivity within-subject to a large sample of participants on Amazon Mechanical Turk. Our findings revealed high individual variability in each measure, with little evidence of consistency among different tasks: many participants who were classified as risk-averse in one task were risk-seeking in another, and we observed no significant correlations between continuous measures of risk sensitivity as measured in each of the tasks. Our results cast doubt on the pervasive assumption that risk paradigms measure a single underlying trait, and suggest instead that behavior in risky situations is the result of heterogeneous, interacting, and possibly task-dependent cognitive mechanisms.