One of the clearest challenges to the validity of self-reports of SWB comes from Schwarz and Strack’s judgment model of SWB (for a review, see Schwarz and Strack 1999). The judgment model focuses on global evaluative measures of subjective well-being: i.e., measures like life satisfaction judgments, which ask respondents to consider their life as a whole and to provide a summary evaluation. According to the judgment model, responses to these global measures result from a judgment process that is susceptible to systematic sources of bias and invalidity. Multiple studies from the judgment model perspective have investigated specific processes that are purported to occur when these judgments are made.

For instance, one concern is that SWB judgments are made too quickly to reflect an overall evaluation of all relevant information about a person’s life. Thus, people may quickly scan their memories (or even their current thoughts and feelings) for relevant information, relying too heavily on the information—including potentially unimportant information—that happens to be accessible. Indeed, Schwarz and Strack (1999) suggested that “information that has just been used—for example, to answer a preceding question in a questionnaire—is particularly likely to come to mind later on” (p. 63), and that this temporarily accessible information biases the resulting responses.

In a famous demonstration of this effect, Strack et al. (1988) randomly assigned respondents to answer two questions, one about their global life satisfaction and one about their satisfaction with dating, in one of two orders: dating satisfaction first or global satisfaction first. According to the judgment model, when dating satisfaction is presented first, this life domain should be made especially salient, which should result in strong correlations between the two questions. In contrast, when the global evaluation is presented first, the respondent’s romantic life may not be on his or her mind, and may thus have less of an impact. Results from two separate studies supported this prediction: Correlations between the two questions were only .16 and -.12 when the global satisfaction question was asked first, but they were .55 and .66 when the satisfaction with dating question was asked first. Thus, these studies suggest that the information that happens to be accessible at the time of a global judgment affects the judgment itself.²

Importantly, it is not just information about a person’s life that can affect these judgments; according to judgment model researchers, even feelings that people are experiencing at the time of judgment can affect their judgments in undesirable ways. In one of the most cited examples of research from this tradition, Schwarz and Clore (1983) conducted two studies to assess whether respondents tended to rely on potentially irrelevant current moods when making judgments about their life as a whole. In each of these studies, the authors manipulated the respondents’ current mood (using recall of positive and negative life events in one study and changes in daily weather in the second) to assess whether these manipulations also affected their life satisfaction judgments. In both studies, the respondents in the positive mood conditions reported substantially higher life satisfaction than those in the negative mood conditions (with effect sizes close to a full standard deviation difference). These compelling results were also replicated in other studies that used different mood induction procedures, including placing respondents in a pleasant or an unpleasant environment (Schwarz et al. 1987), surveying people before and after important soccer games with varying outcomes (Schwarz et al. 1987), and even simply arranging for respondents to find a dime before completing the relevant survey (Schwarz 1987).

The findings of other studies have even suggested that feelings that arise simply as a result of the process of searching one’s memory for relevant information can affect judgments. For instance, although this study was not conducted with well-being judgments as an outcome, Schwarz et al. (1991) found that the ease with which respondents could recall examples relevant to a characteristic could affect their ratings of that characteristic. Specifically, participants who were asked to list six examples of times when they acted assertively rated themselves as more assertive than those who were asked to list 12 examples of assertive behaviors. Presumably, the feelings respondents had about the difficulty associated with producing 12 examples provided information that was used in the judgment itself. To my knowledge, studies of this accessibility effect have not been conducted in the well-being domain. However, evidence of this general phenomenon adds support to the idea that a variety of feelings that people experience (including those that are arguably irrelevant for the judgment at hand) can influence their broad judgments.

Taken together, these studies suggest that irrelevant information that happens to be on a person’s mind at the time of judgment can have strong effects on the resulting judgment. These findings represent an important challenge to the validity of global reports of SWB because they suggest that people do not always make SWB judgments in a manner that is consistent with theory or intuition, and that irrelevant information may play a large role in their responses. Indeed, after reviewing research from the judgment model tradition, Schwarz and Strack suggested that one plausible conclusion could be that “There is little to be learned from global self-reports of well-being … [What] is being assessed, and how, seems too context dependent to provide reliable information about a population’s well-being, let alone information that can guide public policy” (1999, p. 80). These authors argued that the instability inherent in the judgment process limited the extent to which SWB judgments reflected the actual quality of the respondents’ lives. Research from the judgment model tradition is often cited in critiques of well-being research, and the presumed problems that these judgment processes create for the validity of these measures have led some scholars to suggest that alternatives to global measures of well-being are needed (e.g., Kahneman 1999; National Academy of Sciences 2013; OECD 2013;Testoni et al. 2018).

Over the past decade, scientists have been grappling with concerns about the replicability of previously published work (e.g., Open Science Collaboration 2015). Although concerns about replicability have touched many areas of science, the field of social psychology—the field that encompasses judgment model research—has been at the heart of this debate. Social psychological studies fared poorly in the first major attempt to replicate large sets of high-profile studies (Open Science Collaboration 2015), and more targeted investigations of specific high-profile findings have also repeatedly failed to replicate initial claims (e.g., O’Donnell et al. 2018; Cheung et al. 2016; Eerland et al. 2016; McCarthy et al. 2018). To be sure, researchers do not know the true rate of replicability of social psychological studies, but the substantial number of failed replications that has emerged has led to a renewed skepticism about even the most foundational studies in the field.

Specific concerns about problematic research practices have strengthened this skepticism. For instance, researchers (not just from social psychology) have admitted to a variety of such practices, including optional stopping (i.e., checking the results before the data collection is complete and stopping once significance is obtained); the strategic omission of outliers, measures, or even experimental conditions that do not support hypotheses; and post hoc theorizing that makes it appear as though unpredicted effects were hypothesized (John et al. 2012; Kerr 1998). When combined with a tendency to use underpowered designs (Button et al. 2013) and a broad bias against publishing null results, these practices can lead to a high rate of false positives in the literature (Ioannidis 2005). Indeed, in a famous investigation of the consequences of these practices, Simmons et al. (2011) showed that support for clearly wrong (and nonsensical) hypotheses can easily be found when a combination of these practices is used.

These issues are important for discussions about the measurement of SWB. When highly cited studies from the judgment model are evaluated from this modern, skeptical perspective, there are reasons for concern. Specifically, almost all original effects in support of the judgment model of SWB came from a single research laboratory; they typically rely on extremely small sample sizes (with per-cell samples often around 10 to 15 participants; see Yap et al. 2017); they often have effect sizes that could be considered implausibly large; and there have been increasing numbers of failures to replicate the basic findings from the original studies. For instance, in the original small sample study showing that asking about one life domain (in this case, dating) before asking about general life satisfaction affected the size of the correlation between the two items, the correlations in the condition where general life satisfaction was asked first were much smaller than those that are typically found. These correlations ranged from -.12 to .16 (Strack et al. 1988), whereas previous research on domain satisfaction consistently shows that satisfaction with just about any domain of life tends to correlate moderately (i.e., in the range of .2 to .5) with global life satisfaction judgments, regardless of the order in which they are assessed (Schimmack and Oishi 2005).

Indeed, years after Strack et al.’s (1988) initial item-order study, Schimmack and Oishi (2005) reviewed all of the studies that they could find that used a similar methodology, and they concluded that these initial results from Strack et al. (1988) were outliers. The average correlation between global satisfaction and narrower domain satisfactions was r=.32 when the global measure was asked first, compared to a just slightly larger r=.40 when the domain satisfaction was asked first. Importantly, the difference was even smaller when the Strack et al. (1988) studies were excluded. Moreover, Schimmack and Oishi (2005) conducted five new studies, all of which found extremely small differences across conditions. Using a different design, Lucas et al. (2018) also showed that questions asked before a life satisfaction measure do not have a strong influence on responses to that global question. Thus, although making information salient before asking questions about life satisfaction may have small effects on those judgments, the striking effects presented by Strack et al. (1988) appear to be a dramatic overestimate of the typical effect of temporarily accessible information on global well-being judgments.

The concerns about replicability extend to other evidence from the judgment model. For instance, to test whether ease of retrieval affects judgments of personality characteristics, Yeager et al. (2019) used a large, nationally representative sample to test whether being asked to list six versus 12 assertive behaviors affected people’s ratings of their own assertiveness. In contrast to the original small sample studies (with N s with approximately eight and 20 participants per cell), the effects found in this large study (N=1,338 for a two-cell design) were not significant. Again, independent replication that used a high-quality, large sample failed to find results that were consistent with the original findings.

The most well-supported part of the judgment model concerns the mood effects on life satisfaction judgments, which have been found in multiple studies conducted by Schwarz, Strack and their colleagues. Again, however, the original studies often have characteristics associated with low rates of replicability (e.g., small sample sizes, inconsistent measures and analytic approaches, and p-values that are close to .05), and few if any successful replications have been published. Indeed, a growing number of large-scale replications have failed to find effects consistent with the original results.

For instance, Schwarz’s (1987) study, which purported to show that finding a dime can affect life satisfaction ratings, had only eight participants in each of the two conditions, and the differences between the two conditions were not significant. In another study that focused on people’s reactions to soccer games, Schwarz et al. (1987) called separate groups of respondents before and after two important soccer games, one resulting in a win for the local team, and one resulting in a tie. The authors found a significant interaction between the time (before the game versus after) and the outcome (win versus tie) when predicting life satisfaction and concluded that life satisfaction judgments were influenced by the outcome of the game. However, a closer examination of the results from this study shows that there was a cross-over interaction that was driven as much by theoretically irrelevant and unexplainable differences in life satisfaction before the game as by the differences after the game, and no follow-up comparisons were conducted. Especially considering the small sample size used in the study, the robustness of these results can be questioned.

The study that received the most attention was Study 2 from Schwarz and Clore (1983). In this study, the researchers called respondents on sunny days or rainy days and found that those contacted on rainy days reported lower life satisfaction than those contacted on sunny days. The explanation was that weather affects mood, and respondents rely on their mood when making life satisfaction judgments. However, this study had six separate conditions, each of which included approximately 14 participants. Importantly, five of six conditions were not significantly different from one another (and had means that were quite similar), and just one very large discrepancy drove the entire effect. The difference in this one 14-person cell was quite large, exceeding the decline in SWB that is typically found after the onset of a serious disability or in the first year of widowhood (e.g., Lucas 2007; Lucas et al. 2003). This effect is, arguably, implausibly large given the design of the study.

In recent years, a number of researchers have attempted to replicate this and other mood studies from the judgment model tradition with little success. For instance, Lucas and Lawless (2013) used data from over one million Americans from all 50 states assessed over a five-year period, and found almost no evidence that any form of weather affected life satisfaction judgments, despite having extremely high power to detect even exceedingly small effects. Using panel data, Feddersen et al. (2016) did find significant associations between weather and life satisfaction, but they also had extremely high power, and the effect of weather found in their study was approximately one one-hundredth the size of the effect reported in the original study; i.e., a size that is arguably inconsequential and unimportant for validity.⁵ Lucas and Lawless reviewed additional evidence that calls into question the robustness of the original finding of a weather effect.

Finally, because studies like those conducted by Lucas and Lawless (2013) and Feddersen et al. (2016) were conceptual replications that relied on existing data with correlational designs, Yap et al. (2017) conducted nine new experimental studies using procedures very similar to the classic judgment model mood studies, but with much larger samples. Although significant effects of mood induction procedures on life satisfaction were found in some of their studies (although not for weather), these effects were substantially smaller than those reported in the original study (e.g., d s of .09 and .11 for the two focal outcome measures). Thus, it is not clear that these effect sizes were large enough to affect the validity of traditional global well-being measures. These findings imply that concerns raised by judgment model studies about the validity of SWB measures may not be well-founded.

The previous sections raised concerns about the robustness of existing evidence that challenges the validity of global measures of SWB. An additional issue that should be considered when comparing different measures is whether the proposed alternatives can solve the problems that global measures are thought to have. Experiential measures, such as those that can be obtained from experience sampling methodologies or the day reconstruction method, address concerns about global measures by reducing reliance on the respondents’ ability to remember and aggregate. Thus, these methods address two of the most salient concerns about global measures. However, these potential strengths do not ensure that scores from experiential measures are in fact more valid than global measures of SWB. Indeed, as was noted above, it is not even clear that the concerns that have been raised about global measures translate into demonstrable decrements in validity. Thus, issues of relative validity are empirical questions that must be assessed directly.

Unfortunately, few direct comparisons of the validity of the two types of measures exist. It seems that researchers have relied on the face validity of experiential measures as the justification for their use, without subjecting these measures to more rigorous tests of validity for the purposes of assessing individual- and population-level SWB. For instance, in introducing the day reconstruction method, Kahneman et al. (2004a) simply compared absolute sample-level frequencies and diurnal patterns of affect found using this new method to those found with the existing experience sampling method. Since the publication of this initial report, few explicit tests of validity have been reported.

Admittedly, the lack of testing is attributable in part to some uncertainty about precisely how researchers should validate experiential measures. There is no gold standard measure that can be used as a criterion, and—as I have noted—widely used global measures have been criticized. Thus, the use of experiential measures has been justified primarily based on their assumed beneficial attributes. In recent years, however, my colleagues and I have attempted to address this oversight by considering additional approaches to validating these methods.

For instance, if experiential measures like those obtained from the experience sampling method and the day reconstruction method are valid, they should provide similar results when used to assess well-being in the same sample over the same time period. Initial investigations that compared the two methods (e.g., Kahneman et al. 2004a) used different samples and only compared sample-level statistics. Studies like this, however, provide an incomplete picture of the convergence across the two methods.

Recently, we asked two samples of students to complete a day of experience sampling (in which they reported on what they were doing and how they were feeling up to nine times per day), followed by a reconstruction of the exact same day using the day reconstruction method (Lucas et al. 2021). This allowed us to compare responses at multiple levels: the sample level (e.g., were the overall sample-level estimates of time spent in specific situations and the average affect similar across methods?), the person level (e.g., did people who reported being happy with others using one method also report being happy with others using the other method?) and the moment level (e.g., when people reported being happy at a particular moment using one method, did they report being similarly happy using the other method?). Results varied depending on the level examined.

When looking at sample-level statistics, the level of agreement was quite high. For instance, when comparing estimates of time spent in various situations, the correlations across the two methods of assessment (using situations as the unit of analysis) exceeded .95. Similarly, when looking at the average positive and negative affect reported in each situation, the correlations across methods were again quite high, ranging from .76 to .85. Thus, if the goal was to estimate sample-level statistics about these experiences, both methods provided similar information.

However, when moving to the person level of analysis, the results were more varied and problematic. First, when assessing individual differences in affect experienced over the course of the day, the two methods converged reasonably well: the between-person correlations (i.e., the correlation between individual differences in average affect estimated with the ESM and average affect estimated with the DRM) for positive and negative affect were .83 and .88, respectively. Yet reports of time spent in specific situations varied across types of situations: person-level agreement was high (ranging from .74 to .83) for reports of how much time a person spent in specific locations, like home, school or work; but it was relatively low for reports of specific activities, like commuting (r=.25), eating (r=.21), or doing housework (r=.28).

Finally, when looking within individuals over time, reports of both affect and activities diverged across methods. These analyses assessed the changes in a person’s affect and activities over time and examined whether the changes measured using one method corresponded to the changes measured using the other. For within-person affect, correlations ranged from r=.34 for positive affect to r=.36 for negative affect (in Study 1; results were similar for Study 2). We calculated within-person kappa coefficients to assess the level of agreement for dichotomous situation variables, and the average kappa across situations was .34. Thus, it appears that two forms of experiential measures—experience sampling and day reconstruction—did not provide the same information when used to assess a day of experience. These results raise some concerns about the validity of these measures.

Moreover, although there is no gold standard against which to compare either global or experiential measures, it is possible to compare both sets of reports to alternative measures that have different strengths and weaknesses. Lucas et al. (2021) obtained informant reports of life satisfaction from friends and family members who knew the respondent well and used these as a criterion that could be predicted from global self-reports, experience-sampling-based reports, and day-reconstruction-based reports. Although the differences in the correlations were not significant, these associations were somewhat higher for the global measures than for the experiential measures, which suggests that at the very least, the validity of the experiential measures did not exceed the validity of the global reports when informant reports were used as a criterion (also see Hudson et al. 2020, 2017; and Hudson et al. 2019a, for similar results). An additional study that looked at the associations over time between well-being and health also found that the associations were slightly stronger for the global measures than for the experiential measures (Hudson et al. 2019b). Overall, these results suggest that there is at least some cause for concern about the validity of experiential measures; and that at the very least, it should not be assumed that such measures provide scores that are more valid than those provided by traditional global measures.

The goal of this paper is not simply to defend global measures and critique experiential alternatives. Instead, the goal is to raise awareness about asymmetries in the ways these classes of measures have been evaluated. Critiques of global measures often proceed by focusing on the processes that underlie the judgments that respondents are asked to make. Experiential measures are often assumed to have advantages over global measures precisely because they do not rely on the same underlying processes. However, as the evidence reviewed in the previous section shows, these fundamental differences in these measures do not always translate into improved reliability and validity for experiential alternatives to global reports. Moreover, few researchers have considered whether there are additional psychological processes underlying these experiential measures that have the potential to systematically (and negatively) affect their psychometric properties. In this final section, I discuss some potential concerns about the processes that underlie experiential measures.

Experiential measures clearly solve two challenges of global measures of SWB: they reduce reliance on memory, and they avoid the need for aggregation on the part of the respondent. However, SWB researchers have, for the most part, failed to consider whether these measures have unique problems that global measures avoid. For instance, it is possible that the increased respondent burden associated with experiential measures affects the quality of the responses that respondents provide (Eisele et al. 2020). In addition, because of logistical challenges associated with experiential measures, these measures are often more difficult to implement than global measures in large-scale survey work. This means that selection bias may affect studies that use experiential measures more than studies that rely on global measures, as studies that use experiential measures must often rely on weaker sampling plans.

One salient difference between experiential measures and global measures is that the former measures require respondents to answer the same question multiple times so that multiple experiences can be assessed. Moods and other evaluative experiences fluctuate from moment to moment (e.g., Epstein 1979); therefore, to assess something that is representative of a person’s life, multiple moments must be aggregated. Thus, respondents must report on many such occasions, either as the occasions occur (in experience sampling), or soon after they have happened (as in the day reconstruction method). However, simply asking the same question repeatedly can change the way that respondents interpret the question, which can, in turn, affect the validity (e.g., Meade and Craig 2012).

Schwarz (1999) described a model of survey response that detailed how the specific questions that researchers ask can shape the answers that respondents provide. Although researchers may assume that respondents interpret survey questions literally, subtle features of the testing environment can influence respondents’ interpretations, which can, in turn, affect their responses. As just one relevant example, Strack et al. (1991) presented two similar questions to respondents: one question about happiness and one about life satisfaction. When these two questions were presented as if they were part of two separate questionnaires, respondents provided very similar responses to the two questions. However, when the questions were presented as part of the same questionnaire, respondents appeared to interpret the questions differently, and provided more discrepant responses.⁶ Specifically, Strack et al. (1991) argued that in the condition in which the two questions were presented as a part of the same questionnaire, respondents assumed that the researcher would not ask the same question twice, and therefore interpreted the two questions in subtly different ways. Schwarz (1999) noted that findings like this suggest that respondents rely on conversational norms to interpret questions; and one important norm is that people do not ask for redundant information.

It is not difficult to see how such a conversational norm could affect experiential measures; i.e., measures that ask respondents to answer the same question over and over again across multiple experiences. It is possible that by presenting questions in this way, respondents interpret the researchers’ focus as being on the change that occurs from moment to moment, rather than on conditions that remain stable. Thus, respondents may be more likely to report on this change, resulting in exaggerated reports of variability.

At least some evidence exists to support this possibility. In a slightly different context—i.e., an investigation into personality variability—Baird and Lucas (2011) tested whether asking the same personality questions multiple times led respondents to report more variability than they otherwise would have. All participants first provided a global evaluation of their personality in an online questionnaire. Approximately one week later, they appeared in person to complete an additional set of questionnaire measures. Participants were randomly assigned to take part in one of two conditions. In the single-role condition, participants were asked to answer a set of 50 different questions about their personality in a single role (e.g., what their personality is like as a friend, as a worker or as a student). In the multiple-role condition, participants were asked a set of 10 different questions five separate times, once for each role. The hypothesis was that the variability in the responses that participants provided (as assessed by comparing each role-specific score to the general score provided a week earlier) would be greater in the multiple-role condition than in the single-role condition.

The results supported this hypothesis. Compared to the personality scores in the multiple-role condition, the personality scores in the single-role condition were more strongly correlated with and less discrepant in an absolute sense from the global scores. Thus, it appeared that compared to the reports they gave when only asked about a single role, simply asking the same question multiple times led respondents to exaggerate the variability of their personality. Moreover, participants provided responses in the multiple-role condition that were more in line with stereotypes of how people typically behave in specific situations. For instance, participants reported greater differences between the ratings of what their extraversion was like in general and what their extraversion was like as a friend in the multiple-role condition than in the single-role condition. Although it is not clear whether such effects occur in standard experiential measures of well-being, Lucas et al. (2021) reported suggestive evidence that this could be the case, especially for the day reconstruction method. Thus, the unique features of experiential measures—especially the fact that these measures rely on repeated questions—may affect the validity of the scores that result when these measures are used.

These results do not directly address the validity of experiential measures of SWB. However, like the judgment model studies of global reports, they provide insight into the processes by which participants create and report responses to the questions that are posed. Baird and Lucas’s (2011) study suggests that simply asking the same question over and over may communicate to respondents that the researchers are more interested in change than in stability. This may, in turn, lead to responses that exaggerate the impact of changing situational factors on the resulting aggregate ratings. Indeed, experiential measures like those obtained from the day reconstruction method are often more strongly correlated with situational variables assessed at the same time and using the same method than they are with other major life circumstances that are often correlated with more global reports (Lucas et al. 2021). Although it is possible that this is because these situational factors truly do affect momentary experiences, it is also quite possible that these associations appear because the use of experiential measures makes these situational factors especially salient to respondents. Again, more work that directly evaluates the validity (and the relative validity compared to global reports) of experiential measures is needed.