Social interactions influence people’s feelings and behavior. Here, we propose that a person’s well-being is influenced not only by interactions they experience themselves, but also by those they observe. In particular, we test and quantify the influence of observed selfishness and observed inequality on a bystanders’ feelings and non-costly punishment decisions. We developed computational models that relate others’ (un)selfish acts to observers’ emotional reactions and punishment decisions. These characterize the rules by which others’ interactions are transformed into bystanders’ reactions, and successfully predict those reactions in out-of-sample participants. The models highlight the impact of two social values-‘selfishness aversion’ and ‘inequality aversion’. As for the latter we find that even small violations from perfect equality have a disproportionately large impact on feelings and punishment. In this age of internet and social media we constantly observe others’ online interactions, in addition to in-person interactions. Quantifying the consequences of such observations is important for predicting their impact on society.