Color, Music, Emotion, and Synesthesia
Research on cross-modal mappings between music and color in non-synesthetes revealed systematic cross-modal associations and strong evidence for the role of emotion as a mediating factor in these associations. We investigated whether synesthetes (music-color synesthetes as determined by the Eagleman Synesthesia Battery) have similar cross-modal associations between dimensions of music and dimensions of color, and if emotion mediates their cross-modal associations. Synesthetes picked the three colors that were the most similar (and later the least similar) to their synesthetic colors while listening to thirty-two controlled single-line melodies that varied in tonality, tempo, note-density, register, and timbre. They also rated each melody and each color along six emotion dimensions (Happy-Sad, Angry-Not Angry, Agitated-Calm, Active-Passive, and Harmonious-Disharmonious). Results indicated that synesthetes’ cross-modal associations were mediated by emotion to some degree, because of correlations between the emotional ratings of the music and the emotional ratings of the colors the synesthetes chose to match their synesthetic colors on some dimensions (.58 for Happy-Sad, .22 for Active-Passive). In addition, synesthetes tended to choose lighter, yellower, and more saturated colors when matching their synesthetic responses to the melodies in the major mode, and darker, bluer, and less saturated colors when matching their synesthetic responses to the melodies in the minor mode. These cross-modal associations are similar to those of non-synesthetes. Also, synesthetes tended to choose darker, more saturated colors when matching their synesthetic responses to the cello melodies, when compared to the colors that they chose when matching their synesthetic responses to the piano melodies. Finally, when asked to select the colors that were most emotionally compatible to the music, correlations between the emotional ratings of the music and the emotional ratings of the colors indicated that synesthetes could match colors to music with similar emotional content (.81 for Happy-Sad, .83 for Active-Passive, .61 for Strong-Weak).
Cross-Modal Sound to Sight Associations with Musical Timbre in Non-Synesthetes
Previously our lab has found that both complex classical music and two-note musical intervals have consistent cross-modal mappings to dimensions of color, and that these associations seem to be mediated by the shared semantic and emotional meaning of colors and music (Palmer & Schloss, 2013; Griscom & Palmer, VSS 2012). In a new set of experiments, we asked participants to indicate what colors, edge contrasts, and onset dynamics "went best" with the sound of 17 different common musical instruments (violin, piano, marimba). We found evidence for emotional mediation in particiant color choices, such that they tended to choose colors whose emotional quality was similar to the rated emotional quality of the sound (e.g., "happy" looking colors, such as saturated yellow, were chosen for "happy" sounding instruments, such as a harpsichord). Systematic sound-to-sight correspondences were also evident in non-synesthetes' choices of "best fitting" edge contrasts (low to high spatial frequencies) and dynamic onsets (temporal dynamics of contrast) for circular shapes, but these were better explained by correspondences among low-level perceptual features such as attack time and spectral brightness than by any kind of emotional mediation. Additionally, we tested a set of stimuli with multiple timbres and found that the visual matches with these audio were well predicted by a simple combination of the visual matches for their component timbres.
The Color of Musical Sounds: Color Associates of Harmony and Timbre in Non-Synesthetes
Previous studies have shown that the tempo and harmonic mode (major vs. minor) of different pieces of classical music produce consistent cross-modal associations with dimensions of individual colors and that these associations are largely mediated by the shared emotional content of the colors and music (Schloss & Palmer, VSS-2008; Xu et al., VSS-2011; Palmer et al., VSS-2011). In the present study we extended this line of research by investigating cross-modal associations between two-color displays and musical sounds that varied in instrumental timbre and the harmonic structure of 2-note intervals and 3-note chords. The color stimuli consisted of color pairs chosen from the 37 colors of the Berkeley Color Project and the sound stimuli consisted of synthesized notes, two-note intervals, and three-note chords, using instrumental timbres derived from Grey's (1977) timbre space. Participants were asked to perform a 2AFC task to indicate which of two color pairs they felt was more strongly associated with the presented musical sound. They also rated the sound and the color pairs separately on each of five emotional dimensions (happy/sad, angry/clam, active/passive, strong/weak, and harmonious/disharmonious). The probability of choosing a given color pair was found to be strongly related to the fit between the degree of harmony of the musical sound and the difference in the degree of harmony between the two color pairs presented on that trial (i.e., harmonious color pairs were chosen to go with harmonious sounds and disharmonious color pairs to go with disharmonious sounds). Instrumental timbre also showed systematic effects on participants' cross-modal choices of visual displays that differed in both color and spatial characteristics.
Color, Music, and Emotion
Arnheim (1986) speculated that different aesthetic domains (e.g., color and music) might be related to each other through common emotional associations. We investigated this hypothesis by having participants pick from among an array of 37 colors the five colors that went best (and later the five that went worst) with each of a set of musical selections that varied in composer, tempo, and mode (major/minor). They also rated each musical selection and each color for its emotional associations (happy-sad, lively-dreary, strong-weak, angry-calm). For both orchestral music and solo piano music, systematic mappings were found between the dimensions of color and music: faster music and major mode were associated with lighter, more saturated, yellower colors, whereas slower music and minor mode were associated with darker, desaturated, bluer colors. These mappings appear to be mediated by common emotional associations, because the correlation between emotional ratings of the musical selections and emotional ratings of the colors chosen to go with them were extremely high (.90 to .98) for all emotional dimensions studied (e.g., people picked happy colors to go with happy music and dreary colors to go with dreary music). Further studies using better-controlled musical stimuli (unaccompanied theme-and-variation melodies by Mozart) dissociated effects due to instrumental timbre (piano/cello), register (high/low pitch), and note density (quarter-note theme vs. eighth-note variation), as well as tempo and mode from the specific influences of different melodic and harmonic structure in the earlier studies. The mediating role of emotion was established by obtaining analogous effects when people picked the colors that went best (and worst) with faces and body poses that expressed emotions (happy-sad and angry-calm). Similarly high correlations were obtained when the emotional ratings of the faces/gestures were compared with corresponding emotional ratings of the colors chosen to go with them.
The Color of Music
We investigated the relations among color, music, and emotion for the 37 colors of the Berkeley Color Project: saturated, desaturated, light, and dark shades of red, orange, yellow, yellow-green, green, blue-green, blue, and purple, plus white, black, and 3 grays. To study color/music relations, participants viewed all 37 colors while listening to 18 orchestral selections in major and minor keys by Bach, Mozart, and Brahms that had slow, moderate, and fast tempos. For each selection, participants chose the five most-consistent and the five least-consistent colors. Across composers, faster tempos and major keys were both associated with brighter, warmer, more saturated colors than slower tempos and minor keys. To determine whether affective responses might mediate these color/emotion associations, we also studied the relation between the color samples and emotion words and between the musical selections and emotion words. Participants produced color/emotion associations by rating the consistency between 16 emotion words and each of the 37 colors. Strong associations were found between many emotion words and colorimetric dimensions: e.g., happy, lively, and enthusiastic with light-warm-saturated colors; sad, dreary, and unenthusiastic with dark-cool-desaturated colors; strong and aggressive with warm-saturated colors; and weak and shy with cool-desaturated colors. Participants also produced music/emotion associations by rating the consistency between each emotion word and each of the 18 musical selections. We found a strong link between the affective response to musical selections and the affective response to the corresponding chosen colors. Specifically, there was a strong positive correlation between the ratings of emotional associations to the 18 musical selections and the ratings of emotional association to the colors people chose as most/least consistent with the same musical selection. This finding suggests that affective response may mediate the relations we found between color and music.
All of the above research is funded in part by NSF Grant BCS-0745820 to Stephen E. Palmer and by a gift from Google, Inc.