Seeing Harmony!

[This was written in a hurry; I'm making some corrections.]

For years, I've wanted to devise a method to "see" chords, and harmony. Now, it seems, the effort is no longer necessary, because finally --actually several years ago, as early as 1983-- a fellow called Stephen Malinowski has invented a method that enables us to visualize harmonic relationships. His idea slowly evolved, as he interacted with various friends and other like-minded people in the computer and animation world (the history of the project is available if you click on the title of this piece) until it is now a really fabulous piece of invention.  [Added later: Mr Malinowski points out that the ideas actually came from fellow musicians, and not computer or animation people.]

Malinowski's remarkable piece of software, named Music Animation Machine "plays" MIDI files. So you enter a piece into a computer notation program like I do, for example Sibelius or Finale or Capella, or Noteworthy Composer, and the program is usually able to convert your piece into an electronic format called MIDI (which was invented about 20 years ago, and is the modern version of the paper tape that Pleyel Pianos generated. MIDI can actually do a lot more than paper tape, but it does it at the cost of some things that the piano rolls did really well) which the MAM then plays.  MIDI stands for Musical Instrument Digital Interface.

The Music Animation Machine reads your piece (in MIDI form), and while playing it, interprets the harmonic relationships in beautiful graphical form. This video of a sonata by Domenico Scarlatti (a major composer whom I have sadly neglected here) illustrates the basic visualization mode of the machine.

It has several different visualization modes, the simplest being the "Piano Roll" style, in which the music is presented as little bars for each note. The longer the note, the longer the bar; the higher the note, the higher the bar. So, instead of the music notes on the staff, you see these bars creeping across the screen from right to left. As you can imagine, this setup is more conducive to representing melody than harmony.

Another of the visualizations is based on note intervals. (Everyone knows "octave" (8^ve), the musical distance between, say, Middle C, and the next C above or below it. Similarly, every pair of notes define an interval, ranging from Unison (two coincident notes), which is zero, to however much you like. The important intervals are: the Fifth (5^th), between C and G above it, the Fourth (4^th), between C and G below it; and the Major Third (Maj 3^rd), between C and E above. Finally, there is the Minor Third (Min 3^rd), from E up to G. Or D up to F, for that matter.)

Malinowski decided to depict the notes that were being sounded at any moment with some shape, and then connect these notes with lines; blue for 5^ths, green for 3^rds, and some other cool color for 4^ths. The other intervals, which all sound more harsh, such as 7^ths and 2^nds, were depicted with red and yellow lines.

The illustration at above right is a composite of several screen shots.

At the top is a blended picture of two piano-roll views; the left part shows the bars colored according to the instrument (the entire melody is the same color for a given instrument or part, so that you can trace intercrossing lines of melody more easily), and the right half has the bar colored according to the note; Blue for C, or whatever.

Next below that is a fascinating view of one single chord, with all its notes lit up. This setup represents each note as one of a honeycomb pattern of note labels, surrounded by six of its harmonically closest neighbors, with the silent ones simply left dark and the sounding ones lit up. This piece of mine has five parts, so you see five of the discs lit up: 1, 2, 4, and 6, with some doubling, obviously. (In case you want to know, this is the minor seventh on the "supertonic", the note next up from the base note-- "Ray, a drop of golden sun." Or rather, the chord Re, Fa, La, Do.) A chord with closely related notes in it will be clustered tightly together; a wild and crazy chord will have tentacles all over. Well, at least out in the second concentric circle. For example, the chord B, D, F-Sharp, A-Flat will look pretty wild, I'd bet. [7, 2, 4#, 5# in this diagram.]

Objectively, I suppose, this would be the most illuminating view of the music, once you got used to it.

But Malinowski has yet another fascinating representation. He indicates notes on a rotating 12-tone scale of the circle of fifths --just the circle C, G, D, A, E, B, F#, C#, etc opened up and copied over and over-- with longer notes represented by large diamonds, which get thinner as the note fades away, and each note is connected to every other note by lines, whose colors represent the intervals! So we have both pitch and intervals here.  The Circle of Fifths is shown at right; click to enlarge.   (This has a mathematical analogy that is fascinating. For various reasons, it is convenient to open up a circle into a line. Trigonometry students are familiar with the correspondence between 1/2 Pi and 90 degrees, for instance. Similarly, folding the entire piano keyboard into a circle, or opening out the circle of fifths into a line, or a cylinder of fifths into a plane are all familiar procedures for at least some mathematicians.  The diagram was created using Geometer's Sketchpad, currently owned and operated by Key Curriculum Press.)  [Please feel free to uses my Circle of Fifths ad libitum, except that you may not sell it to anyone, and must tell anyone to whom you give it that he or she should not sell it either, and so on until the end of time.  I do not insist on being identified as its creator, since better minds than mine were involved in its invention, even if the idea is obvious.]

Look at the last two images in the composite. The upper one is the opening chord, and let's pretend the piece is in C major. That opening chord has 3 C's, a G and an E. The 3 C's are all on the same horizontal line, the G is right above, on the very next line. (Why? Because G is [one of the] closest note[s] to C harmonically, though not interval-wise.) And some four notes away is our friend E, connected to the C's with green lines. The E makes two kinds of thirds; the major third is represented with a bluey-green line, it seems to me, and the major third with a yellowy-green line, but I could be wrong. They're both green, anyway. The octaves and the fifths and fourths are represented by blue lines.

The very next image, the lowest in the composite, shows the chord C, C, C, F#, A. As before, the lines between the 3 C's are blue. A to C, and F# to A are both thirds, both minor thirds, incidentally, and they're shown in green. But the flaming red lines indicate the C to F# interval, which is not far on the keyboard, but harmonically the notes are antipodal. So we see three sets of red lines, between each of the C's and the F#. (To modern ears, the C to F# interval is not so horribly dissonant, but at one time it was called the Wolf, or the Devil in Music: Diabolus in Musica.)

Harmony, though, is a tiny bit more than just the sum of the intervals involved, and to do it justice probably needs a representation that's more complex. But complexity has to start somewhere, and it looks as though we're never going to have to turn back from what Stephen Malinowski has achieved with his Music Animation Machine.

Initially, there was a very primitive video embedded here.  I have replaced it with a sharper (HD 720) video made in 2013, with the video captured using updated software.  This also uses far better sound samples.  I hope you like it!  Many thanks to Stephen Malinowski, as always.