The document describes a project to sonify the movie Kung Fu Panda by mapping pixel values from frames of the movie to musical notes and sounds. Pixels were mapped to specific audio files based on their red, green, and blue values and volume was scaled based on pixel brightness. Scenes from the movie were divided into measures and mapped to different character themes to add texture. Computational choices like nested for loops and conditionals were used to place audio files on the soundtrack based on pixel values. The process involved exploring ideas, converting pixel data to arrays, and creating the musical composition through pixel to audio mapping and assigning character themes. Problems encountered included the difficulty of expressing ideas musically and adhering to code structure while focused on

1. Kung-Fu Panda
A Musical
Apurva Gupta
MS-HCI

2. Brief
The aim of the sonification was to create a musical representation for
the movie – Kung Fu Panda.
The most direct way was to map each pixel with a sound and let the composition tell the
story. To overcome the monotony, brightness of the pixel was explored and played around
with using volume gain. Based on how high the value of RGB is, the sound was assigned
accordingly.
The Asian context of the movie influenced the choice of music used.
For recording, the sound was created by striking a half-filled, full filled and empty metal
container. These sounds were mapped to value of Green color.
To add texture to the composition, the total of 160 measures were sub-divided based on
the scenes in the movies and leading cast in that scene. Beat compositions were taken and
assigned to each character of the movie. Based on its appearance, would the composition
play.

3. Computational Choices
• To achieve the musical composition, the data structure used was 2
dimensional lists to store the RGB values of the pixel. 1-D arrays was
also used to store audio files
• An extensive use of “for loops” and “conditionals” was made.
Nested for loops was used to place an appropriately mapped audio
onto the track. Conditionals were used to compare the values of RGB
and place the media file accordingly.
• Fitmedia, MakeBeat, SetEffect were used in abundance to place audio
onto track, create beats or set volume gain or pan effect.

4. Process
Explore Convert Create

5. EXPLORE
Various ideas were explored and various computational methods were tried. Finally
the idea to sonify a movie barcode was chosen and elaborated upon.

6. The movie Kung Fu panda was chosen because it was one of my most
favorite movies and I am always excited to work around it in any form.
Also, I was well acquainted with the storyline and characters of the
movie.

7. Story-Line

8. Movie Barcode
Based on the scenes of the movie

9. CONVERT
The 1280X480 pixel barcode was scaled down to 160X60 pixels. From the new image the
pixel values for the middle row “image[x][30]” was extracted and saved in an array.
The array was 2 dimensional which has 160 elements with 3 sub-elements each. The array
was obtained using the PIL (Python Image Library) Library of Python.

10. barcode = [[83, 71, 55], [94, 64, 38],[133, 74, 40],[172, 101, 71],[115, 67, 44],[61, 49, 33],[63, 63,
55],[71, 60, 58],[70, 59, 53],[79, 87, 66],[42, 66, 40],[28, 64, 62],[1, 68, 74],[76, 84, 71],[100, 71,
37],[105, 92, 60],[125, 116, 83],[85, 85, 51],[95, 98, 67],[99, 99, 73],[66, 58, 35],[90, 84, 60],[68, 75,
42],[101, 123, 84],[142, 163, 107],[100, 112, 76],[93, 93, 67],[86, 80, 46],[91, 80, 48],[107, 95, 81],[50,
39, 45],[13, 5, 18],[7, 11, 14],[7, 0, 7],[12, 13, 15],[25, 31, 17],[65, 61, 36],[55, 56, 25],[31, 39, 18],[38,
31, 25],[40, 23, 13],[43, 30, 22],[43, 30, 24],[41, 24, 16],[42, 22, 13],[39, 29, 28],[28, 37, 52],[14, 39,
69],[22, 43, 60],[26, 44, 64],[14, 31, 51],[24, 35, 57],[37, 44, 63],[21, 24, 39],[10, 10, 20],[14, 13, 19],[11,
8, 25],[15, 13, 26],[25, 27, 42],[14, 23, 52],[5, 19, 56],[10, 17, 36],[70, 59, 41],[126, 101, 47],[143, 113,
61],[85, 76, 37],[81, 34, 14],[114, 46, 27],[71, 44, 23],[43, 36, 28],[69, 37, 22],[94, 56, 7],[29, 42, 35],[38,
41, 32],[73, 47, 20],[56, 38, 24],[8, 35, 62],[14, 47, 78],[30, 52, 73],[9, 54, 93],[59, 48, 42],[63, 45,
33],[54, 32, 18],[55, 41, 32],[22, 23, 25],[14, 28, 39],[9, 27, 39],[31, 45, 54],[26, 38, 52],[25, 21, 22],[70,
45, 38],[61, 27, 25],[59, 32, 39],[53, 41, 41],[116, 115, 87],[111, 115, 64],[71, 87, 76],[76, 63, 46],[127,
90, 63],[136, 98, 62],[140, 104, 70],[126, 87, 56],[124, 88, 52],[124, 97, 52],[64, 66, 61],[59, 64, 58],[98,
103, 99],[91, 96, 92],[72, 78, 76],[75, 81, 79],[107, 116, 113],[88, 97, 94],[88, 65, 31],[92, 55, 28],[64, 55,
22],[51, 65, 30],[51, 46, 24],[39, 47, 24],[25, 47, 24],[50, 46, 34],[42, 39, 22],[13, 16, 21],[16, 19, 26],[38,
34, 25],[33, 29, 18],[52, 55, 60],[38, 42, 41],[52, 50, 29],[52, 57, 34],[2, 19, 29],[13, 31, 35],[15, 24,
19],[24, 42, 62],[22, 69, 79],[10, 53, 43],[20, 30, 39],[54, 42, 18],[90, 71, 39],[99, 74, 34],[92, 70, 29],[74,
61, 27],[85, 76, 47],[96, 79, 49],[96, 72, 38],[84, 64, 40],[86, 76, 66],[108, 88, 55],[88, 63, 32],[147, 113,
68],[157, 116, 54],[101, 79, 40],[99, 82, 38],[52, 63, 31],[102, 68, 30],[107, 30, 4],[51, 24, 3],[71, 41,
30],[52, 34, 10],[35, 2, 0],[91, 88, 99]]
1280 X 480
160 X 60

11. CREATE
This section enumerates how the sonification of the pixels was achieved

12. Mapping Audio
• Step 1:
Find maximum and minimum values for R, G, B color values
• Step 2:
Segment into 3 sets based on their range (R, G,B each)
• Step 3:
Map 9 audio files 3 each for R, G, B based on the sets
The selection of audio files was such that the 3 chosen audio files for each
color belonged to the same family of sounds.
Red: Gong
Green: Metal Strike (Recordings)
Blue: Drums

13. Mapping Volume Level
• Step 1:
Find maximum and minimum values for R, G, B color values
• Step 2:
Scale min and max values of each color to -10 and 12
• Step 3:
Using Linear equation to find respective value of volume gain for each
color

14. Red
Green
Blue
Volume
Mapping

15. Adding Texture
For this, an 8X4 matrix was made and scenes were mapped to the lead
cast in the scene. Each character was allotted a beat composition which
was played according to the characters appearance in the scene.

16. MEASURE PO FURIOUS FIVE TAI LUNG SHIFU/OOGWAY
BEGIN 1-20
TOURNAMENT 21-40
OOGWAY’S WISDOM 41-60
ENTERS TAI LUNG 61-80
TRAINING 81-100
BRIDGE 101-120
FIGHT 121-140
END 141-160
*
* : Ideally should be mapped with repetition of bright pixels vs dark pixels

17. Tracks for Oogway / Shifu

18. Tracks for Po

19. Tracks for Furious Five

20. Insights/Learnings/Problems
• The expression of ideas to music is difficult
• I could work my way through the code but the piece didn’t sound as
pleasing as expected.
• Its hard to adhere to naming conventions and comments while
“engrossed” in writing the code but it is the ABSOLUTE necessity to
save yourself from wasting time (and frustration!!) later.
• Its fun to be able to morph colors into numbers and numbers into
sounds.
• Indentation: The flexibility of the language can sometime make you
pull your hair and bang into the screen.