New York Philharmonic - Foreign Bodies

It was a stunning finish to Esa-Pekka Salonen’s run as the New York Philharmonic’s composer in residence. His final program in David Geffen Hall included Salonen’s own work, ‘Foreign Bodies,’ featuring a live video installation designed by Video Artist Tal Rosner. A disguise gx 1 media server drove the dynamic visuals, which responded to prismatic changes in the music with imaginative graphics and bold colour schemes and used a live feed of the orchestra as the basis for more patterns.

‘Foreign Bodies’ opened the program: “we set the tone for a groundbreaking evening. Even if people didn’t understand how it worked, they could see the relationship of the orchestra, the music, the visuals and the audience. It was an art piece in response to an existing art piece, like a live ‘Fantasia 2018’” comments Tal.

Tal has been creating visuals for classical music performed by various orchestras for 15 years. “Rock and pop music always have visuals, but classical has not had it,” he notes. “It’s very exciting for me to create for classical music; it’s very close to my heart. I’ve worked on everything from Stravinsky to Benjamin Britten.”

Tal has partnered with Salonen before animating a previous piece for the composer-conductor. “‘Foreign Bodies’ was already in existence but he wanted to do a new spin on it, to challenge himself with a multidisciplinary approach. To make the music come alive I wanted to not only respond live to the orchestra, but also generate dynamic images from the orchestra.” Tal’s hyper-synched video interpretation of the orchestral piece drew inspiration from action paintings and Expressionist abstractions. By animating textural shifts and instrumental gestures and by manipulating live-feed images of both the players and the conductor on stage, the performance became a sort of closed circuit where the flow of sound and projected images were intertwined. 

“We only had two actual rehearsals with the orchestra; we were very tight for time,” Technical Director/Video Software David Shepherd recalls. “Everything else was done with recordings. Before we left for the show we needed to build as much as possible.” He praises disguise’s “extremely tight integration with Notch,” which they used extensively. “When we develop a Notch look we know it will work with the media server, that it will run smoother and faster than it ran on the laptop.”

The orchestra was initially seen with no visuals displayed on the giant projection screen behind them, then visuals slowly began being introduced as the orchestra continued to play. “Layered on top of that were treatments of that particle data: the particles took the colors from the stage then created their own look.  Although we had the freedom to abstract the movement viewers still felt that it followed the actual movement.  Trumpet moves didn’t look like a trumpet, but the audience could still sense its specific movement.”

David explains that in pre-pro, “we used pre-exiting clips of live orchestras playing and we developed some very simple looks and rendered them out and into After Effects to do something quite a bit more interesting. Then we took them back to Notch for the dynamic effects.”

During the program, Tal and David used two 4K cameras facing the orchestra to create eight HD shots for the live feed. “We took those eight outputs into a 12x12 matrix, which disguise was switching live,” David says. “We keyframed on the disguise timeline to switch between the eight camera shots and focus on part of the raster.  That night we had almost the equivalent of eight cameras. We had 140 cues: eight camera views, a lot of looks created in Notch plus some pre-rendered stuff.”

According to Tal, “This was a very Notch-heavy piece. At times we were running a very intensive Notch layer with two pre-rendered video layers and two 4K camera layers coming in.”