JUNKYARD RAVE

March-April 2023

JUNKYARD RAVE (Realtime Audio Variational autoEncoder: https://arxiv.org/abs/2111.05011) is an interactive, mechatronic, welded sculpture that invites participants to musically explore latent space in real-time using hand-fabricated controls parametrically mapped to multiple RAVE neural network models.

Frequently, AI is used to produce transactional outcomes; however, JUNKYARD RAVE empowers participants to co-perform with AI. The sculpture becomes an augmented musical instrument. Through embodied action–pushing eight foot high levers and switching flips–the audience and neural networks become co-navigators of sonic latent space. As audio is fed into the RAVE models, data is recycled and transformed into a limitless variety of new sonic artifacts. The models enable fast and high quality neural audio synthesis as the participants engage with the unique controls. Vibrant Arduino-programmed LEDs provide additional real-time visual feedback for control manipulation. The sculptural frame was welded and fabricated in-house from recycled metal sourced locally from local junkyards in Everett, MA.

Conceptually, JUNKYARD RAVE formed from the desire to create an interactive, tangible sonically interactive sculpture that invited individuals to explore the intangible latent space of AI Models as a musical instrument. In current applications of AI, the user puts in a singular input and receives a singular direct output. Examples of this include recent popular interfaces such as DALLE-2 which generates AI visual artworks from user text inputs. Contrary to this static request-receive approach, the magic of music is in performing and expressing with an instrument. Through JUNKYARD RAVE we wish to bring the agency of music making and intentionally “reaching into” an AI model to our audience.

Standing at eight feet high, six feet wide, and four feet deep, JUNKYARD RAVE is a one-of-a-kind sonic exploration and interactive sculptural installation. It contains custom tactile and musical interfaces through the form of extensive controls (detailed in technical outline below) that produce analog signals in real time that musically manipulate musical outputs from the RAVE models. Participants get to sonically saturate themselves in their live musical performance with AI. The unique aesthetic sculptural framework is designed to be a kind of metal jungle–full of life, curiosity, and unique possibilities for interaction. We invite users to play; create; perform; or just be with this welded machina.

Originally fabricated for Arts Around the Lab at MIT, we assembled a multidisciplinary team of engineers and artists to construct JUNKYARD RAVE. Recycled metal from the Everett, MA junkyard, was MIG welded and cut to create the sculpture’s core aesthetic design. Junkyard metal scraps were re-imagined and altered with additional hardware to become housing for switches.

Three long steel pipes were mounted onto a custom wood and rebar electro-mechanical lever system. Each mechanical lever is wired with a potentiometer that produces an analog signal. The analog control outputs are passed through Arduinos to a laptop computer running Max MSP, a digital audio workstation (DAW). This DAW contains multiple custom RAVE models. The analog signals become musical composers of the musical, AI-Augmented DAW audio outputs.

A set of large speakers resides next to the sculpture, so participants can hear their musical performance real-time. A rotary switch allows users to switch between the different RAVE models, thereby changing the entire sonic environment. Strips of LEDs are attached to the underside of the sculpture, illuminating the floor with a slight neon glow. The LEDS are also connected via arduino to reflect the real-time changes in the three lever’s positions.

Sculpture Materials: Recycled steel; Plywood; 3D Printed PLA; Arduinos; Potentiometers; pre-fabbed electronic switches; custom neural networks

Components

Levers

Three custom “T” shaped levers, one for each eight foot pole, were welded with steel and housed in milled wood.
A coupler connects the T lever to the potentiometer.
Each eight foot pole slides onto the rebar lever, allowing for easy dismantling of the system.

Solidworks CAD of rebar and wood of custom mechanical lever design with attached (orange) potentiometer housing to generate analog signals from the levers.

One of three physically fabricated actual lever systems for JUNKYARD RAVE.

Arduino

The Arduino read analog signals from the potentiometers and switches in order to control the RAVE model. This is possible by mapping the signals from the range of motion to the range of latent space parameters in the RAVE model. An arduino board connects the controls via soldered wires to a laptop which then mitages the analog signals and translates them into electro-acoustic outputs via the MAX MSP digital audio workspace.

Electronic boards interfacing between the mechanical controls and computer running MAX MSP.

RAVE Neural Network Models

We custom trained six RAVE models on six distinct sonic datasets to produce six different musical models representing a wide range of sonic worlds (percussions, synthesizers, orchestral, environmental sounds etc.) Each model took one week to train, and is able to be manipulated and augmented in real time by the user through their physical, embodied usage of the controls mounted on the sculpture’s base.

The controls manipulate parameters for the latent space so the users are - in real time - manipulating the model inputs and therefore transforming the audio outputs.

Vibe Switches:
Switching between different inputs ranging from musical and atmospheric soundscapes

Sonic World Switches:
A rotary switch to select between the different AI models producing a wide collection of novel sounds.

Exploration Levers:
Three control levers to navigate the high dimensional latent space of the AI model. This creates a wide range of musical outputs shaped by the way you perform with the exploration levers.

Artificial Intelligence

Custom Trained RAVE models for various sonic worlds (percussion, voices, industrial and orchestral sounds)
Realtime Timbre transfer using the various autoencoder models to translate microphone and other live interactions from the audience.
Latent Space Exploration: mechatronic interactions to navigate and sample from the model latent space with a tangible gesture.

Music

Several loops of Electronic dance music and Atmospheric soundscapes were produced to act as the starting points for the interactive JUNKYARD RAVE experience.
A MAX MSP Interface is used for realtime audio synthesis and interaction control with the Arduino component.

Artists

Nick Bira; Anna Buchele; Marcus Edmonds; Chris Lock; Manaswi Mishra; Paris Myers; Erik Strand; Saina Rezvani; Serge Vasylechko