Augmenting Creativity with AIFlow Machines
Making music like
never before
Flow Machines, an AI-assisted music-production project at
Sony Computer Science Laboratories Inc. (Sony CSL), aims to expand the creativity of creators in music.
In the process of readying the project for full-scale social implementation, Sony designers have played integral roles
every step of the way: from developing the concept to designing the user interface (UI) for the Flow Machines
Professional AI-assisted music-composing tool and cultivating a community of users.
The result is Flow Machines, a gateway to new musical frontiers where artists and AI co-create new, exciting sounds.
Charting a clear course to
the project’s end goals
Embracing the idea that AI could help music creators expand their creative capabilities, Sony CSL engineers on the Flow Machines project have used machine learning and music-analysis technologies to develop an AI-driven tool for automated composition. The project team eventually reached out to Sony designers to help prepare the tool for rollout with an overhaul of the UI screens, and the designers got to work by establishing a clear idea of the project’s core essence: what the tool was supposed to be for music creators.
With some of the designers coming from a musical background, the design team drew on its own experience and talked with a variety of pro musicians about what kinds of tools they were looking for. That process produced valuable input, including the common threads of wanting "lots of ideas to keep new sounds coming" and ways to "jam with musicians from other genres and styles to stimulate creativity." In addition to gathering insights from musicians, the designers also discussed the tool’s target identity with Sony CSL personnel.
The conversations brought a basic course for the project into view. In developing Flow Machines, the group set out to teach musical styles and techniques from across the genre spectrum to the Style Palette*1 machine-learning model, use that foundation to build AI musicians with a far-reaching command of every genre, and fuse everything together into a dynamic, co-creative partner to work alongside users. The team also laid out the "Co-write with AI Musicians" concept, defining the Flow Machines AI system not as a tool that simply writes songs automatically but as a collaborative songwriting partner. Having solidified that identity and set clear project goals, the designers got to work on their initial task: creating a UI design for Flow Machines Professional (FM Pro), an AI-assisted music-composing tool.
*1: Style Palette is a machine-learning model that analyzes music data. FM Pro includes machine-learned Style Palette presets for over 100 different genres, including pop, rock, and hip hop. Users can also create their own Style Palettes by having the model learn original melodies.
Injecting more creative energy
into the songwriting experience
The FM Pro app, a plugin for use in DAW software, uses Style Palettes in myriad genres to compose melodies, chords, and bass lines that match the user’s sonic preferences, giving music creators a plethora of options to play around with. As the design team started crafting the UI for the app, they decided to downsize the FM Pro screen to keep it from crowding the DAW display space and craft a design and color scheme that would mesh with DAW screens for pitch-perfect visual harmony.
The biggest focus for the team, however, was making the interface easier to connect with—an aim they decided to pursue by giving the Style Palette interface a "persona" that would be able to resonate with creators and get their creative juices going on a realer, more emotional level. Instead of going with the conventional method of rendering the AI as an anthropomorphic avatar, the designers wanted to capture the unique personalities of the various Style Palettes in a way no one had ever tried before. With the help of Sony CSL researcher Alexis André, the team worked up an interface that plays an auto-generated moving visual behind the Style Palette to reflect the AI’s "personality"—a kinetic visual complement, changing with the energy and presence of the corresponding sounds in real time to make the songwriting experience like co-creating with a kind of living "sound being" right there in the workspace.
For the designers, an even more important point was deciding how to present FM Pro’s multitude of Style Palette presets to users in an intuitive, effective way. The team knew that just listing out all the presets from top to bottom would do very little to inspire music creators or spark their creativity. Instead of that approach, they wanted a preset interface that would grab users visually and immerse them in the creative experience. What they eventually came up with was a 3D-map framework, one that arranges the presets in the three-dimensional selection space according to tempo and complexity. The setup lets users explore the different options in a more intuitive way, much like browsing different areas of a preset market. "I wonder what the song ideas over here sound like," users might think as they head around to various areas in search of good matches for their song ideas. In addition to making good fits easier to locate, the space interface also fosters discovery; while finding something new in a basic list might just be a matter of random chance, the 3D preset space in FM Pro lays things out in a way that sparks users’ curiosity—they might be listening to a certain melody when a preset one spot over catches their eye, naturally piquing their interest about how it might be a shade different.
The team also wanted to make using the Style Palettes feel like jamming with an actual musician. Initially, the interface design required users to configure a bevy of different parameters before getting song suggestions. The design team was looking for a smoother, more natural interactivity. With the help of Sony CSL engineers, the team eventually arrived at a design where the interface pops out optimal song ideas at one press of the "Compose" button. Users can jam out certain spots in a song with the interface to try out different musical directions, too. If a user wants more suggestions for the specific phrase in the third measure of a passage, for example, they can select the third measure and click the "Compose" button again for an alternative. A session with the tool is strikingly similar to a session with an actual band, with everybody contributing ideas and tweaking parts until the song takes solid shape.
Supporting the songwriting workflow
from start to finish
There was more to the designers’ work than the UI. They also looked at the entire songwriting workflow, which actually goes beyond arranging the different elements like melodies, chords, and bass lines that FM Pro can compose. It starts before that, with creators laying the groundwork for the song in the forms of rhythm patterns and drum tones. That initial step, or the "track-making" process, has a massive impact on the song’s overall vibe. The design team wanted to help make that step less of a time-consuming conundrum for creators to get through.
That led the designers to start looking at track-making software that they might be able to integrate into the FM Pro app platform. Sony CSL Paris was already at work on research and development for technologies that fit the bill: Impact Drums, which generates a wide range of drum samples, and DrumNet and BassNet, which auto-generate rhythm patterns. Seeing the potential in the pipeline, the design team reached out to the Sony CSL side about redesigning the programs with the same tone and manner as the FM Pro screen to align the aesthetics for optimal integration as a brand. To unify the UI for the track-making element with the rest of the interface, the designers went with a circular, AI-esque search space and populated the area with drum tones and bass patterns to help users experiment with sounds more easily, streamline the songwriting process, and maximize creativity.
Partnering with creators
to explore the musical unknown
When Sony CSL floated the possibility of co-designing the website and platform for Flow Machines, too, the designers started formulating ideas for an ecosystem that would extend artists’ creativity even further. The team designed an entire platform with that expansion in mind, incorporating cloud services and social media to create a communal space for FM Pro users to post their work, interact with other users, and inspire each other. The whole idea centers on putting the full scope of the creative process in one place: from writing a song to getting it out to the ears of the world, the FM Pro ecosystem enables the creators of tomorrow to pursue their creative energies across every phase of the process without ever coming up against functional limitations or needing to look elsewhere.
Since its release, FM Pro has already carved out a sizable following among the creator community and provided the AI-driven foundations for an outpouring of new songs. The tool is just getting started on its evolutionary arc, though. The core goal of the project is a continuing one: to get more and more music creators to embrace co-creation with AI technologies and make music like never before. With their ears always open to user input, the team is committed to fine-tuning their work, taking the project even deeper, and helping creators find new, richer grooves for their musical imaginations to grow in.
（L to R) Sony Group Creative Center: Satoshi Asai, Keisuke Ito, Junichiro Sakata, Sony Computer Science Laboratories: Haruhiko Kishi