Real-time music generation using Lyria RealTime  |  Gemini API  |  Google AI for Developers

The Gemini API, using
Lyria RealTime,
provides access to a state-of-the-art, real-time, streaming music
generation model. It allows developers to build applications where users
can interactively create, continuously steer, and perform instrumental
music.

Lyria RealTime music generation uses a persistent, bidirectional,
low-latency streaming connection using
WebSocket.

To experience what can be built using Lyria RealTime, try it on AI Studio
using the Prompt DJ or the
MIDI DJ apps.

Generate and control music

Lyria RealTime works similarly to the Live API
in that it uses Websockets to maintain real-time communication with the model.

The following code demonstrates how to generate music:

  import asyncio
  from google import genai
  from google.genai import types

  client = genai.Client(http_options={'api_version': 'v1alpha'})

  async def main():
      async def receive_audio(session):
        """Example background task to process incoming audio."""
        while True:
          async for message in session.receive():
            audio_data = message.server_content.audio_chunks[0].data
            # Process audio...
            await asyncio.sleep(10**-12)

      async with (
        client.aio.live.music.connect(model='models/lyria-realtime-exp') as session,
        asyncio.TaskGroup() as tg,
      ):
        # Set up task to receive server messages.
        tg.create_task(receive_audio(session))

        # Send initial prompts and config
        await session.set_weighted_prompts(
          prompts=[
            types.WeightedPrompt(text='minimal techno', weight=1.0),
          ]
        )
        await session.set_music_generation_config(
          config=types.LiveMusicGenerationConfig(bpm=90, temperature=1.0)
        )

        # Start streaming music
        await session.play()
  if __name__ == "__main__":
      asyncio.run(main())

import { GoogleGenAI } from "@google/genai";
import Speaker from "speaker";
import { Buffer } from "buffer";

const client = new GoogleGenAI({
  apiKey: GEMINI_API_KEY,
    apiVersion: "v1alpha" ,
});

async function main() {
  const speaker = new Speaker({
    channels: 2,       // stereo
    bitDepth: 16,      // 16-bit PCM
    sampleRate: 44100, // 44.1 kHz
  });

  const session = await client.live.music.connect({
    model: "models/lyria-realtime-exp",
    callbacks: {
      onmessage: (message) => {
        if (message.serverContent?.audioChunks) {
          for (const chunk of message.serverContent.audioChunks) {
            const audioBuffer = Buffer.from(chunk.data, "base64");
            speaker.write(audioBuffer);
          }
        }
      },
      onerror: (error) => console.error("music session error:", error),
      onclose: () => console.log("Lyria RealTime stream closed."),
    },
  });

  await session.setWeightedPrompts({
    weightedPrompts: [
      { text: "Minimal techno with deep bass, sparse percussion, and atmospheric synths", weight: 1.0 },
    ],
  });

  await session.setMusicGenerationConfig({
    musicGenerationConfig: {
      bpm: 90,
      temperature: 1.0,
      audioFormat: "pcm16",  // important so we know format
      sampleRateHz: 44100,
    },
  });

  await session.play();
}

main().catch(console.error);

You can then use session.play(), session.pause(), session.stop() and
session.reset_context() to start, pause, stop or reset the session.

Steer music in real-time

You can steer music generation in real-time by sending prompts and updating
generation parameters in real time.

Prompt Lyria RealTime

While the stream is active, you can send new WeightedPrompt messages at any
time to alter the generated music. The model will smoothly transition based
on the new input.

The prompts need to follow the right format with a text (the
actual prompt), and a weight. The weight can take any value except 0. 1.0
is usually a good starting point.

  from google.genai import types

  await session.set_weighted_prompts(
    prompts=[
      {"text": "Piano", "weight": 2.0},
      types.WeightedPrompt(text="Meditation", weight=0.5),
      types.WeightedPrompt(text="Live Performance", weight=1.0),
    ]
  )

  await session.setMusicGenerationConfig({
    weightedPrompts: [
      { text: 'Harmonica', weight: 0.3 },
      { text: 'Afrobeat', weight: 0.7 }
    ],
  });

Note that the model transitions can be a bit abrupt when drastically changing
the prompts so it’s recommended to implement some kind of cross-fading by
sending intermediate weight values to the model.

Update the configuration

You can steer music generation by updating the music generation parameters in
real time. You can’t just update a parameter, you need to set the whole
configuration otherwise the other fields will be reset back to their default
values.

Since updating the bpm or the scale is a drastic change for the model you’ll
also need to tell it to reset its context using reset_context() to take the
new config into account. It won’t stop the stream, but it will be a hard
transition. You don’t need to do it for the other parameters.

  from google.genai import types

  await session.set_music_generation_config(
    config=types.LiveMusicGenerationConfig(
      bpm=128,
      scale=types.Scale.D_MAJOR_B_MINOR,
      music_generation_mode=types.MusicGenerationMode.QUALITY
    )
  )
  await session.reset_context();

  await session.setMusicGenerationConfig({
    musicGenerationConfig: { 
      bpm: 120,
      density: 0.75,
      musicGenerationMode: MusicGenerationMode.QUALITY
    },
  });
  await session.reset_context();

Prompt guide for Lyria RealTime

Here’s a non-exhaustive list of prompts you can use to prompt Lyria RealTime:

• Instruments: 303 Acid Bass, 808 Hip Hop Beat, Accordion, Alto Saxophone,
Bagpipes, Balalaika Ensemble, Banjo, Bass Clarinet, Bongos, Boomy Bass,
Bouzouki, Buchla Synths, Cello, Charango, Clavichord, Conga Drums,
Didgeridoo, Dirty Synths, Djembe, Drumline, Dulcimer, Fiddle, Flamenco
Guitar, Funk Drums, Glockenspiel, Guitar, Hang Drum, Harmonica, Harp,
Harpsichord, Hurdy-gurdy, Kalimba, Koto, Lyre, Mandolin, Maracas, Marimba,
Mbira, Mellotron, Metallic Twang, Moog Oscillations, Ocarina, Persian Tar,
Pipa, Precision Bass, Ragtime Piano, Rhodes Piano, Shamisen, Shredding
Guitar, Sitar, Slide Guitar, Smooth Pianos, Spacey Synths, Steel Drum, Synth
Pads, Tabla, TR-909 Drum Machine, Trumpet, Tuba, Vibraphone, Viola Ensemble,
Warm Acoustic Guitar, Woodwinds, ...
• Music Genre: Acid Jazz, Afrobeat, Alternative Country, Baroque, Bengal Baul,
Bhangra, Bluegrass, Blues Rock, Bossa Nova, Breakbeat, Celtic Folk, Chillout,
Chiptune, Classic Rock, Contemporary R&B, Cumbia, Deep House, Disco Funk,
Drum & Bass, Dubstep, EDM, Electro Swing, Funk Metal, G-funk, Garage Rock,
Glitch Hop, Grime, Hyperpop, Indian Classical, Indie Electronic, Indie Folk,
Indie Pop, Irish Folk, Jam Band, Jamaican Dub, Jazz Fusion, Latin Jazz, Lo-Fi
Hip Hop, Marching Band, Merengue, New Jack Swing, Minimal Techno, Moombahton,
Neo-Soul, Orchestral Score, Piano Ballad, Polka, Post-Punk, 60s Psychedelic
Rock, Psytrance, R&B, Reggae, Reggaeton, Renaissance Music, Salsa, Shoegaze,
Ska, Surf Rock, Synthpop, Techno, Trance, Trap Beat, Trip Hop, Vaporwave,
Witch house, ...
• Mood/Description: Acoustic Instruments, Ambient, Bright Tones, Chill,
Crunchy Distortion, Danceable, Dreamy, Echo, Emotional, Ethereal Ambience,
Experimental, Fat Beats, Funky, Glitchy Effects, Huge Drop, Live Performance,
Lo-fi, Ominous Drone, Psychedelic, Rich Orchestration, Saturated Tones,
Subdued Melody, Sustained Chords, Swirling Phasers, Tight Groove,
Unsettling, Upbeat, Virtuoso, Weird Noises, ...

These are just some examples, Lyria RealTime can do much more. Experiment
with your own prompts!

Best practices

• Client applications must implement robust audio buffering to ensure smooth
  playback. This helps account for network jitter and slight variations in
  generation latency.
• Effective prompting:
  • Be descriptive. Use adjectives describing mood, genre, and instrumentation.
  • Iterate and steer gradually. Rather than completely changing the prompt,
    try adding or modifying elements to morph the music more smoothly.
  • Experiment with weight on WeightedPrompt to influence how strongly a new
    prompt affects the ongoing generation.

Technical details

This section describes the specifics of how to use Lyria RealTime music
generation.

Specifications

• Output format: Raw 16-bit PCM Audio
• Sample rate: 48kHz
• Channels: 2 (stereo)

Controls

Music generation can be influenced in real time by sending messages containing:

• WeightedPrompt: A text string describing a musical idea, genre, instrument,
  mood, or characteristic. Multiple prompts can potentially be supplied to blend
  influences. See above for more details on how to best prompt
  Lyria RealTime.
• MusicGenerationConfig: Configuration for the music generation process,
  influencing the characteristics of the output audio.). Parameters
  include:
  • guidance: (float) Range: [0.0, 6.0]. Default: 4.0.
    Controls how strictly the model follows the prompts. Higher guidance
    improves adherence to the prompt, but makes transitions more abrupt.
  • bpm: (int) Range: [60, 200].
    Sets the Beats Per Minute you want for the generated music. You need to
    stop/play or reset the context for the model it take into account the new
    bpm.
  • density: (float) Range: [0.0, 1.0].
    Controls the density of musical notes/sounds. Lower values produce sparser
    music; higher values produce “busier” music.
  • brightness: (float) Range: [0.0, 1.0].
    Adjusts the tonal quality. Higher values produce “brighter” sounding
    audio, generally emphasizing higher frequencies.
  • scale: (Enum)
    Sets the musical scale (Key and Mode) for the generation. Use the
    Scale enum values provided by the SDK. You need to
    stop/play or reset the context for the model it take into account the new
    scale.
  • mute_bass: (bool) Default: False.
    Controls whether the model reduces the outputs’ bass.
  • mute_drums: (bool) Default: False.
    Controls whether the model outputs reduces the outputs’ drums.
  • only_bass_and_drums: (bool) Default: False.
    Steer the model to try to only output bass and drums.
  • music_generation_mode: (Enum)
    Indicates to the model if it should focus on QUALITY (default value) or
    DIVERSITY of music. It can also be set to VOCALIZATION to let the
    model generate vocalizations as another instrument (add them as new
    pompts).
• PlaybackControl: Commands to control playback aspects, such as play, pause,
  stop or reset the context.

For bpm, density, brightness and scale, if no value is provided, the
model will decide what’s best according to your initial prompts.

More classical parameters like temperature (0.0 to 3.0, default 1.1), top_k
(1 to 1000, default 40), and seed (0 to 2 147 483 647, randomly selected by
default) are also customizable in the MusicGenerationConfig.

Scale Enum Values

Here are all the scale values that the model can accept:

The model is capable of guiding the notes that are played, but does
not distinguish between relative keys. Thus each enum corresponds both to the
relative major and minor. For example, C_MAJOR_A_MINOR would correspond to all
the white keys of a piano, and F_MAJOR_D_MINOR would be all the white keys
except B flat.

Limitations

• Instrumental only: The model generates instrumental music only.
• Safety: Prompts are checked by safety filters. Prompts triggering the filters
  will be ignored in which case an explanation will be written in the output’s
  filtered_prompt field.
• Watermarking: Output audio is always watermarked for identification following
  our Responsible AI principles.

What’s next

• Generate full songs and vocal tracks with Lyria 3,
• Instead of music, learn how to generate multi-speakers conversation using
  the TTS models,
• Discover how to generate images or videos,
• Instead of generation music or audio, find out how to Gemini can
  understand Audio files,
• Have a real-time conversation with Gemini using the
  Live API.

Explore the Cookbook for more
code examples and tutorials.