Server Events (Part 2)

response.output_item.done

Returned when an Item is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.done.
response_id	false	string	The ID of the Response to which the item belongs.
output_index	false	string	The index of the output item in the Response.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

response.content_part.added

Returned when a new content part is added to an assistant message item during response generation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.content_part.added.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item to which the content part was added.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
part	false	object	The content part that was added.
type	false	string	The content type ("text", "audio").
text	false	string	The text content (if type is "text").
audio	false	string	Base64-encoded audio data (if type is "audio").
transcript	false	string	The transcript of the audio (if type is "audio").

response.content_part.done

Returned when a content part is done streaming in an assistant message item. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.content_part.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item to which the content part was added.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
part	false	object	The content part that was added.
type	false	string	The content type ("text", "audio").
text	false	string	The text content (if type is "text").
audio	false	string	Base64-encoded audio data (if type is "audio").
transcript	false	string	The transcript of the audio (if type is "audio").

response.output_text.delta

Returned when the text value of a "text" content part is updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.output_text.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the response.
content_index	false	integer	The index of the content part in the item's content array.
delta	false	string	The text data.

response.output_text.done

Returned when the text value of a "text" content part is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.output_text.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
text	false	string	The final text content.

response.output_audio_transcript.delta

Returned when the model-generated transcription of audio output is updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.output_audio_transcript.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
delta	false	string	The transcript delta.

response.output_audio_transcript.done

Returned when the model-generated transcription of audio output is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.output_audio_transcript.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
transcript	false	string	The final transcript of the audio.

response.output_audio.delta

Returned when the model-generated audio is updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.output_audio.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
delta	false	string	Base64-encoded audio data delta.

response.output_audio.done

Returned when the model-generated audio is done. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.output_audio.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.

response.function_call_arguments.delta

Returned when the model-generated function call arguments are updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.function_call_arguments.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the function call item.
output_index	false	integer	The index of the output item in the Response.
call_id	false	string	The ID of the function call.
delta	false	string	The arguments delta as a JSON string.

response.function_call_arguments.done

Returned when the model-generated function call arguments are done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.function_call_arguments.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the function call item.
output_index	false	integer	The index of the output item in the Response.
call_id	false	string	The ID of the function call.
arguments	false	string	The arguments delta as a JSON string.

rate_limits.updated

Emitted at the beginning of a Response to indicate the updated rate limits. When a Response is created some tokens will be "reserved" for the output tokens, the rate limits shown here reflect that reservation, which is then adjusted accordingly once the Response is completed.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be rate_limits.updated.
rate_limits	false	array	List of rate limit information.
name	false	string	The name of the rate limit (requests, tokens).
limit	false	integer	The maximum allowed value for the rate limit.
remaining	false	integer	The remaining value before the limit is reached.
reset_seconds	false	number	Seconds until the rate limit resets.

Request Parameters

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type must be session.update.
session	false	string	Realtime session object configuration.
modalities	N/A	N./A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
model	false	string	The Realtime model used for this session.
instructions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
input_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
input_audio_transcription	false	object	Configuration for input audio transcription defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through Whisper and should be treated as rough guidance rather than the representation understood by the model.
turn_detection	false	object	The model to use for transcription, whisper-1 is the only currently supported model.
threshold	false	number	Type of turn detection, only server_vad is currently supported.
prefix_padding_ms	false	integer	Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
create_response	false	boolean	Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	true	string	The name of the function.
description	false	string	The description of the function, including guidance on when and how to call it, and guidance about what to tell the user when calling (if anything).
parameters	false	object	Parameters of the function in JSON Schema.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function.
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.

input_audio_buffer.append

Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually.

The client may choose how much audio to include in each event up to a maximum of 15 MiB. Streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike with other client events, the server will not send a confirmation response to this event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.append.
audio	false	string	Base64-encoded audio bytes. This must be in the format specified by the input_audio_format field in the session configuration.

input_audio_buffer.commit

Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.

Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.commit.

input_audio_buffer.clear

Send this event to clear the audio bytes in the buffer. The server will respond with an input_audio_buffer.cleared event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.clear.