Realtime API Reference

Realtime

Communicate with a GPT-4o class model live, in real-time, over WebSocket. Produces both audio and text transcriptions. Learn more about the Realtime.

Compass supports the following Realtime models:

gpt-4o-realtime-preview (defaults to gpt-4o-realtime-preview-2024-12-17)
gpt-4o-realtime-preview-2024-12-17

Client Events

These are events that the OpenAI Realtime WebSocket server will accept from the client.

Azure OpenAI

Request

wss://api.core42.ai/openai/realtime

OpenAI

Request

wss://api.core42.ai/v1/realtime

Note: See the Realtime section under Model Capabilities on how to send the requests.

session.update

Send this event to update the session’s default configuration. The client may send this event at any time to update the session configuration, and any field may be updated at any time, except for "voice". The server will respond with a session.updated event that shows the full effective configuration. Only fields that are present are updated, thus the correct way to clear a field like "instructions" is to pass an empty string.

Request Parameters

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type must be session.update.
session	false	string	Realtime session object configuration.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
model	false	string	The Realtime model used for this session.
instrcutions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
input_audio_format	false	string	The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
input_audio_transcription	false	object	Configuration for input audio transcription defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through Whisper and should be treated as rough guidance rather than the representation understood by the model.
model	false	string	The model to use for transcription, whisper-1 is the only currently supported model.
turn_detection	false	object	Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
type	false	string	Type of turn detection, only server_vad is currently supported.
threshold	false	number	Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments.
prefix_padding_ms	false	integer	Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
silence_duration_ms	false	integer	Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
create_response	false	boolean	Whether or not to automatically generate a response when VAD is enabled. true by default.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	true	string	The name of the function.
description	false	string	The description of the function, including guidance on when and how to call it, and guidance about what to tell the user when calling (if anything).
parameters	false	object	Parameters of the function in JSON Schema.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function.
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf"	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

input_audio_buffer.append

Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually.

The client may choose how much audio to include in each event up to a maximum of 15 MiB. Streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike with other client events, the server will not send a confirmation response to this event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.append.
audio	false	string	Base64-encoded audio bytes. This must be in the format specified by the input_audio_format field in the session configuration.

input_audio_buffer.commit

Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.

Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.commit.

input_audio_buffer.clear

Send this event to clear the audio bytes in the buffer. The server will respond with an input_audio_buffer.cleared event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.clear.

conversation.item.create

Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.

If successful, the server will respond with a conversation.item.created event, otherwise an error event will be sent.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be conversation.item.create.
previous_item_id	false	object	The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation. If set, it allows an item to be inserted mid-conversation. If the ID cannot be found, an error will be returned and the item will not be added.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These do not affect the conversation but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), is only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content Message items of role user support input_text and input_audio content Message items of role assistant support text content.
type	false	string	The content type (input_text, input_audio, text).
text	false	string	The text content used for input_text and text content types.
audio	false	string	Base64-encoded audio bytes used for input_audio content type.
transcript	false	string	The transcript of the audio used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
argument	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

conversation.item.truncate

Send this event to truncate a previous assistant message’s audio. The server produces audio faster than realtime, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback. Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user.

If successful, the server will respond with a conversation.item.truncated event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be conversation.item.truncate.
item_id	false	object	The ID of the assistant message item to truncate. Only assistant message items can be truncated.
content_index	false	integer	The index of the content part to truncate. Set this to 0.
audio_end_ms	false	integer	Inclusive duration up to which audio is truncated, in milliseconds. If the audio_end_ms is greater than the actual audio duration, the server will respond with an error.

conversation.item.delete

Send this event when you want to remove any item from the conversation history. The server will respond with a conversation.item.deleted event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be conversation.item.delete.
item_id	false	object	The ID of the item to delete.

response.create

This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically. A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history. The server will respond with a response.created event, events for Items and content created, and finally a response.done event to indicate the Response is complete. The response.create event includes inference configuration like instructions, and temperature. These fields will override the Session's configuration for this Response only.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be response.create.
response	false	object	Realtime session object configuration.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
instrcutions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	false	string	The name of the function.
description	false	string	The description of the function, including guidance on when and how to call it, and guidance about what to tell the user when calling (if anything).
parameters	false	object	Parameters of the function in JSON Schema.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function, like "type": "function", "function": "name": "my_function".
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf"	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.
conversation	false	string	Controls which conversation the response is added to. Currently supports auto and none, with auto as the default value. The auto value means that the contents of the response will be added to the default conversation. Set this to none to create an out-of-band response which will not add items to default conversation.
metadata	false	map	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format. Keys can be a maximum of 64 characters long and values can be a maximum of 512 characters long.
input	false	array	Input items to include in the prompt for the model. Creates a new context for this response, without including the default conversation. Can include references to items from the default conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These do not affect the conversation but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), is only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content Message items of role user support input_text and input_audio content Message items of role assistant support text content.
type	false	string	The content type (input_text, input_audio, text).
text	false	string	The text content used for input_text and text content types.
audio	false	string	Base64-encoded audio bytes used for input_audio content type.
transcript	false	string	The transcript of the audio used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
argument	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

response.cancel

Send this event to cancel an in-progress response. The server will respond with a response.cancelled event or an error if there is no response to cancel.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be response.cancel.

Server Events

These events are emitted from the OpenAI Realtime WebSocket server to the client.

error

Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be error.
error	false	object	Details of the error.
type	false	string	The type of error (e.g., "invalid_request_error", "server_error").
code	false	string or null	Error code, if any.
message	false	string	A human-readable error message.
param	false	string or null	Parameter related to the error, if any.
event_id	false	string or null	The event ID of the client event that caused the error, if applicable.

session.created

Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be session.created.
session	false	string	Realtime session object configuration.
id	false	string	Unique identifier for the session object.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
model	N/A	N/A	The Realtime model used for this session.
instrcutions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
input_audio_format	false	string	The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
input_audio_transcription	false	object	Configuration for input audio transcription defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through Whisper and should be treated as rough guidance rather than the representation understood by the model.
model	false	string	The model to use for transcription, whisper-1 is the only currently supported model.
turn_detection	false	object	Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
type	false	string	Type of turn detection, only server_vad is currently supported.
threshold	false	number	Activation threshold for VAD (0.0 to 1.0). The default is 0.5. A higher threshold will require louder audio to activate the model and thus might perform better in noisy environments.
prefix_padding_ms	false	integer	Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
silence_duration_ms	false	integer	Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	true	string	The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
description	false	string	A description of what the function does, used by the model to choose when and how to call the function.
parameters	false	object	The parameters that functions accept are described as a JSON Schema object. See the guide for examples, and the JSON Schema reference documentation about the format. Omitting parameters defines a function with an empty parameter list.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function.
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf"	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

session.updated

Returned when a session is updated with a session.update event, unless there is an error.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be session.updated.
session	false	string	Realtime session object configuration.
id	false	string	Unique identifier for the session object.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
model	N/A	N/A	The Realtime model used for this session.
instrcutions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
input_audio_format	false	string	The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
input_audio_transcription	false	object	Configuration for input audio transcription defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through Whisper and should be treated as rough guidance rather than the representation understood by the model.
model	false	string	The model to use for transcription, whisper-1 is the only currently supported model.
turn_detection	false	object	Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
type	false	string	Type of turn detection, only server_vad is currently supported.
threshold	false	number	Activation threshold for VAD (0.0 to 1.0). The default is 0.5. A higher threshold will require louder audio to activate the model and thus might perform better in noisy environments.
prefix_padding_ms	false	integer	Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
silence_duration_ms	false	integer	Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	true	string	The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
description	false	string	A description of what the function does, used by the model to choose when and how to call the function.
parameters	false	object	The parameters that functions accept are described as a JSON Schema object. See the guide for examples, and the JSON Schema reference documentation about the format. Omitting parameters defines a function with an empty parameter list.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function.
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf""	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

conversation.created

Returned when a conversation is created. Emitted right after session creation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.created.
conversation	false	object	The conversation resource.
id	false	string	The Unique ID of the conversation.
object	false	string	The object type, must be realtime.conversation.

conversation.item.created

Returned when a conversation item is created. There are several scenarios that produce this event:

The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function_call.
The input audio buffer has been committed, either by the client or the server (in server_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item.
The client has sent a conversation.item.create event to add a new Item to the Conversation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.created.
previous_item_id	false	string	The ID of the preceding item in the Conversation context, allows the client to understand the order of the conversation.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These do not affect the conversation but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), is only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content Message items of role user support input_text and input_audio content Message items of role assistant support text content.
type	false	string	The content type (input_text, input_audio, text).
text	false	string	The text content used for input_text and text content types.
audio	false	string	Base64-encoded audio bytes used for input_audio content type.
transcript	false	string	The transcript of the audio used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
argument	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

conversation.item.input_audio_transcription.completed

This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (in server_vad mode). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events.

Realtime models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model, currently always whisper-1. Thus the transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.input_audio_transcription.completed.
item_id	false	string	The ID of the user message item containing the audio.
content_index	false	integer	The index of the content part containing the audio.
transcript	false	string	The transcribed text.

conversation.item.input_audio_transcription.failed

Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other error events so that the client can identify the related Item.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.input_audio_transcription.failed.
item_id	false	string	The ID of the user message item.
content_index	false	integer	The index of the content part containing the audio.
error	false	object	Details of the transcription error.
type	false	string	The type of error.
code	false	string	Error code, if any.
message	false	string	A human-readable error message.
param	false	string	Parameter related to the error, if any.

conversation.item.truncated

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event. This event is used to synchronize the server's understanding of the audio with the client's playback.

This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.truncated.
item_id	false	string	The ID of the assistant message item that was truncated.
content_index	false	integer	The index of the content part that was truncated.
audio_end_ms	false	integer	The duration up to which the audio was truncated, in milliseconds.

conversation.item.deleted

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event. This event is used to synchronize the server's understanding of the conversation history with the client's view.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.deleted.
item_id	false	string	The ID of the item that was deleted.

input_audio_buffer.committed

Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.committed.
previous_item_id	false	string	The ID of the preceding item after which the new item will be inserted.
item_id	false	string	The ID of the user message item that will be created.

input_audio_buffer.cleared

Returned when the input audio buffer is cleared by the client with a input_audio_buffer.clear event.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.cleared.

input_audio_buffer.speech_started

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user.

The client should expect to receive a input_audio_buffer.speech_stopped event when speech stops. The item_id property is the ID of the user message item that will be created when speech stops and will also be included in the input_audio_buffer.speech_stopped event (unless the client manually commits the audio buffer during VAD activation).

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.speech_started.
audio_start_ms	false	integer	Milliseconds from the start of all audio written to the buffer during the session when speech was first detected. This will correspond to the beginning of audio sent to the model, and thus includes the prefix_padding_ms configured in the Session.
item_id	false	string	The ID of the user message item that will be created when speech stops.

input_audio_buffer.speech_stopped

Returned in server_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.speech_stopped.
audio_start_ms	false	integer	Milliseconds since the session started when speech stopped. This will correspond to the end of audio sent to the model, and thus includes the min_silence_duration_ms configured in the Session.
item_id	false	string	The ID of the user message item that will be created.

response.created

Returned when a new Response is created. The first event of response creation, where the response is in an initial state of in_progress.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.created.
response	false	object	The response resource.
id	false	string	The ID of the response.
object	false	string	The object type, must be realtime.response.
status	false	string	The final status of the response (completed, cancelled, failed, or incomplete).
status_details	false	object	Additional details about the status.
type	false	string	The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).
reason	false	string	The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).
error	false	object	A description of the error that caused the response to fail, populated when the status is failed.
type	false	string	The type of error.
code	false	string	Error code, if any.
output	false	string	The list of output items generated by the response.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	string	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).
metadata	false	map	Developer-provided string key-value pairs associated with this response.
usage	false	object	Usage statistics for the Response, this will correspond to billing. A Realtime session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.
total_tokens	false	integer	The total number of tokens in the Response including input and output text and audio tokens.
input_tokens	false	integer	The total number of input tokens in the Response including text and audio tokens.
output_tokens	false	integer	The total number of output tokens in the Response including text and audio tokens.
input_token_details	false	integer	Details about the input tokens used in the Response.
cached_tokens	false	integer	The number of cached tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.
output_token_details	false	integer	Details about the output tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.

response.done

Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.done.
response	false	object	The response resource.
id	false	string	The ID of the response.
object	false	string	The object type, must be realtime.response.
status	false	string	The final status of the response (completed, cancelled, failed, or incomplete).
status_details	false	object	Additional details about the status.
type	false	string	The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).
reason	false	string	The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).
error	false	object	A description of the error that caused the response to fail, populated when the status is failed.
type	false	string	The type of error.
code	false	string	Error code, if any.
output	false	string	The list of output items generated by the response.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).
metadata	false	map	Developer-provided string key-value pairs associated with this response.
usage	false	object	Usage statistics for the Response, this will correspond to billing. A Realtime session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.
total_tokens	false	integer	The total number of tokens in the Response including input and output text and audio tokens.
input_tokens	false	integer	The total number of input tokens in the Response including text and audio tokens.
output_tokens	false	integer	The total number of output tokens in the Response including text and audio tokens.
input_token_details	false	integer	Details about the input tokens used in the Response.
cached_tokens	false	integer	The number of cached tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.
output_token_details	false	integer	Details about the output tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.

response.output_item.added

Returned when a new Item is created during Response generation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.done.
response_id	false	string	The response resource.
output_index	false	string	The index of the output item in the Response.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

response.output_item.done

Returned when an Item is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.done.
response_id	false	string	The ID of the Response to which the item belongs.
output_index	false	string	The index of the output item in the Response.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

response.content_part.added

Returned when a new content part is added to an assistant message item during response generation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.content_part.added.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item to which the content part was added.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
part	false	object	The content part that was added.
type	false	string	The content type ("text", "audio").
text	false	string	The text content (if type is "text").
audio	false	string	Base64-encoded audio data (if type is "audio").
transcript	false	string	The transcript of the audio (if type is "audio").

response.content_part.done

Returned when a content part is done streaming in an assistant message item. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.content_part.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item to which the content part was added.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
part	false	object	The content part that was added.
type	false	string	The content type ("text", "audio").
text	false	string	The text content (if type is "text").
audio	false	string	Base64-encoded audio data (if type is "audio").
transcript	false	string	The transcript of the audio (if type is "audio").

response.text.delta

Returned when the text value of a "text" content part is updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.text.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the response.
content_index	false	integer	The index of the content part in the item's content array.
delta	false	string	The text data.

response.text.done

Returned when the text value of a "text" content part is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.text.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
text	false	string	The final text content.

response.audio_transcript.delta

Returned when the model-generated transcription of audio output is updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.audio_transcript.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
delta	false	string	The transcript delta.

response.audio_transcript.done

Returned when the model-generated transcription of audio output is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.audio_transcript.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
transcript	false	string	The final transcript of the audio.

response.audio.delta

Returned when the model-generated audio is updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.audio.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.
delta	false	string	Base64-encoded audio data delta.

response.audio.done

Returned when the model-generated audio is done. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.audio.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the item.
output_index	false	integer	The index of the output item in the Response.
content_index	false	integer	The index of the content part in the item's content array.

response.function_call_arguments.delta

Returned when the model-generated function call arguments are updated.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.function_call_arguments.delta.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the function call item.
output_index	false	integer	The index of the output item in the Response.
call_id	false	string	The ID of the function call.
delta	false	string	The arguments delta as a JSON string.

response.function_call_arguments.done

Returned when the model-generated function call arguments are done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.function_call_arguments.done.
response_id	false	string	The ID of the response.
item_id	false	string	The ID of the function call item.
output_index	false	integer	The index of the output item in the Response.
call_id	false	string	The ID of the function call.
arguments	false	string	The arguments delta as a JSON string.

rate_limits.updated

Emitted at the beginning of a Response to indicate the updated rate limits. When a Response is created some tokens will be "reserved" for the output tokens, the rate limits shown here reflect that reservation, which is then adjusted accordingly once the Response is completed.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be rate_limits.updated.
rate_limits	false	array	List of rate limit information.
name	false	string	The name of the rate limit (requests, tokens).
limit	false	integer	The maximum allowed value for the rate limit.
remaining	false	integer	The remaining value before the limit is reached.
reset_seconds	false	number	Seconds until the rate limit resets.