Server Events (Part 1)

Server Events

These events are emitted from the OpenAI Realtime WebSocket server to the client.

error

Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be error.
error	false	object	Details of the error.
type	false	string	The type of error (e.g., "invalid_request_error", "server_error").
code	false	string ot null	Error code, if any.
message	false	string	A human-readable error message.
param	false	string ot null	Parameter related to the error, if any.
event_id	false	string ot null	The event ID of the client event that caused the error, if applicable.

session.created

Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be session.created.
session	false	string	Realtime session object configuration.
id	false	string	Unique identifier for the session object.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
model	N/A	N/A	The Realtime model used for this session.
instructions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
input_audio_format	false	string	The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
input_audio_transcription	false	object	Configuration for input audio transcription defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through Whisper and should be treated as rough guidance rather than the representation understood by the model.
model	false	string	The model to use for transcription, whisper-1 is the only currently supported model.
turn_detection	false	object	Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
type	false	string	Type of turn detection, only server_vad is currently supported.
threshold	false	number	Activation threshold for VAD (0.0 to 1.0). The default is 0.5. A higher threshold will require louder audio to activate the model and thus might perform better in noisy environments.
prefix_padding_ms	false	integer	Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
silence_duration_ms	false	integer	Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	true	string	The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
description	false	string	A description of what the function does, used by the model to choose when and how to call the function.
parameters	false	object	The parameters that functions accept are described as a JSON Schema object. See the guide for examples, and the JSON Schema reference documentation about the format. Omitting parameters defines a function with an empty parameter list.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function.
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf"	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

session.updated

Returned when a session is updated with a session.update event, unless there is an error.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be session.updated.
session	false	string	Realtime session object configuration.
id	false	string	Unique identifier for the session object.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
model	N/A	N/A	The Realtime model used for this session.
instructions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
input_audio_format	false	string	The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
input_audio_transcription	false	object	Configuration for input audio transcription defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through Whisper and should be treated as rough guidance rather than the representation understood by the model.
model	false	string	The model to use for transcription, whisper-1 is the only currently supported model.
turn_detection	false	object	Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
type	false	string	Type of turn detection, only server_vad is currently supported.
threshold	false	number	Activation threshold for VAD (0.0 to 1.0). The default is 0.5. A higher threshold will require louder audio to activate the model and thus might perform better in noisy environments.
prefix_padding_ms	false	integer	Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
silence_duration_ms	false	integer	Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	true	string	The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.
description	false	string	A description of what the function does, used by the model to choose when and how to call the function.
parameters	false	object	The parameters that functions accept are described as a JSON Schema object. See the guide for examples, and the JSON Schema reference documentation about the format. Omitting parameters defines a function with an empty parameter list.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function.
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf""	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

conversation.created

Returned when a conversation is created. Emitted right after session creation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.created.
conversation	false	object	The conversation resource.
id	false	string	The Unique ID of the conversation.
obejct	false	string	The object type, must be realtime.conversation.

conversation.item.created

Returned when a conversation item is created. There are several scenarios that produce this event:

The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function_call.
The input audio buffer has been committed, either by the client or the server (in server_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item.
The client has sent a conversation.item.create event to add a new Item to the Conversation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.created.
previous_item_id	false	string	The ID of the preceding item in the Conversation context, allows the client to understand the order of the conversation.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These do not affect the conversation but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), is only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content Message items of role user support input_text and input_audio content Message items of role assistant support text content.
type	false	string	The content type (input_text, input_audio, text).
text	false	string	The text content used for input_text and text content types.
audio	false	string	Base64-encoded audio bytes used for input_audio content type.
transcript	false	string	The transcript of the audio used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
argument	false	string	The arguments of the function call (for `function_call` items).
output	false	string	The output of the function call (for `function_call_output` items).

conversation.item.input_audio_transcription.completed

This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (in server_vad mode). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events.

Realtime models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model, currently always whisper-1. Thus the transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.input_audio_transcription.completed
item_id	false	string	The ID of the user message item containing the audio.
content_index	false	integer	The index of the content part containing the audio.
transcript	false	string	The transcribed text.

conversation.item.input_audio_transcription.failed

Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other error events so that the client can identify the related Item.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.input_audio_transcription.failed
item_id	false	string	The ID of the user message item.
content_index	false	integer	The index of the content part containing the audio.
error	false	object	Details of the transcription error.
type	false	string	The type of error.
code	false	string	Error code, if any.
message	false	string	A human-readable error message.
param	false	string	Parameter related to the error, if any.

conversation.item.truncated

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event. This event is used to synchronize the server's understanding of the audio with the client's playback.

This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.truncated.
item_id	false	string	The ID of the assistant message item that was truncated.
content_index	false	integer	The index of the content part that was truncated.
audio_end_ms	false	integer	The duration up to which the audio was truncated, in milliseconds.

conversation.item.deleted

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event. This event is used to synchronize the server's understanding of the conversation history with the client's view.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be conversation.item.deleted.
item_id	false	string	The ID of the item that was deleted.

input_audio_buffer.committed

Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.committed.
previous_item_id	false	string	The ID of the preceding item after which the new item will be inserted.
item_id	false	string	The ID of the user message item that will be created.

input_audio_buffer.cleared

Returned when the input audio buffer is cleared by the client with a input_audio_buffer.clear event.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.cleared.

input_audio_buffer.speech_started

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user.

The client should expect to receive a input_audio_buffer.speech_stopped event when speech stops. The item_id property is the ID of the user message item that will be created when speech stops and will also be included in the input_audio_buffer.speech_stopped event (unless the client manually commits the audio buffer during VAD activation).

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.speech_started.
audio_start_ms	false	Integer	Milliseconds from the start of all audio written to the buffer during the session when speech was first detected. This will correspond to the beginning of audio sent to the model, and thus includes the prefix_padding_ms configured in the Session.
item_id	false	string	The ID of the user message item that will be created when speech stops.

input_audio_buffer.speech_stopped

Returned in server_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be input_audio_buffer.speech_stopped.
audio_start_ms	false	Integer	Milliseconds since the session started when speech stopped. This will correspond to the end of audio sent to the model, and thus includes the min_silence_duration_ms configured in the Session.
item_id	false	string	The ID of the user message item that will be created.

response.created

Returned when a new Response is created. The first event of response creation, where the response is in an initial state of in_progress.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.created.
response	false	object	The response resource.
id	false	string	The object type, must be realtime.response.
status	false	string	The final status of the response (completed, cancelled, failed, or incomplete).
status_details	false	object	Additional details about the status.
type	false	string	The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).
reason	false	string	The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).
error	false	object	A description of the error that caused the response to fail, populated when the status is failed.
type	false	string	The type of error.
code	false	string	Error code, if any.
output	false	string	The list of output items generated by the response.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	string	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).
metadata	false	map	Developer-provided string key-value pairs associated with this response.
usuage	false	object	Usage statistics for the Response, this will correspond to billing. A Realtime session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.
total_tokens	false	integer	The total number of tokens in the Response including input and output text and audio tokens.
input_tokens	false	integer	The total number of input tokens in the Response including text and audio tokens.
output_tokens	false	integer	The total number of output tokens in the Response including text and audio tokens.
input_token_details	false	integer	Details about the input tokens used in the Response.
cached_tokens	false	integer	The number of cached tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.
output_token_details	false	integer	Details about the output tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.

response.done

Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data.

Name	Required	Type	Descriptions
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.done.
response	false	object	The response resource.
id	false	string	The ID of the response.
object	false	string	The object type, must be realtime.response.
status	false	string	The final status of the response (completed, cancelled, failed, or incomplete).
status_details	false	object	Additional details about the status.
type	false	string	The type of error that caused the response to fail, corresponding with the status field (completed, cancelled, incomplete, failed).
reason	false	string	The reason the Response did not complete. For a cancelled Response, one of turn_detected (the server VAD detected a new start of speech) or client_cancelled (the client sent a cancel event). For an incomplete Response, one of max_output_tokens or content_filter (the server-side safety filter activated and cut off the response).
error	false	object	A description of the error that caused the response to fail, populated when the status is failed.
type	false	string	The type of error.
code	false	string	Error code, if any.
output	false	string	The list of output items generated by the response.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).
metadata	false	map	Developer-provided string key-value pairs associated with this response.
usage	false	object	Usage statistics for the Response, this will correspond to billing. A Realtime session will maintain a conversation context and append new Items to the Conversation, thus output from previous turns (text and audio tokens) will become the input for later turns.
total_tokens	false	integer	The total number of tokens in the Response including input and output text and audio tokens.
input_tokens	false	integer	The total number of input tokens in the Response including text and audio tokens.
output_tokens	false	integer	The total number of output tokens in the Response including text and audio tokens.
input_token_details	false	integer	Details about the input tokens used in the Response.
cached_tokens	false	integer	The number of cached tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.
output_token_details	false	integer	Details about the output tokens used in the Response.
text_tokens	false	integer	The number of text tokens used in the Response.
audio_tokens	false	integer	The number of audio tokens used in the Response.

response.output_item.added

Returned when a new Item is created during Response generation.

Name	Required	Type	Description
event_id	false	string	The unique ID of the server event.
type	false	string	The event type, must be response.done.
response_id	false	string	The response resource.
output_index	false	string	The index of the output item in the Response.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item
status	false	string	The status of the item (completed, incomplete). These have no effect on the conversation, but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content. Message items of role user support input_text and input_audio content. Message items of role assistant support text content.
type	false	string	The type of the item (message, function_call, function_call_output).
text	false	string	The text content, used for input_text and text content types.
id	false	string	ID of a previous conversation item to reference (for item_reference content types in response.create events). These can reference both client and server created items.
audio	false	string	Base64-encoded audio bytes, used for input_audio content type.
transcript	false	string	The transcript of the audio, used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
arguments	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).