Realtime API Reference

Communicate with a GPT-4o class model live, in real-time, over WebSocket. Produces both audio and text transcriptions. Learn more about the Realtime.

Compass supports the following Realtime models:

gpt-4o-realtime-preview (defaults to gpt-4o-realtime-preview-2024-12-17)
gpt-4o-realtime-preview-2024-12-17
gpt-realtime

Client Events

These are events that the OpenAI Realtime WebSocket server will accept from the client.

Azure OpenAI

Request

wss://api.core42.ai/openai/realtime

OpenAI

Request

wss://api.core42.ai/v1/realtime

See the Realtime section under Model Capabilities on how to send the requests.

session.update

Send this event to update the session’s default configuration. The client may send this event at any time to update the session configuration, and any field may be updated at any time, except for "voice" and “model”. Voice can be updated only if there have been no other outputs yet. The server will respond with a session.updated event that shows the full effective configuration. Only fields that are present are updated, thus the correct way to clear a field like "instructions" is to pass an empty string. To clear fields such as tools and turn_detection pass an empty array and null.

Request Parameters

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type must be session.update.
session	false	string	Realtime session object configuration.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
model	false	string	The Realtime model used for this session.
instructions	false	string	The default system instructions (i.e. system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g. "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once. gpt-realtime also supports marin and cedar voices.
input_audio_format	false	string	The format of input audio. Options are pcm16, g711_ulaw, or g711_alaw.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
input_audio_transcription	false	object	Configuration for input audio transcription defaults to off and can be set to null to turn off once on. Input audio transcription is not native to the model, since the model consumes audio directly. Transcription runs asynchronously through Whisper and should be treated as rough guidance rather than the representation understood by the model.
turn_detection F	false	object	Configuration for turn detection. Can be set to null to turn off. Server VAD means that the model will detect the start and end of speech based on audio volume and respond at the end of user speech.
type	false	string	Type of turn detection, only server_vad is currently supported.
threshold	false	number	Activation threshold for VAD (0.0 to 1.0), this defaults to 0.5. A higher threshold will require louder audio to activate the model, and thus might perform better in noisy environments.
prefix_padding_ms	false	integer	Amount of audio to include before the VAD detected speech (in milliseconds). Defaults to 300ms.
silence_duration_ms	false	integer	Duration of silence to detect speech stop (in milliseconds). Defaults to 500ms. With shorter values the model will respond more quickly, but may jump in on short pauses from the user.
create_response	false	boolean	Whether or not to automatically generate a response when VAD is enabled. true by default.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e. function.
name	true	string	The name of the function.
description	false	string	The description of the function, including guidance on when and how to call it, and guidance about what to tell the user when calling (if anything).
parameters	false	object	Parameters of the function in JSON Schema.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function.
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf"	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.

input_audio_buffer.append

Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually.

The client may choose how much audio to include in each event up to a maximum of 15 MiB. Streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike with other client events, the server will not send a confirmation response to this event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.append.
audio	false	string	Base64-encoded audio bytes. This must be in the format specified by the input_audio_format field in the session configuration.

input_audio_buffer.commit

Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically.

Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.commit.

input_audio_buffer.clear

Send this event to clear the audio bytes in the buffer. The server will respond with an input_audio_buffer.cleared event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be input_audio_buffer.clear.

conversation.item.create

Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages.

If successful, the server will respond with a conversation.item.created event, otherwise an error event will be sent.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be conversation.item.create.
previous_item_id	false	object	The ID of the preceding item after which the new item will be inserted. If not set, the new item will be appended to the end of the conversation. If set, it allows an item to be inserted mid-conversation. If the ID cannot be found, an error will be returned and the item will not be added.
item	false	object	The item to add to the conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These do not affect the conversation but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), is only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content Message items of role user support input_text and input_audio content Message items of role assistant support text content.
type	false	string	The content type (input_text, input_audio, text).
text	false	string	The text content used for input_text and text content types.
audio	false	string	Base64-encoded audio bytes used for input_audio content type.
transcript	false	string	The transcript of the audio used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
argument	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

conversation.item.truncate

Send this event to truncate a previous assistant message’s audio. The server produces audio faster than realtime, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback. Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user.

If successful, the server will respond with a conversation.item.truncated event.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be conversation.item.truncate.
item_id	false	object	The ID of the assistant message item to truncate. Only assistant message items can be truncated.
content_index	false	integer	The index of the content part to truncate. Set this to 0.
audio_end_ms	false	integer	Inclusive duration up to which audio is truncated, in milliseconds. If the audio_end_ms is greater than the actual audio duration, the server will respond with an error.

conversation.item.delete

Send this event when you want to remove any item from the conversation history. The server will respond with a conversation.item.deleted event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be conversation.item.delete.
item_id	false	object	The ID of the item to delete.

response.create

This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically. A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history. The server will respond with a response.created event, events for Items and content created, and finally a response.done event to indicate the Response is complete. The response.create event includes inference configuration like instructions, and temperature. These fields will override the Session's configuration for this Response only.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be response.create.
response	false	object	Realtime session object configuration.
modalities	N/A	N/A	The set of modalities the model can respond with. To disable audio, set this to ["text"].
instructions	false	string	The default system instructions (i.e., system message) prepended to model calls. This field allows the client to guide the model on desired responses. The model can be instructed on response content and format, (e.g. "be extremely succinct", "act friendly", "here are examples of good responses") and on audio behavior (e.g., "talk quickly", "inject emotion into your voice", "laugh frequently"). The instructions are not guaranteed to be followed by the model, but they provide guidance to the model on the desired behavior. Note that the server sets default instructions which will be used if this field is not set and are visible in the session.created event at the start of the session.
voice	false	string	The voice, the model uses to respond. Current voice options are alloy, ash, ballad, coral, echo, sage, shimmer, and verse. Voice cannot be changed during the session once the model has responded with audio at least once.
output_audio_format	false	string	The format of output audio. Options are pcm16, g711_ulaw, or g711_alaw.
tools	false	array	Tools (functions) available to the model.
type	false	string	The type of the tool, i.e., function.
name	false	string	The name of the function.
description	false	string	The description of the function, including guidance on when and how to call it, and guidance about what to tell the user when calling (if anything).
parameters	false	object	Parameters of the function in JSON Schema.
tools_choice	false	string	How the model chooses tools. Options are auto, none, required, or specify a function, like "type": "function", "function": "name": "my_function".
temperature	false	number	Sampling temperature for the model, limited to [0.6, 1.2]. Defaults to 0.8.
max_response_output_tokens	false	integer or "inf"	Maximum number of output tokens for a single assistant response, including tool calls. Provide an integer between 1 and 4096 to limit output tokens, or inf for the maximum available tokens for a given model. Defaults to inf.
conversation	false	string	Controls which conversation the response is added to. Currently supports auto and none, with auto as the default value. The auto value means that the contents of the response will be added to the default conversation. Set this to none to create an out-of-band response which will not add items to default conversation.
metadata	false	map	Set of 16 key-value pairs that can be attached to an object. This can be useful for storing additional information about the object in a structured format. Keys can be a maximum of 64 characters long and values can be a maximum of 512 characters long.
input	false	array	Input items to include in the prompt for the model. Creates a new context for this response, without including the default conversation. Can include references to items from the default conversation.
id	false	string	The unique ID of the item, this can be generated by the client to help manage server-side context, but is not required because the server will generate one if not provided.
type	false	string	The type of the item (message, function_call, function_call_output).
object	false	string	Identifier for the API object being returned - always realtime.item.
status	false	string	The status of the item (completed, incomplete). These do not affect the conversation but are accepted for consistency with the conversation.item.created event.
role	false	string	The role of the message sender (user, assistant, system), is only applicable for message items.
content	false	array	The content of the message, applicable for message items. Message items of role system support only input_text content Message items of role user support input_text and input_audio content Message items of role assistant support text content.
type	false	string	The content type (input_text, input_audio, text).
text	false	string	The text content used for input_text and text content types.
audio	false	string	Base64-encoded audio bytes used for input_audio content type.
transcript	false	string	The transcript of the audio used for input_audio content type.
call_id	false	string	The ID of the function call (for function_call and function_call_output items). If passed on a function_call_output item, the server will check that a function_call item with the same ID exists in the conversation history.
name	false	string	The name of the function being called (for function_call items).
argument	false	string	The arguments of the function call (for function_call items).
output	false	string	The output of the function call (for function_call_output items).

response.cancel

Send this event to cancel an in-progress response. The server will respond with a response.cancelled event or an error if there is no response to cancel.

Name	Required	Type	Description
event_id	false	string	An optional client-generated ID used to identify this event.
type	false	string	The event type, must be response.cancel.