Skip to content
English
Contact us
  • There are no suggestions because the search field is empty.

Batch

Using Batch you can send asynchronous groups of requests, a separate pool of significantly higher rate limits, and a clear 24-hour turnaround time. This is ideal for processing jobs that do not require immediate responses.

Batch processing jobs are helpful in use cases similar to the below:

  • Running evaluations
  • Classifying large datasets
  • Embedding content repositories

How to Use Batch?

To get started with batch, read the following sections:

  1. Prepare Your Batch File
  2. Upload Your Batch Input File
  3. Create Batch
  4. Check the Status of a Batch
  5. Retrieve the Results
  6. Cancel a Batch
  7. Get a List of All Batches

Note: The Batch is currently supported with the GPT-4o and GPT-4o mini models.

To ensure data security, all the uploaded and downloaded files are accessible only to the authorized user with the appropriate keys. All the files will be permanently deleted from our premises after 30 days.

Prepare Your Batch File

Batches start with a .jsonl file where each line contains the details of an individual request to the API.

For now, the available endpoint is /v1/chat/completions.

For a given input file, the parameters in each line's body field are the same as the parameters for the underlying endpoint. Each request must include a unique custom_id value, which you can use to reference results after completion. Following is an example request.

{"custom_id": "request-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o", "messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}
{"custom_id": "request-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "gpt-4o", "messages": [{"role": "system", "content": "You are an unhelpful assistant."},{"role": "user", "content": "Hello world!"}],"max_tokens": 1000}}

Note: Each input file can only include requests to a single model.

Upload Your Batch Input File

You must upload your input file to correctly reference it when creating batches.

Note: When submitting the request, the parameter, purpose ="batch" must be provided.

Example Request

curl --location https://api.core42.ai/batch-api/v1/files   
 -H "api-key: $COMPASS_API_KEY"   
 -F purpose="batch"   
-F file="@batchinput.jsonl"

where $COMPASS_API_KEY is your Compass API Key.

Response

{
    "id": "file_id",
    "bytes": 448,
    "created_at": 1735553308,
    "filename": "@batchinput.jsonl",
    "object": "file",
    "purpose": "batch"
}

Create Batch

Once you've successfully uploaded your input file, use the input File object's ID to create a batch. In the following example, let's assume the file ID is file-abc123. For now, the completion window can only be set to 24h. You can also provide custom metadata via an optional metadata parameter.

curl --location https://api.core42.ai/batch-api/v1/batches  
 -H "api-key: $COMPASS_API_KEY"  
 -H "Content-Type: application/json"   -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

where $COMPASS_API_KEY is your Compass API Key.

The request returns a Batch object with metadata about your batch:

{
  "id": "batch_abc123",
  "object": "batch",
  "errors": null,
  "input_file_id": "file-abc123",
  "completion_window": "24h",
  "status": "validating",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1714508499,
  "in_progress_at": null,
  "expires_at": 1714536634,
  "completed_at": null,
  "failed_at": null,
  "expired_at": null,
  "cancelling_at": null,
   "cancelled_at": null,
  "request_counts": {
    "total": 0,
    "completed": 0,
    "failed": 0
  },
  "metadata": null
}

Check the Status of a Batch

You can check the status of a batch at any time, which will also return a Batch object.

where $COMPASS_API_KEY is your Compass API Key and batch_id is the batch ID whose status you want to check. Replace this value with your batch ID.

The status of a given Batch object can be any of the following:

Status Description
validating The input file is being validated before the batch can begin.
failed The input file has failed the validation process.
in_progress The input file was successfully validated and the batch is currently being run.
finalizing The batch has been completed and the results are being prepared.
completed The batch has been completed and the results are ready.
completed_with_error The batch has been partially completed with errors.
expired The batch was not able to be completed within the 24-hour time window.
cancelling The batch is being cancelled (may take up to 10 minutes).
cancelled The batch was cancelled.

Retrieve the Results

Once the batch is complete, you can download the output by making a request against the retrieved file content via the output_file_id field from the Batch object and writing it to a file on your machine, in this case, batch_output.jsonl.

curl --location https://api.core42.ai/batch-api/v1/files/file-xyz123/content   
 -H "api-key: $COMPASS_API_KEY" > batch_output.jsonl

where $COMPASS_API_KEY is your Compass API Key.

The output batch_output.jsonl file will have one response line for every successful request line in the input file. Any failed requests in the batch will have their error information written to an error file that can be found via the batch's error_file_id.

Note: The output line order may not match the input line order. Instead of relying on order to process your results, use the custom_id field which will be present in each line of your output file and allow you to map requests in your input to results in your output.

Below is the sample retrieval file:

{"id": "batch_req_123", "custom_id": "request-2", "response": {"status_code": 200, "request_id": "req_123", "body": {"id": "chatcmpl-123", "object": "chat.completion", "created": 1711652795, "model": "gpt-4o", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello."}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 22, "completion_tokens": 2, "total_tokens": 24}, "system_fingerprint": "fp_123"}}, "error": null}
{"id": "batch_req_456", "custom_id": "request-1", "response": {"status_code": 200, "request_id": "req_789", "body": {"id": "chatcmpl-abc", "object": "chat.completion", "created": 1711652789, "model": "gpt-4o", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello! How can I assist you today?"}, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 20, "completion_tokens": 9, "total_tokens": 29}, "system_fingerprint": "fp_3ba"}}, "error": null}

Cancel a Batch

If necessary, you can cancel an ongoing batch. The batch's status will change to cancelling until in-flight requests are complete (up to 10 minutes), after which the status will change to cancelled.

curl --location https://api.core42.ai/batch-api/v1/batches/batch_abc123/cancel   
 -H "api-key: $COMPASS_API_KEY"   
 -H "Content-Type: application/json"   
 -X POST

where $COMPASS_API_KEY is your Compass API Key, and batch_abc123 is the batch ID to be canceled. Replace this value with your batch ID.

Get a List of All Batches

You can see all your batches at any time. Users with multiple batches can use the limit and after parameters to paginate their results.

curl --location https://api.core42.ai/batch-api/v1/batches?limit=10&after=batch_abc123  
 -H "api-key: $COMPASS_API_KEY"  
 -H "Content-Type: application/json"

where $COMPASS_API_KEY is your Compass API Key.

Batch Expiration

Batches that do not complete in time eventually move to an expired state; unfinished requests within that batch are canceled, and any responses to completed requests are made available via the batch's output file. You will be charged for tokens consumed from any completed requests.

Expired requests will be written to your error file with the message. You can use the custom_id to retrieve the request data for expired requests. Below is the sample error file.

{"id": "batch_req_123", "custom_id": "request-3", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}
{"id": "batch_req_123", "custom_id": "request-7", "response": null, "error": {"code": "batch_expired", "message": "This request could not be executed before the completion window expired."}}