Guides API

Start runs and read results through the API

Your scripts and CI pipelines start test runs and read the results straight from gentesty, including every evaluator's verdict and the full transcript.

For
Developers and gentesty administrators
Time needed
About 15 minutes
Plan
Enterprise

Overview

With the gentesty API v1 you start runs and read their results without opening the dashboard. The API can:

  • list bots, test cases and tags
  • start runs for single test cases, whole bots or all test cases with a tag
  • read run groups and runs, with each evaluator’s verdict and the full transcript

All paths start with /v1 and live on your workspace’s backend. If your dashboard is at https://acme.gentesty.com, the base URL is https://acme-backend.gentesty.com. The full OpenAPI description is at /v1/openapi.yaml, no sign-in needed.

Before you start

  • Your gentesty workspace is on the Enterprise plan.
  • You have the gentesty role Owner or Workspace Administrator. Only these roles can create, replace and delete API clients.
  • You have an HTTP client. The examples use curl and jq.

01 Create an API client

gentesty Account › User Management › API

An API client is the access for one script or pipeline. Create one per application, so you can revoke a single access.

  1. Click Create API client.

  2. Enter a Name, for example CI pipeline (64 characters at most).

  3. Choose the Permission level:

    Permission levelAllows
    Read resultsRead bots, test cases, tags, runs and results.
    Start runs and read resultsEverything in Read results, plus starting runs.
  4. Click Create. The dialog shows the Client ID, Client Secret, Token URL and a ready-made example.

  5. Copy the client secret into your secret store, for example your CI pipeline’s protected variables.

  6. Close the dialog with I have saved the secret.

The permission level cannot be changed later. For a different level, create a new client and delete the old one. A workspace can have up to 25 API clients.

02 Request an access token

Every call needs an access token. You get it from the token URL with the client ID and client secret, using the OAuth 2.0 client credentials flow.

export TOKEN_URL="<token URL from the dialog>"
export CLIENT_ID="<client ID>"
export CLIENT_SECRET="<client secret>"
export BASE_URL="https://<workspace>-backend.gentesty.com"

TOKEN=$(curl -s -X POST "$TOKEN_URL" \
  -d grant_type=client_credentials \
  -d client_id="$CLIENT_ID" \
  -d client_secret="$CLIENT_SECRET" | jq -r .access_token)

A token is valid for 5 minutes. After that the API answers 401. Request a new token then; there is no refresh token.

03 Call the API

Send the token in the Authorization header of every call:

curl -H "Authorization: Bearer $TOKEN" "$BASE_URL/v1/bots"

The answer lists all chat and voice bots of your workspace with id, name and type. API client tokens only work on paths under /v1.

04 Start runs and read the results

A typical CI script has four steps. It needs a client with the level Start runs and read results and the variables from step 2.

These helpers request a fresh token for every call, so the script keeps working when waiting takes longer than 5 minutes.

token() {
  curl -s -X POST "$TOKEN_URL" -d grant_type=client_credentials \
    -d client_id="$CLIENT_ID" -d client_secret="$CLIENT_SECRET" | jq -r .access_token
}
api() { curl -s -H "Authorization: Bearer $(token)" "$@"; }
  1. Look up the IDs of your bots and tags:

    api "$BASE_URL/v1/bots" | jq '.items[] | {id, name, type}'
    api "$BASE_URL/v1/tags" | jq '.items[] | {id, name}'
  2. Start runs, here for all test cases with a tag. Instead of tag_ids you can send bot_ids (all test cases of a bot) and case_ids (single test cases), also combined. bot_type is required: chatbot or voicebot.

    START=$(api -X POST "$BASE_URL/v1/runs" \
      -H "Content-Type: application/json" \
      -d '{"bot_type":"chatbot","tag_ids":["<tag-id>"]}')
    GROUP_ID=$(echo "$START" | jq -r .run_group_id)
    [ "$GROUP_ID" != null ] || { echo "$START"; exit 1; }

    The 202 answer holds the run_group_id and the run_ids of the started runs. If the start fails, the last line prints the error and stops.

  3. Wait until the run group has finished:

    STATUS=running
    until [ "$STATUS" = passed ] || [ "$STATUS" = failed ]; do
      sleep 15
      STATUS=$(api "$BASE_URL/v1/run-groups/$GROUP_ID" | jq -r .status)
    done
  4. Read the results, with the failed evaluators per test case:

    api "$BASE_URL/v1/run-groups/$GROUP_ID/runs?size=50" | jq '.items[] | {case_name, status,
      failed: [.evaluations[] | select(.passed | not) | .evaluator_name]}'

    If the group has more than 50 runs, read on with &page=2 and so on, until page * size >= total.

api "$BASE_URL/v1/runs/<run-id>" returns the transcript and detailed comparisons of one run. End the script in your pipeline with [ "$STATUS" = passed ], so the job fails when a test case fails.

API runs count towards your plan’s usage like runs from the dashboard, and appear there with the initiator API.

Endpoints

MethodPathReturnsLevel
GET/v1/botsAll chat and voice bots, sorted by nameRead results
GET/v1/casesTest cases, paged. Filters bot_id and tag_idRead results
GET/v1/tagsAll tags, sorted by nameRead results
POST/v1/runsStarts one run per selected test caseStart runs
GET/v1/runs/{id}One run with evaluations, detailed comparisons and transcriptRead results
GET/v1/run-groupsRun groups, newest first, pagedRead results
GET/v1/run-groups/{id}One run group with its status and the number of runs per statusRead results
GET/v1/run-groups/{id}/runsThe runs of a group with evaluations and transcripts, pagedRead results
GET/v1/openapi.yamlThe OpenAPI descriptionNo token
  • Pages: page starts at 1. size is 20 by default and 100 at most, 50 at most for /v1/run-groups/{id}/runs. The answer holds items, page, size and total.
  • Run group status: running until every run has finished. Then failed if a run failed or ended in an error, else passed.
  • Run status: new and invoked (queued), running, success (at least one evaluator met every expected output), failed (no evaluator did), error (the run could not be completed).
  • Detailed comparisons: how each bot message compared with each expected output (matches) is only in /v1/runs/{id}. /v1/run-groups/{id}/runs leaves the field out to keep pages small.

Limits

  • Each API client may send 5 requests per second, with bursts of up to 20.
  • One start covers at most 100 bots, 1,000 test cases and 100 tags.
  • A workspace processes API starts one at a time. A second start in the meantime gets 429.
  • Above the limits the API answers 429 with a Retry-After header. Wait that many seconds, then repeat the call.

Manage API clients

gentesty Account › User Management › API

  • Replace secret creates a new client secret and shows it once. The old secret stops working immediately. Update every script that uses it.
  • Delete API client removes the access for good.
  • Tokens issued before either action stay valid until they expire, so for 5 minutes at most.
  • If your workspace moves from the Enterprise plan to another plan, the clients remain, but every call gets 403.

Choose Read results when a script only evaluates. Keep the client secret in a secret store only, never in a code repository or in logs. Replace it when someone with access leaves your team.

Troubleshooting

Errors come as JSON, message gives the reason:

{ "status": "FAILED", "httpCode": 403, "message": "forbidden" }
What you seeCauseFix
400 with unknown fieldThe body holds a field the API does not know.Send only bot_type, bot_ids, case_ids and tag_ids.
400 with Select at least one bot, case or tagNothing selected in the body.Give at least one ID in bot_ids, case_ids or tag_ids.
401 authorization failedThe token is missing, expired or invalid.Request a new token, see step 2.
402Your plan’s monthly allowance or budget allows no more runs.Check usage and budget under Billing.
403 forbidden when starting runsThe client has the level Read results.Use a client with the level Start runs and read results.
403 forbidden on a path under /apiAPI client tokens only work under /v1.Use a path under /v1, see Endpoints.
403 API access is not part of this subscriptionYour workspace is not on the Enterprise plan.Change the plan or contact us.
404The bot, run or run group does not exist in this workspace.Compare the ID with /v1/bots, /v1/run-groups or the answer of the start.
429Too many requests, or another start of your workspace is still running.Wait as many seconds as Retry-After says, then repeat.
Token URL answers Invalid client or Invalid client credentialsThe client ID or client secret is wrong, or the client was deleted.Check the values or replace the secret, see Manage API clients.

Stuck? Contact us with the client ID, the path and the time of the call.