Overview
With the gentesty API v1 you start runs and read their results without opening the dashboard. The API can:
- list bots, test cases and tags
- start runs for single test cases, whole bots or all test cases with a tag
- read run groups and runs, with each evaluator’s verdict and the full transcript
All paths start with /v1 and live on your workspace’s backend. If your dashboard is at https://acme.gentesty.com,
the base URL is https://acme-backend.gentesty.com. The full OpenAPI description is at /v1/openapi.yaml, no sign-in
needed.
Before you start
- Your gentesty workspace is on the Enterprise plan.
- You have the gentesty role Owner or Workspace Administrator. Only these roles can create, replace and delete API clients.
- You have an HTTP client. The examples use
curlandjq.
01 Create an API client
gentesty Account › User Management › API
An API client is the access for one script or pipeline. Create one per application, so you can revoke a single access.
Click Create API client.
Enter a Name, for example
CI pipeline(64 characters at most).Choose the Permission level:
Permission level Allows Read results Read bots, test cases, tags, runs and results. Start runs and read results Everything in Read results, plus starting runs. Click Create. The dialog shows the Client ID, Client Secret, Token URL and a ready-made example.
Copy the client secret into your secret store, for example your CI pipeline’s protected variables.
Close the dialog with I have saved the secret.
The permission level cannot be changed later. For a different level, create a new client and delete the old one. A workspace can have up to 25 API clients.
02 Request an access token
Every call needs an access token. You get it from the token URL with the client ID and client secret, using the OAuth 2.0 client credentials flow.
export TOKEN_URL="<token URL from the dialog>"
export CLIENT_ID="<client ID>"
export CLIENT_SECRET="<client secret>"
export BASE_URL="https://<workspace>-backend.gentesty.com"
TOKEN=$(curl -s -X POST "$TOKEN_URL" \
-d grant_type=client_credentials \
-d client_id="$CLIENT_ID" \
-d client_secret="$CLIENT_SECRET" | jq -r .access_token)A token is valid for 5 minutes. After that the API answers 401. Request a new token then; there is no refresh token.
03 Call the API
Send the token in the Authorization header of every call:
curl -H "Authorization: Bearer $TOKEN" "$BASE_URL/v1/bots"The answer lists all chat and voice bots of your workspace with id, name and type. API client tokens only work on
paths under /v1.
04 Start runs and read the results
A typical CI script has four steps. It needs a client with the level Start runs and read results and the variables from step 2.
These helpers request a fresh token for every call, so the script keeps working when waiting takes longer than 5 minutes.
token() {
curl -s -X POST "$TOKEN_URL" -d grant_type=client_credentials \
-d client_id="$CLIENT_ID" -d client_secret="$CLIENT_SECRET" | jq -r .access_token
}
api() { curl -s -H "Authorization: Bearer $(token)" "$@"; }Look up the IDs of your bots and tags:
api "$BASE_URL/v1/bots" | jq '.items[] | {id, name, type}' api "$BASE_URL/v1/tags" | jq '.items[] | {id, name}'Start runs, here for all test cases with a tag. Instead of
tag_idsyou can sendbot_ids(all test cases of a bot) andcase_ids(single test cases), also combined.bot_typeis required:chatbotorvoicebot.START=$(api -X POST "$BASE_URL/v1/runs" \ -H "Content-Type: application/json" \ -d '{"bot_type":"chatbot","tag_ids":["<tag-id>"]}') GROUP_ID=$(echo "$START" | jq -r .run_group_id) [ "$GROUP_ID" != null ] || { echo "$START"; exit 1; }The
202answer holds therun_group_idand therun_idsof the started runs. If the start fails, the last line prints the error and stops.Wait until the run group has finished:
STATUS=running until [ "$STATUS" = passed ] || [ "$STATUS" = failed ]; do sleep 15 STATUS=$(api "$BASE_URL/v1/run-groups/$GROUP_ID" | jq -r .status) doneRead the results, with the failed evaluators per test case:
api "$BASE_URL/v1/run-groups/$GROUP_ID/runs?size=50" | jq '.items[] | {case_name, status, failed: [.evaluations[] | select(.passed | not) | .evaluator_name]}'If the group has more than 50 runs, read on with
&page=2and so on, untilpage * size >= total.
api "$BASE_URL/v1/runs/<run-id>" returns the transcript and detailed comparisons of one run. End the script in your
pipeline with [ "$STATUS" = passed ], so the job fails when a test case fails.
API runs count towards your plan’s usage like runs from the dashboard, and appear there with the initiator API.
Endpoints
| Method | Path | Returns | Level |
|---|---|---|---|
| GET | /v1/bots | All chat and voice bots, sorted by name | Read results |
| GET | /v1/cases | Test cases, paged. Filters bot_id and tag_id | Read results |
| GET | /v1/tags | All tags, sorted by name | Read results |
| POST | /v1/runs | Starts one run per selected test case | Start runs |
| GET | /v1/runs/{id} | One run with evaluations, detailed comparisons and transcript | Read results |
| GET | /v1/run-groups | Run groups, newest first, paged | Read results |
| GET | /v1/run-groups/{id} | One run group with its status and the number of runs per status | Read results |
| GET | /v1/run-groups/{id}/runs | The runs of a group with evaluations and transcripts, paged | Read results |
| GET | /v1/openapi.yaml | The OpenAPI description | No token |
- Pages:
pagestarts at 1.sizeis 20 by default and 100 at most, 50 at most for/v1/run-groups/{id}/runs. The answer holdsitems,page,sizeandtotal. - Run group status:
runninguntil every run has finished. Thenfailedif a run failed or ended in an error, elsepassed. - Run status:
newandinvoked(queued),running,success(at least one evaluator met every expected output),failed(no evaluator did),error(the run could not be completed). - Detailed comparisons: how each bot message compared with each expected output (
matches) is only in/v1/runs/{id}./v1/run-groups/{id}/runsleaves the field out to keep pages small.
Limits
- Each API client may send 5 requests per second, with bursts of up to 20.
- One start covers at most 100 bots, 1,000 test cases and 100 tags.
- A workspace processes API starts one at a time. A second start in the meantime gets
429. - Above the limits the API answers
429with aRetry-Afterheader. Wait that many seconds, then repeat the call.
Manage API clients
gentesty Account › User Management › API
- Replace secret creates a new client secret and shows it once. The old secret stops working immediately. Update every script that uses it.
- Delete API client removes the access for good.
- Tokens issued before either action stay valid until they expire, so for 5 minutes at most.
- If your workspace moves from the Enterprise plan to another plan, the clients remain, but every call gets
403.
Choose Read results when a script only evaluates. Keep the client secret in a secret store only, never in a code repository or in logs. Replace it when someone with access leaves your team.
Troubleshooting
Errors come as JSON, message gives the reason:
{ "status": "FAILED", "httpCode": 403, "message": "forbidden" }| What you see | Cause | Fix |
|---|---|---|
400 with unknown field | The body holds a field the API does not know. | Send only bot_type, bot_ids, case_ids and tag_ids. |
400 with Select at least one bot, case or tag | Nothing selected in the body. | Give at least one ID in bot_ids, case_ids or tag_ids. |
401 authorization failed | The token is missing, expired or invalid. | Request a new token, see step 2. |
402 | Your plan’s monthly allowance or budget allows no more runs. | Check usage and budget under Billing. |
403 forbidden when starting runs | The client has the level Read results. | Use a client with the level Start runs and read results. |
403 forbidden on a path under /api | API client tokens only work under /v1. | Use a path under /v1, see Endpoints. |
403 API access is not part of this subscription | Your workspace is not on the Enterprise plan. | Change the plan or contact us. |
404 | The bot, run or run group does not exist in this workspace. | Compare the ID with /v1/bots, /v1/run-groups or the answer of the start. |
429 | Too many requests, or another start of your workspace is still running. | Wait as many seconds as Retry-After says, then repeat. |
Token URL answers Invalid client or Invalid client credentials | The client ID or client secret is wrong, or the client was deleted. | Check the values or replace the secret, see Manage API clients. |
Stuck? Contact us with the client ID, the path and the time of the call.