testrail-testng

A TestNG listener that pushes automated test results into TestRail, plus the Retrofit client underneath it if you want to talk to the API directly.

Annotate a test with the case it covers, register the listener, and results land in TestRail when the suite finishes:

@Listeners(TestRailListener.class)
public class CheckoutTest {

    @Test
    @TestCase(1042)
    public void guestCanCheckOut() {
        // …
    }
}

mvn test \
  -Dtestrail.url=https://acme.testrail.io \
  -Dtestrail.username=qa@acme.io \
  -Dtestrail.apiKey=$TESTRAIL_API_KEY \
  -Dtestrail.projectId=1 \
  -Dtestrail.suiteId=2

Why this rather than a few lines of RestAssured

The interesting parts are the ones that only show up once a suite grows:

• One request per suite, not per test. Results accumulate in memory and go out through add_results_for_cases when the run finishes. On 400 tests that is one HTTP round trip instead of 400, which on a slow TestRail instance is the difference between seconds and minutes of tail time on every build.
• The worst status wins. When several tests map to the same case, or a test is retried, the case ends up reflecting the most severe outcome. A green retry never quietly paints over a red first attempt.
• Reporting failures never fail the build. Every TestRail interaction is caught and logged. An expired API key is a reporting problem, not a test failure, and it should not turn a green suite red at 3am.
• Credentials never reach the log. Authorization, Cookie and friends are redacted before anything is written, at every log level. Test logs end up in CI artifacts and bug reports, and a Basic-auth header printed verbatim has a very long tail.
• Both response shapes parse. TestRail 6.7 wrapped list endpoints in a pagination envelope; older instances and a few endpoints still return a bare array. Either one deserializes.

Install

Not published to Maven Central. Build and install locally:

git clone https://github.com/sp1r1n/testrail.git
cd testrail
mvn install

<dependency>
    <groupId>io.github.sp1r1n</groupId>
    <artifactId>testrail-testng</artifactId>
    <version>1.0.0</version>
    <scope>test</scope>
</dependency>

TestNG is declared provided, so your project's own version is the one that gets used. SLF4J is API-only — bring whichever binding you already have.

Requires Java 17+ and TestRail 6.7+.

Configuration

Every setting resolves in the same order: builder call, then JVM system property, then environment variable, then the default. That lets a project pin the stable parts in code while CI supplies credentials without touching the source.

System property	Environment	Default	Meaning
testrail.url	TESTRAIL_URL	—	Instance URL, e.g. https://acme.testrail.io
testrail.username	TESTRAIL_USERNAME	—	Account email
testrail.apiKey	TESTRAIL_API_KEY	—	API key from My Settings → API Keys
testrail.enabled	TESTRAIL_ENABLED	true when a URL is set	Master switch
testrail.projectId	TESTRAIL_PROJECT_ID	—	Project to create the run in
testrail.suiteId	TESTRAIL_SUITE_ID	—	Suite the run covers
testrail.runId	TESTRAIL_RUN_ID	—	Report into an existing run instead of creating one
testrail.runName	TESTRAIL_RUN_NAME	Automated run	Prefix; a timestamp is appended
testrail.closeRun	TESTRAIL_CLOSE_RUN	false	Close the run afterwards (only if this listener created it)
testrail.skipAssigned	TESTRAIL_SKIP_ASSIGNED	false	Leave cases whose result someone has claimed
testrail.logLevel	TESTRAIL_LOG_LEVEL	BASIC	NONE, BASIC, HEADERS, BODY
testrail.timeoutSeconds	TESTRAIL_TIMEOUT_SECONDS	30	Connect, read and write timeout

Give it either runId, or projectId and suiteId so it can create a run. With neither, the listener logs a warning and stays out of the way.

An API key rather than a password: it can be revoked on its own, without locking the account out of TestRail.

Usage

Mapping tests to cases

@Test
@TestCase({1042, 1043})          // one result per id
public void userCanResetPassword() { … }

@Test
@TestCase(1044)
@Defects("JIRA-317")             // result arrives already linked to the ticket
public void knownBrokenFlow() { … }

Methods without @TestCase are ignored, so the listener can stay registered suite-wide while only the mapped tests reach TestRail.

Status mapping: passed → Passed, failed → Failed, skipped → Blocked. TestRail has no "skipped", and Blocked keeps the case visibly unverified rather than letting it read as untested.

Reporting into a run created elsewhere

When a pipeline step creates the run — so several parallel jobs share it — pass the id in:

mvn test -Dtestrail.runId=8123

or set it before the suite starts:

TestRailListener.useRun(8123);

Registering the listener

Either the annotation:

@Listeners(TestRailListener.class)
public class BaseTest { }

or testng.xml, which also gets you the suite-level callbacks:

<suite name="regression">
    <listeners>
        <listener class-name="io.github.sp1r1n.testrail.TestRailListener"/>
    </listeners>
    …
</suite>

Using the client on its own

The API client has no TestNG dependency and is usable from any code:

TestRailConfig config = TestRailConfig.builder()
        .baseUrl("https://acme.testrail.io")
        .username("qa@acme.io")
        .apiKey(System.getenv("TESTRAIL_API_KEY"))
        .build();

try (TestRailClient client = new TestRailClient(config)) {
    Cases cases = client.execute(client.cases().getCases(1, 2, null, 250, 0));
    cases.items().forEach(c -> System.out.println(c.id + " " + c.title));
}

execute unwraps the Retrofit response and turns anything non-2xx into a TestRailException carrying TestRail's own error message, which is far more useful than a bare 400.

Covered endpoints: cases (get_cases, get_case, add_case, update_case), runs (get_run, get_runs, add_run, update_run, close_run), results (get_results, get_results_for_case, get_results_for_run, add_result, add_result_for_case, add_results, add_results_for_cases) and suites.

How the URL rewriting works

TestRail routes its whole API through one PHP entry point and puts the API path inside the query string:

https://acme.testrail.io/index.php?/api/v2/get_case/42&suite_id=3

Retrofit cannot produce that — it rejects ? in a base URL and joins query parameters the normal way. So the client is configured with the marker-free path index.php/api/v2/, Retrofit builds an ordinary URL, and UrlInterceptor does the last-mile translation: every ? becomes &, then the single marker TestRail wants is inserted after index.php. UrlInterceptorTest pins the behaviour down.

Development

mvn verify          # compile, test, coverage report in target/site/jacoco
mvn test -Dtest=TestRailListenerTest

The listener's tests drive a real TestNG suite against a MockWebServer and assert on the requests that come out. Running TestNG inside TestNG looks odd, but the listener's contract is its interaction with the framework — annotation lookup, callback ordering, the suite lifecycle — and calling the callbacks by hand would only test a model of TestNG.

License

MIT