GradleTestKit is the core API that ships with Gradle since Gradle 2.6. which is meant for plugin developers to test compatibility across Gradle versions. Early versions could only test against the same version of Gradle that was running, but it has been enhanced over different Gradle releases. Today it is powerful allowing multiple versions to be tested and a number of different environments to be set up.

It is also a core component of GradleTest when the version of Gradle is 2.13 or later. (When the current build version is older GradleTest falls back to legacy mode, simply because the GradleTestKit API either is non-existent or not mature enough.

GradleTestKit works well for integration testing when writing plugins if a select small group of Gradle versions are used or even if it only targets the current Gradle version. In this way one can quickly determine issues in a plugin which cannot be discovered via normal testing. GradleTest used GradleTestKit directly for its integration tests for exactly this reason.

Testing multiple versions of Gradle with GradleTestKit soon becomes a burden for developers as the API differs and classpath setups can be tricky. Older versions of Gradle give the most issues. In addition GradleTestKit is essentially a library that focuses on the developer rather than the build script user.

In this regard GradleTest steps in as it allow tests to be written as if a normal buildscript would be written. It also manages differences (or non-existence) in API from Gradle 2.0 onwards in a completely transparent way to the plugin author.

Overall the approach should be to write integration tests with GradleTestKit using a limited or even just one Gradle version. This is then complimented by some GradleTest scripts to test a wider range of Gradle versions.

Stutter is another library by Andy Oberstar to aid with plugin development. It focuses on the cases where you might have to run the same tests against all supported versions, but don’t want to deal with making each test loop over those versions (or @Unroll with a where block in Spock). Stutter generates one task per supported version for you and runs the full set of tests against it. It definitely removes some of the complexity of setting up the GradleTestKit environment.

It adds a compatTest source set, which will not clash with the gradleTest source set from GradleTest.

Stutter is essentialy developer-centric whereas GradleTest is script author-centric. This distinction is important as it clarifies different levels of validation:

It can be an easy way to manage some more complex plugin integration tests. There is no reason to not use it alongside GradleTest as part of plugin development.