Configuring Test Sets

Declaration

gradleTestSets {
  testSets {
    main { (1)
    }

    myExtra { (2)
      // .. configure test set
    }
  }
}

1	Configures the `gradleTest` test task and other relevant tasks.
2	Configures the `myExtraGradleTest` test task and other relevant tasks.

For Kotlin users, when you see the above in Groovy examples, change them as follows for Kotlin DSL

import org.ysb33r.gradle.gradletest.GradleTestSet

gradleTestSets {
  testSets {
    testSets.named < GradleTestSet > ("main") {
        versions("8.13", "9.7.1")
    }

    testSets.register < GradleTestSet > ("myExtra") {
        versions("8.13", "9.7.1")
    }
  }
}

Configuring a test set

Define versions to use

The most import configuration is to set which version of Gradle will be used for compatibility testing.

gradleTestSets {
  testSets {
    main {
      versions '7.6.1', '8.9' (1)
    }
  }
}

1	This can be called more than once. A `Provider<List<String>>` can also be used.

These versions can be temporarily overridden from the command-line: See Command-line Awareness.

Treat deprecation messages as failures

By default, all Gradle-emitted deprecation warnings are treated as failures. It is possible to customise this behaviour so that none or a select few are treated as errors. It is even possible to do this on a per version basis.

gradleTestSets {
  testSets {
    main {
      deprecationMessages {
        failIfFound = false (1)
        failOn '.+also check for.+' (2)
        failOn = [ '.+also check for.+'] (3)
        ignoreIf '.+will fail for.+ (4)
      }

      forVersionsMatching(~/^8\./) { (5)
        deprecationMessages {
          failIfFound = true
        }
      }

      forVersion('8.9') { (6)
        deprecationMessages {
          failOn =  [ '.+also check for.+'] (7)
        }
      }
    }
  }
}

1	Turn off checking for all versions by default. The default is `true` and uses a predefined set of message patterns.
2	Add one or more messages to be checked to the default set. Messages are regex strings.
3	Replace the existing set of deprecation message patterns.
4	Ignore any deprecation messages if they also contain this pattern. These patterns are applied after deprecation patterns found warnings and are then used to remove appropriate warnings.
5	Configure on a set of versions that match a pattern. In this example, deprecation messages will fail for Gradle versions starting with `8.`.
6	Configure for one specific Gradle version.
7	By default, deprecation messages that are added on a per-version basis, inherit the set of deprecation messages for the global configuration of the test set. By replacing them using assignment, the inheritance chain is disconnected.

forVersion has specific behaviour for version customisation of deprecation messages.

No version matches

The default configuration in deprecationMessages applies.

One match

The configuration from that version is applied. If inheritance of global test set configuration was not broken, then failure and ignore message patterns are included as applicable.

Multiple matches

Multiple matches are applied in the order that they were declared in the DSL.
If no deprecationMessages` config was executed in the forVersion block, then that match will be excluded from the list of matches specifically for deprecation messages.
All patterns - failures and ignores - are added together.
The behaviour of failIfFound is significant. If one match sets it to false and one sets it to true, then the global test set failure messages are always included. This is something to be aware of when there is a possibility of multiple matches.

Run test with configuration cache

It is possible to run checks for specific Gradle versions with configuration cache enabled. If the specific version does not support it, a warning will be printed and that test will be run without configuration cache. It is recommended to only use this to test with Gradle 8.0+.

There are four values:

NONE: Do not use configuration cache.
WARN: Only print warnings, but do not fail.
FAIL: Fail the build on configuraion cache issues.
DEFAULT: Use the default for specific Gradle version.

import org.ysb33r.gradle.gradletest.ConfigurationCacheMode

gradleTestSets {
  testSets {
    main {
      configurationCache {
        mode = ConfigurationCacheMode.FAIL (1)
        mode = 'fail' (2)
      }

      forVersion('8.3') { (3)
        configurationCache {
          mode = 'none'
        }
      }
    }
  }
}

1	Configure the configuration cache behaviour using the enumeration `org.ysb33r.gradle.gradletest.ConfigurationCacheMode`.
2	Configure the configure cache behaviour using a case-insensitive string
3	Customise configuration cache behaviour on a per-version basis.

forVersion has specific behaviour related to configurationCache.

No version matches

The default configuration in configurationCache applies.

One match

The configuration from that version is applied and overrides whatever was set in the global test set configuration.

Multiple matches

Multiple matches are applied in the order that they were declared in the DSL and the last one wins.
If no configurationCache config was executed in the forVersion block, then that match will be excluded from the list of matches.

Run test for isolated projects

It is possible to run checks for specific Gradle versions with isolated projects enabled. If the specific version does not support it, the configuration will be silently ignored. THe minimum version required for a test is Gradle 9.7.0 as it is the first version where the incubating API was stable enough.

There are four values:

NONE: Do not test isolated projects.
DIAGNOTICS: Only print diagnostics.
WARN: Try to use isolated projects, printing warnings where issues are met.
FAIL: Fail the build on isolated project issues.

import org.ysb33r.gradle.gradletest.ConfigurationCacheMode

gradleTestSets {
  testSets {
    main {
      isolatedProjects {
        mode = IsolatedProjectMode.FAIL (1)
        mode = 'fail' (2)
      }

      forVersion('8.3') { (3)
        isolatedProjects {
          mode = 'none'
        }
      }
    }
  }
}

1	Configure isolation project behaviour using the enumeration `org.ysb33r.gradle.IsolatedProjectMode.ConfigurationCacheMode`.
2	Configure isolation project behaviour using a case-insensitive string
3	Customise isolation project behaviour on a per-version basis.

forVersion has specific behaviour related to isolatedProjects.

No version matches

The default configuration in isolatedProjects applies.

One match

The configuration from that version is applied and overrides whatever was set in the global test set configuration.

Multiple matches

Multiple matches are applied in the order that they were declared in the DSL and the last one wins.
If no isolatedProjects config was executed in the forVersion block, then that match will be excluded from the list of matches.

Customise tests to fail

You can also write some tests where you expect the build script to fail either at configuration or execution time. To achieve this, pass one or more patterns to match the test groups that will fail.

gradleTestSets {
  testSets {
    main {
      expectedFailures 'badConfig' (1)
      expectedFailures 'badConfig.+' (2)
    }
  }
}

1	Expect a test called `badConfig` to fail.
2	Expect any test starting with the name `badConfig` to fail.

Use a custom manifest

When the standard plugin, as well as the java-gradle-plugin is applied, a manifest will automatically be added to the test run.

It is possible to use a custom generated manifest.

gradleTestSets {
  testSets {
    main {
      useCustomManifest()
    }
  }
}

If your project does not use the java-gradle-plugin, you’ll also need to set useCustomManifest().

Override the JDK version

If you want to declare additional tese sets which use different JDK versions you can declare this right in the DSL. You also have the option of declaring a different JDK with variant reselection. The latter is useful when transitive dependencies change is you use different JDK versions and you want to ensure that the correct ones are on the classpath.

gradleTestSets {
  testSets {
    main {
      jdk {
        useJdk(21) (1)
        useWithVariantReselection( 21 ) (2)
        useWithVariantReselection( 21 ) { (3)
            it.attribute( Usage.USAGE_ATTRIBUTE, objects.named(Usage, Usage.JAVA_RUNTIME) )
        }
      }
    }
  }
}

1	Use a custom JDK. There is also an option to provide a `Provider<JavaLauncher>`.
2	Use a custom JDK but override the JDK attribute to match.
3	Like before, but perform additional attribute manipulation.

The settings plugin org.gradle.toolchains.foojay-resolver-convention is not supported on a Gradle build version < 7.6. If this is your build version, then you’ll need to configure the jvmToolchainRepository manually in settings.gradle.

Passing additional command-line arguments

Use gradleArguments to pass additional arguments.

gradleTestSets {
  testSets {
    main {
      gradleArguments '-s'
    }
  }
}

Do not use this to modify configuration cache behaviour, or activate daemon mode.

Debugging tests

If you want to debug tests then you can turn on Debug mode. Note that if you use configuration cache on tests or Kotlin DSL on some older version of Gradle, debug will not be turned on and a warning will be logged.

gradleTestSets {
  testSets {
    main {
      debug = true
    }
  }
}

If the Jacoco plugin is applied, debugging is turned on automatically.

Configuring a test set (Advanced)

Copy over symlink

Project structures are symlinked back to the files in the project source directory, saves a lot of disk space and speeds things up, but it can cause issues if the test writes anything back into directory that was symlinked. This behaviour can be turned off, so that directories are copied, but toplevel files are still symlinked.

Since 4.1.0 the default behaviour is to copy, rather than symlink.

gradleTestSets {
  testSets {
    main {
      copyNotSymlink(true) (1)
    }
  }
}

1	`true` will make full copies. `false` will try to symlink toplevel files.

Downloading Gradle distributions

The default behaviour is to download Gradle distributions for testing from the Gradle repository. If you want to use another repository, or you are using something like Ivypot, then you can set a new URI. GradleTest will then append gradle-<VERSION>-bin.zip to the URI and pass that to GradleTestKit.

gradleTestSets {
  testSets {
    main {
      gradleDistributionURI = 'https://our.internal.server/gradle/distributions' (1)
      gradleDistributionURI = ivypot.repoDetails.getBinariesRootFor('gradle').map { new File(it,'distributions') } (2)
    }
  }
}

1	Set a URI. It can be a string, a URI, a File, a Path or anything lazy-evaluated to one of those.
2	Example of using it with Ivypot.

TestKit configuration

It is possible to configure how TestKit caching and data is handled by GradleTest, This is usually a trade-off between storage and speed. It can also be used to debug some hard-to-find issues.

gradleTestSets {
  testSets {
    main {
      testKitDirectoryShared() (1)
      testKitDirectoryPerGroup() (2)
      testKitDirectoryPerTest() (3)
    }
  }
}

1	Share on directory between all tests in the source set. For the default test set, the folder will be `build/gradleTest/testkitdata`
2	Keep one folder for each group in the test set. If you have `src/gradleTest/testA` and `src/gradleTest/testB`, then the TestKit data will be in `build/gradleTest/testkitdata/testA` and `build/gradleTest/testkitdata/testB`.
3	Keep one TestKit directory per test. This option uses the most space, but allows detailed analysis per test. A folder could be something like `build/gradleTest/testkitdata/8.9/testA`.

Clean the project cache directory

By default, the .gradle folder of a defined project GradleTest folder is cleaned before the test starts. It is possible to turn off this behaviour.

gradleTestSets {
  testSets {
    main {
      cleanCache = false
    }
  }
}

Override the default task to run

By default, GradleTest will look for a task called runGradleTest. If you want to override this for a test set, then do the following.

gradleTestSets {
  testSets {
    main {
      defaultTaskToRun = 'startTestsHere'
    }
  }
}

Override the source directory

By default, the source directory is src/gradleTest or for a test called myTest, it will be src/myTestGradleTest.

It is possible to override the source directory.

gradleTestSets {
  testSets {
    main {
      sourceDirectory = 'src/meNotFollowingConvention'
    }
  }
}

Override the generated source directory

By default, test code is generated below the .generated-src/gradleTestPlugin/gradleTest/src/groovy folder. For a test set called myTest, this will be `.generated-src/gradleTestPlugin/myTestGradleTest/src/groovy

It is possible to override thyis output directory.

gradleTestSets {
  testSets {
    main {
      generatedSourceDirectory = 'build/src/meNotFollowingConvention'
    }
  }
}