Test sets

Declaration

gradleTestSets {
  testSets {
    main { (1)
    }

    myExtra { (2)
      // .. configure test set
    }
  }
}

1	Configures the `gradleTest` test task and other relevant tasks.
2	Configures the `myExtraGradleTest` test task and other relevant tasks.

For Kotlin users, when you see the above in Groovy examples, change them as follows for Kotlin DSL

import org.ysb33r.gradle.gradletest.GradleTestSet

gradleTestSets {
  testSets {
    val main by getting(GradleTestSet::class) {
    }

    val myExtra by creating(GradleTestSet::class) {
      // .. configure test set
    }
  }
}

Configuring a test set

Define versions to use

The most import configuration is to set which version of Gradle will be used for compatibility testing.

gradleTestSets {
  testSets {
    main {
      versions '7.6.1', '8.9'
    }
  }
}

Passing additional command-line arguments

Use gradleArguments to pass additional arguments.

gradleTestSets {
  testSets {
    main {
      gradleArguments '-s'
    }
  }
}

Do not use this to modify configuration cache behaviour, or activate daemon mode.

Treat deprecation messages as failures

By default, all Gradle-emitted deprecation warnings are treated as failures. It is possible to customise this behaviour so that none or a select few are treated as errors. It is even possible to do this on a per version basis.

gradleTestSets {
  testSets {
    main {
      deprecationMessagesAreFailuresForAllVersions = false (1)
      deprecationMessagesAreFailures { ver -> (2)
        ver.startsWith('8.')
      }

      defaultDeprecationMessageChecks '.+also check for.+' (3)
      defaultDeprecationMessageChecks = [ '.+also check for.+'] (4)

      replaceDeprecationMessageChecksForVersion '8.9', ['.+also check for.+'] (5)
      deprecationMessageChecksForVersion('8.9', '.+also check for.+') (6)
      augmentMessageChecksForVersionFromDefault '8.9', 'only check this message.+' (7)
    }
  }
}

1	Turn off checking for all versions.
2	Only check for deprecation warnings when the version matches.
3	Add one or more messages to be checked, to the default set.
4	Replace the existing set of deprecation message patterns.
5	Replace the existing set of deprecation message patterns for a specific version.
6	Adds one or more patterns for a given version. If the version has not been added before, it will be the same as if `replaceDeprecationMessageChecksForVersion` was called.
7	Replaces the existing set of patterns for a given version with a combination of the default set of patterns and the provided set of patterns.

Customise tests to fail

You can also write some tests where you expect the build script to fail either at configuration or execution time. To achieve this, pass one or more patterns to match the test groups that will fail.

gradleTestSets {
  testSets {
    main {
      expectedFailures 'badConfig' (1)
      expectedFailures 'badConfig.+' (2)
    }
  }
}

1	Expect a test called `badConfig` to fail.
2	Expect any test starting with the name `badConfig` to fail.

Use a custom manifest

When the standard plugin, as well as the java-gradle-plugin is applied, a manifest will automatically be added to the test run.

It is possible to use a custom generated manifest.

gradleTestSets {
  testSets {
    main {
      useCustomManifest()
    }
  }
}

If your project does not use the java-gradle-plugin, you’ll also need to set useCustomManifest().

Debugging tests

If you want to debug tests then you can turn on Debug mode. Note that if you use configuration cache on tests or Kotlin DSL on some older version of Gradle, debug will not be turned on and a warning will be logged.

gradleTestSets {
  testSets {
    main {
      debug = true
    }
  }
}

If the Jacoco plugin is applied, debugging is turned on automatically.

Run test with configuration cache

It is possible to run checks for specific Gradle versions with configuration cache enabled. If the specific version does not support it, a warning will be printed and that test will be run without configuration cache. It is recommended to only use this to test with Gradle 8.0 and up.

import org.ysb33r.gradle.gradletest.ConfigurationCacheMode

gradleTestSets {
  testSets {
    main {
      configurationCacheMapping { ver -> (1)
        ver.startsWith('8.') ? ConfigurationCacheMode.FAIL : ConfigurationCacheMode.NONE
      }
    }
  }
}

1	The function is passed a Gradle version as a string and should return an instance of `org.ysb33r.gradle.gradletest.ConfigurationCacheMode`. There are three values: `NONE`, `WARN` and `FAIL`.

Configuring a test set (Advanced)

Copy over symlink

By default, project structures are symlinked back to the files in the project source directory. This saves a lot of disk space and speeds things up, but it can cause issues if the test writes anything back into directory that was symlinked. This behaviour can be turned off, so that directories are copied, but toplevel files are still symlinked.

gradleTestSets {
  testSets {
    main {
      copyNotSymlink()
    }
  }
}

Downloading Gradle distributions

The default behaviour is to download Gradle distributions for testing from the Gradle repository. If you want to use another repository, or you are using something like Ivypot, then you can set a new URI. GradleTest will then append gradle-<VERSION>-bin.zip to the URI and pass that to GradleTestKit.

gradleTestSets {
  testSets {
    main {
      gradleDistributionURI = 'https://our.internal.server/gradle/distributions' (1)
      gradleDistributionURI = ivypot.repoDetails.getBinariesRootFor('gradle').map { new File(it,'distributions') } (2)
    }
  }
}

1	Set a URI. It can be a string, a URI, a File, a Path or anything lazy-evaluated to one of those.
2	Example of using it with Ivypot.

TestKit configuration

It is possible to configure how TestKit caching and data is handled by GradleTest, This is usually a trade-off between storage and speed. It can also be used to debug some hard-to-find issues.

gradleTestSets {
  testSets {
    main {
      testKitDirectoryShared() (1)
      testKitDirectoryPerGroup() (2)
      testKitDirectoryPerTest() (3)
    }
  }
}

1	Share on directory between all tests in the source set. For the default test set, the folder will be `build/gradleTest/testkitdata`
2	Keep one folder for each group in the test set. If you have `src/gradleTest/testA` and `src/gradleTest/testB`, then the TestKit data will be in `build/gradleTest/testkitdata/testA` and `build/gradleTest/testkitdata/testB`.
3	Keep one TestKit directory per test. This option uses the most space, but allows detailed analysis per test. A folder could be something like `build/gradleTest/testkitdata/8.9/testA`.

Clean the project cache directory

By default, the .gradle folder of a defined project GradleTest folder is cleaned before the test starts. It is possible to turn off this behaviour.

gradleTestSets {
  testSets {
    main {
      cleanCache = false
    }
  }
}

Override the default task to run

By default, GradleTest will look for a task called runGradleTest. If you want to override this for a test set, then do the following.

gradleTestSets {
  testSets {
    main {
      defaultTaskToRun = 'startTestsHere'
    }
  }
}

Override the source directory

By default, the source directory is src/gradleTest or for a test called myTest, it will be src/myTestGradleTest.

It is possible to override the source directory.

gradleTestSets {
  testSets {
    main {
      sourceDirectory = 'src/meNotFollowingConvention'
    }
  }
}

Override the generated source directory

By default, test code is generated below the .generated-src/gradleTestPlugin/gradleTest/src/groovy folder. For a test set called myTest, this will be `.generated-src/gradleTestPlugin/myTestGradleTest/src/groovy

It is possible to override thyis output directory.

gradleTestSets {
  testSets {
    main {
      generatedSourceDirectory = 'build/src/meNotFollowingConvention'
    }
  }
}