Not sure if I precisely described it in the title but see this example:

apply plugin: 'com.android.application'
apply plugin: 'checkstyle'

android {
    ...

    android.applicationVariants.all { variant ->
        task "checkstyle${variant.name.capitalize()}"(type: Checkstyle) {
            showViolations true
            dependsOn variant.javaCompile
            classpath = project.fileTree(variant.javaCompile.destinationDir)

            source variant.javaCompile.source // Contains files from both build and src 
            exclude '**/generated/source/**'
        }
    }
}

As I see Checkstyle is SourceTask that uses FileTree.matching(PatternFilterable) under the hood to get input files.

Expected Behavior

All files under generated/source/ are excluded from checkstyle task.

Current Behavior

Doesn't work:
exclude '**/generated/source/**'
exclude '**/build/**'

Works:
exclude '**/R.java'
exclude '**/com/**' (com is located under generated/source)

Steps to Reproduce (for bugs)

1. Clone the repo from https://github.com/technoir3/gradle-exclude-subdirectory-bug
2. Run ./gradlew checkstyleDebug
3. Notice the failures even though **/generated/source/** is listed in exclude patterns

Your Environment

Gradle 4.3
Android Gradle Plugin 3.0.0