Working with files docs #4293

pledbrook · 2018-02-07T15:31:37Z

DO NOT MERGE!

This is a series of changes to the Working With Files chapter of the user guide that I'm collecting into a pull request to make reviewing easier. Hopefully the changes will eventually propagate to the user guide, but for now they should not be merged.

I have started the chapter with a series of common tasks that users want to accomplish with files. The aim is to answer some commonly asked questions while directing readers to more detailed information so that they have a broader understanding of the topic.

bmuschko

Hi Peter,

I ran over the preliminary content and left a couple of comments. Overall, the content is already much better than before. I am going to give a more thorough pass when you think the changes are in a somewhat final state.

bmuschko · 2018-02-07T18:52:30Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

 == Working With Files

-Most builds work with files. Gradle adds some concepts and APIs to help you achieve this.
+Almost every Gradle build works with files: think source files, library dependencies, reports and so on. That's why Gradle comes with a comprehensive API that makes it simple to achieve common tasks.


Instead of "library dependencies" we settled on the term "file dependency". See https://docs.gradle.org/nightly/userguide/declaring_dependencies.html#sec:declaring_file_dependency for example.

Use "file dependency" in place of "library dependency"

bmuschko · 2018-02-07T18:53:03Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

 == Working With Files

-Most builds work with files. Gradle adds some concepts and APIs to help you achieve this.
+Almost every Gradle build works with files: think source files, library dependencies, reports and so on. That's why Gradle comes with a comprehensive API that makes it simple to achieve common tasks.


In this section we can probably be more concrete. I'd say "file operations" instead of "common tasks".

Be more concrete and less ambiguous than "common tasks"

bmuschko · 2018-02-07T18:55:51Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+You copy a file by creating an instance of Gradle's builtin api:org.gradle.api.tasks.Copy[] task and configuring it with the location of the file and where you want to put it. This example mimics copying a generated report into a directory that will be packed into an archive (zip, tarball, or whatever):
+
+----


All code samples should be an embedded code snippet from an actual build script under docs/src/samples. Furthermore, all of them should be tested e.g. underneath integ-test/src/integTest/groovy/org/gradle/integtests/samples.

Understood. I simply wanted to get the text done first. BTW, are there any instructions for running the docs integration tests or a subset of them? I remember working it out last year, but have of course forgotten and can't find any info on it.

Turn code examples into tested samples

Update existing samples to use best practice and reflect their corresponding points as succinctly and well as possible

There are two types of tests you can run:

The UserGuideSamplesIntegrationTest which basically only verifies that the expected output has been produced according to the *.out that follow the naming convention of the sample ID. Instructions are in the comment of the test class.

The corresponding integration test(s) that execute the sample code and make assertions about the outcome. One example is SamplesDeclaringDependenciesIntegrationTest. After extracting the samples (as described in 1) you can just run it from the IDE. It's possible that they do not exist for this page. I'd set up a new test class if it doesn't exist yet.

When reworking content in the dependency management chapter I have been leaning mostly toward 2 as it provides a way verify more than just the output.

I'm running into an issue trying to execute the dedicated integration tests from IntelliJ:

java.lang.ExceptionInInitializerError at org.gradle.integtests.fixtures.AbstractIntegrationSpec.$spock_initializeFields(AbstractIntegrationSpec.groovy:56) Caused by: org.gradle.internal.service.ServiceCreationException: Could not create service of type ClassLoaderRegistry using GlobalScopeServices.createClassLoaderRegistry(). at org.gradle.integtests.fixtures.executer.AbstractGradleExecuter.<clinit>(AbstractGradleExecuter.java:92) ... 1 more Caused by: java.lang.IllegalArgumentException: Cannot find JAR 'slf4j-api-1.7.16.jar' required by module 'gradle-core' using classpath. at org.gradle.api.internal.classpath.DefaultModuleRegistry.findDependencyJar(DefaultModuleRegistry.java:254) ... 9 more

I imported the the project after running ./gradlew idea. Not sure what else I need to do. Any ideas?

Hhhm, doesn't sounds familiar. Can you try the following?

Run ./gradlew cleanIdea idea from the command line. Do not import the project by pointing to the root build.gradle file. There're still some things that don't work properly yet with the built-in Gradle import functionality. Just open the generated gradle.ipr file.

Run ./gradlew intTestImage from the command line.

Run the test from the IDE.

Thanks. I ran into another issue with a conflicting groovy-all library, but hacked that to get the tests running from within the IDE.

OK, great. Saw your question on the channel. Looks like it was a bug.

bmuschko · 2018-02-07T18:57:14Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+We look at these parts in more detail later in the chapter, but we'll start with how you accomplish some of the most common tasks users need.
+
+=== Copying a single file


I think it would make sense to also explain the from/into methods to a certain extent.

Agreed, although I'll keep it minimal. Now that I have the later sections in place, it will be easier to reference them appropriately.

Explain what from() and into() do/are for

bmuschko · 2018-02-07T18:58:58Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+Note that hardcoding paths like this makes the build brittle. It's better to use the task that generates a file as the source of the file's location. In this modified example, we use a report task defined elsewhere that has the report's location stored in the `outputFile` property:
+
+----
+ext.archiveDir = file("${buildDir}/toArchive")


I'd define the extra property for the task and not the project. That will provide a narrower context and can only be used with the reference of this specific task. Ultimately, it avoids introducing global properties which makes the build harder to maintain.

Agreed. I was thinking in terms of a generic directory where all archives are put, but that's not the case here. It's specific to this task.

Move project property onto the task

bmuschko · 2018-02-07T19:03:27Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+----
+task packageDistribution(type: Zip) {
+    archiveName = "my-distribution.zip"


Maybe we should promote the use of the base plugin here instead as it provides the conventions.

Since there are two examples, I'd like to have one without the plugin, to demonstrate that it's not needed, and one with to show the convenience. I'll probably trim this section a bit too, as I've now updated the in-depth section on creating archives which covers the base plugin and naming conventions in detail.

Use one sample without the Base Plugin, and one with

bmuschko · 2018-02-07T19:04:59Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+}
+----
+
+=== Moving files and directories


Good to spell it out!

bmuschko · 2018-02-07T19:07:29Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+}
+----
+
+As you can see, you can have multiple `from()` declarations in a copy specification, each with its own configuration.


We should have a section that explain the file operation methods on Project (e.g. project.copy) and the difference between the task types and the methods.

That was already in the chapter, but I've updated it. You'll find it in the File Copying in Depth section. It's actually focused on the copy() method, but now you mention it, there isn't anything on delete() or its corresponding task right now. Are there any others I should cover?

Add information on the following:

Delete task and method

sync() method

I can only think of Project.mkdir() and Project.delete() right now. I don't think we need to show examples for all of them. It's probably OK to just list them and link to the DSL guide.

bmuschko · 2018-02-07T19:08:35Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+One thing to note is how the `include()` directive applies only to the `from()`, whereas the directive in the previous section applied to the whole task. These different levels of granularity in the copy specification allow you to easily handle most requirements that you will come across. TODO: Link to more details on copy specs.
+
+=== Creating archives (zip, tar, etc.)


Frequently asked about on the user forum: How do I create an uberjar? It would be helpful to explain it here.

Good idea. If it's that frequently asked, it might make sense to include it as one of the early examples in the chapter, with perhaps some more detail in the in-depth section.

Provide an example on creating an uber JAR

@bmuschko Do you mean one in which the contents of the other JARs are included in the fat JAR, or the JARs themselves are included as is in the fat JAR (the Spring Boot approach — or at least it was).

I meant 1. For example:

task uberjar(type: Jar) { from zipTree(jar1) from zipTree(jar2) }

2 is similar except that the contents of the JAR files (to be included) are not extracted via zipTree.

bmuschko · 2018-02-07T19:15:08Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+This is not a common requirement and should be used sparingly as you lose information and can easily break a build. It's generally preferable to copy directories and files instead.
+
+=== Renaming files on copy


One example on more complex processing would be great as well e.g. the use of eachFile.

Is there a common scenario for users in which something like eachFile() is the best solution? Or is it only for specialised cases?

pledbrook · 2018-02-15T09:09:02Z

@bmuschko @eriwen Are we using the guides style guide on the user manual? Is there another document? I hate not knowing how to refer to plugins by name, whether to capitalize archive format names, and so on.

bmuschko · 2018-02-16T22:29:28Z

Are we using the guides style guide on the user manual? Is there another document? I hate not knowing how to refer to plugins by name, whether to capitalize archive format names, and so on.

The guides style guide contains some good advice. I don't think we have been following it in the past when writing user guide content. It would make sense to enhance the style guide by additional rules and then use it across the board. @eriwen Any additional thoughts?

bmuschko · 2018-02-16T22:50:31Z

subprojects/docs/docs.gradle


-    attributes website: 'http://www.gradle.org'
+    attributes website: 'http://www.gradle.org',
+               javaApi: 'https://docs.oracle.com/javase/8/docs/api'


We might want to reference the Java 7 docs here as we still supporting it as the minimal JDK version.

I'm fine with that. One of the reasons I made it a document attribute :)

Reference Java 7 API docs rather than Java 8

eriwen

Overall, these are incredible updates. I want to ship this as soon as possible, even if it's not completely done, because I think the ~40000 users who will visit this page between Gradle 4.6 and 4.7 will really appreciate it. There are only a couple fixes necessary for this, which I've marked with an ❌

eriwen · 2018-02-20T17:06:05Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+----
+task copyReports(type: Copy) {
+    from "${buildDir}/reports/my-report.pdf", "src/docs/manual.pdf"


Should we use myReportTask.outputFile here as well?

Explicit file paths are easier to understand, so I prefer them in simple examples. But I do think we need to warn against the use of hard-coded paths in more places than is currently done.

Warn against the use of hard-coded paths in all appropriate places within the chapter

Sounds good

eriwen · 2018-02-20T17:08:56Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+}
+----
+
+This approach has the side effect of copying the directory structure below `reports` as well as the files. This may be what you want. It may not be. There are many such decisions you need to make when copying files, but rest assured that there are very few requirements that can't be handled elegantly by Gradle copy specifications.


This may be what you want. It may not be. There are many such decisions you need to make when copying files

I weakly feel that this particular text doesn't really help users much. I think we can be more succinct here.

Yes, I'm not particularly happy with it. Will revisit.

Reword above

eriwen · 2018-02-20T17:10:36Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+You may have the need to copy not just files, but the directory structure they reside in as well. This is, in fact, the default behavior when you specify a directory as the `from()` argument. So the following will copy everything in the `reports` directory, including all its subdirectories:
+
+----
+task copyReports(type: Copy) {


As we convert these to samples, I expect these bits will have meaningful example subtitles, which will be useful for those folks just skimming this doc.

eriwen · 2018-02-20T17:12:01Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+----
+task copyReports(type: Copy) {
+    from "${buildDir}/reports"


One thing I've always found confusing is how much the order of these things matter. I think it'd be nice to be explicit whether include must come before into or that it doesn't matter.

Another option is simply to assume order independence and call out when order does matter. That reflects my own view that this is configuration and should inherently be order independent. Making this explicit everywhere in the manual might be a little over the top. Not sure.

I'm okay with that approach as I don't feel strongly here

eriwen · 2018-02-20T17:12:24Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+One thing to note is how the `include()` directive applies only to the `from()`, whereas the directive in the previous section applied to the whole task. These different levels of granularity in the copy specification allow you to easily handle most requirements that you will come across. You can learn more about this in the section on <<Using child specifications,child specifications>>.
+
+[[sec:creating_archives_example]]
+=== Creating archives (zip, tar, etc.)


Very good title

eriwen · 2018-02-21T18:55:05Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
+As you can see, the api:org.gradle.api.file.CopySpec#with(org.gradle.api.file.CopySpec...)[] method is the key to incorporating an existing copy specification into another one.
+
+===== Using child specifications


eriwen · 2018-02-21T18:56:36Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+One final thing to be aware of is that a child copy spec inherits its destination path, include patterns, exclude patterns, copy actions, name mappings and filters from its parent. So be careful where you place your configuration.
+
+[[sec:project_copy_method]]
+==== Copying files in your own tasks


I wonder if this deserves it's own subsection because it feels like a common use case. That is, use of === Copying ... to ensure it gets it's own spot in the side navigation.

I don't know. It would certainly be easier for readers to find currently if they could browse the second-level headings as well. Even if it doesn't make sense to do this in the side nav, I think having a TOC at the front of the chapter would help.

On a general note, I'm worried that we'll end up with a suboptimal structure because we're trying to satisfy a constraint of only top-level headings being visible in the navigation. It makes browsing and discovery harder than it would otherwise be.

Hmmm...I thought I wrote something on this, but I can't remember where. I feel like the single-level heading in the navigation is forcing a structure that isn't that easy for readers to parse and comprehend. Using the copy method is still technically file copying and it would be confusing to lift it up a level just so that it appears in the side nav. And I would use the same argument for file collections and file trees, both of which should be discoverable without a page search in my view.

If I have talked about this elsewhere, sorry for repeating myself 😄

OK, browser had a cached copy of the page without my previous comments in it. Sorry about that!

eriwen · 2018-02-21T18:58:07Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+TODO: Use a generic archive example. No need to use Java here.
+
 [TIP]
 .Why are you using the Java plugin?


Just a reminder to remove this if you use an example that doesn't need the Java plugin.

eriwen · 2018-02-21T19:00:08Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

 ==== Archive naming

-The format of `__projectName__-__version__.__type__` is used for generated archive file names. For example:
+Gradle has several conventions around the naming of archives and where they are created based on the plugins your project uses. The main convention is provided by the `base` plugin, which defaults to creating archives in the `$buildDir/distributions` directory and typically uses archive names of the form _[projectName]-[version].[type]_. The following example comes from a project named 'zipProject', hence the `myZip` task creates an archive named 'zipProject-1.0.zip':


We mention the base plugin here a few times. Is there a good place we can link to where a user can learn more about what it is?

Link to information on the Base Plugin

eriwen · 2018-02-21T19:05:06Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc

+
 [[archiveTasksNamingProperties]]
 .Archive tasks - naming properties
 [cols="a,a,a,a", options="header"]


We should try to convert most tables we come across to a definition list format. For example: https://docs.gradle.org/current/userguide/java_plugin.html#sec:java_tasks as done in 4ae351f#diff-cada99fea6a20ffb6871debbd42b6d7f which makes the content respond much better to various widths.

I'll look into it, but it is a 4-column table

eriwen · 2018-02-21T19:17:48Z

subprojects/docs/src/docs/userguide/workingWithFiles.adoc


 [[sec:properties_files]]
-=== Properties files
+=== Java properties files


This subsection doesn't seem to fit in very well. Is there another place it belongs?

Completely agree, but left it in for now. Should probably go in the Java Plugin chapter.

Would you be opposed to moving it in this PR?

pledbrook added 4 commits February 2, 2018 13:59

Restructure section on file collections et al.

caf531d

Merge branch 'master' into working-with-files-docs

4c30a1a

Add missing image files

fe18c54

eriwen self-requested a review February 7, 2018 15:53

eriwen assigned pledbrook Feb 7, 2018

eriwen added from:member in:documentation-infrastructure labels Feb 7, 2018

eriwen requested a review from bmuschko February 7, 2018 15:55

bmuschko suggested changes Feb 7, 2018

View reviewed changes

Finish first draft of Working with Files update

20ceae0

Incorporate some of bmuschko's feedback

337ca34

bmuschko reviewed Feb 16, 2018

View reviewed changes

pledbrook added 2 commits February 20, 2018 10:05

Intermediate commit to update source tree

e48ef69

Merge branch 'master' into working-with-files-docs

d534e86

eriwen suggested changes Feb 21, 2018

View reviewed changes

eriwen reviewed Feb 21, 2018

View reviewed changes


		You copy a file by creating an instance of Gradle's builtin api:org.gradle.api.tasks.Copy[] task and configuring it with the location of the file and where you want to put it. This example mimics copying a generated report into a directory that will be packed into an archive (zip, tarball, or whatever):

		----


		We look at these parts in more detail later in the chapter, but we'll start with how you accomplish some of the most common tasks users need.

		=== Copying a single file


		One thing to note is how the `include()` directive applies only to the `from()`, whereas the directive in the previous section applied to the whole task. These different levels of granularity in the copy specification allow you to easily handle most requirements that you will come across. TODO: Link to more details on copy specs.

		=== Creating archives (zip, tar, etc.)


		This is not a common requirement and should be used sparingly as you lose information and can easily break a build. It's generally preferable to copy directories and files instead.

		=== Renaming files on copy


		As you can see, the api:org.gradle.api.file.CopySpec#with(org.gradle.api.file.CopySpec...)[] method is the key to incorporating an existing copy specification into another one.

		===== Using child specifications

Working with files docs #4293

Working with files docs #4293

Uh oh!

Conversation

pledbrook commented Feb 7, 2018

Uh oh!

bmuschko left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pledbrook Feb 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmuschko Feb 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pledbrook Feb 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pledbrook Feb 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pledbrook Feb 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmuschko Feb 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

pledbrook Feb 16, 2018 •

edited

Loading

bmuschko Feb 16, 2018 •

edited

Loading

pledbrook Feb 16, 2018 •

edited

Loading

pledbrook Feb 16, 2018 •

edited

Loading

pledbrook Feb 16, 2018 •

edited

Loading

bmuschko Feb 16, 2018 •

edited

Loading

pledbrook Feb 21, 2018 •

edited

Loading

pledbrook Feb 23, 2018 •

edited

Loading