-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Creating Cache Artifact #890
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
d4d70e3
to
2403554
Compare
<Target Name="CacheWorkerMapper"> | ||
|
||
<ItemGroup> | ||
<PackageToRestore Include ="@(PackageReference)" Condition="'%(PackageReference.IsImplicitlyDefined)' != 'true'"/> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will remove this and replace it with DisableImplicitFrameworkReferences=true in the invocation of CacheWorkerMapper
using NuGet.Frameworks; | ||
using NuGet.Packaging.Core; | ||
using NuGet.ProjectModel; | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will add a new line
</PackageReference> | ||
</ItemGroup> | ||
<ItemGroup> | ||
<Compile Include="$(RepositoryRootDirectory)src\Tasks\Microsoft.NET.Build.Tasks\PackageInfoHelpers.cs" Exclude="$(GlobalExclude)" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dsplaisted is it fine to include product code in test ? Or is there any other good way to consume this logic in tests
also fixes #830 |
610ee2d
to
f46dfc0
Compare
namespace Microsoft.NET.Build.Tasks | ||
{ | ||
/// <summary> | ||
/// Resolves the assemblies to be published for a .NET app. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
".NET Core App"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Resolves the list of packages..."
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will change
} | ||
} | ||
|
||
public class CacheArtifactParser |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please add some comments on the role of the type?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will do
if (pkgname != null && version != null) | ||
{ | ||
listofPackages.Add(new PackageInfo(pkgname.Value, version.Value)); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For diagnostics, can we emit details of the package reference being worked upon?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i am emitting them in its caller. as this code is shared with tests
https://github.com/dotnet/sdk/pull/890/files#diff-12149d9a659a09923162cd8cd2ac24d0R62
{ | ||
if (filterLookup.ContainsKey(pkg.Name)) | ||
{ | ||
filterLookup[pkg.Name].Add(pkg); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the key case-sensitive?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are we trying to add a new value for an existing key? Per https://msdn.microsoft.com/en-us/library/k7z0zy8k(v=vs.110).aspx, Add would throw an exception.
Instead, should the key be made unique so that such a conflict does not happen when there are multiple entries for different versions of a given package?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The key is case-sensitive , as you can see from definition of filterLookup, it is a dictionary of hashset. This is necessary as we are indexing packages with just the package name for the purpose of efficient filtering, and out list could have same packages with different versions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens when the entry XML file contains two entries with different versions but same cased name of the package?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
HashSet does not throw, it just ignores duplicates
private static string testarch; | ||
private static string tfm = "netcoreapp1.0"; | ||
|
||
static GivenThatWeWantToCacheAProjectWithDependencies() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we have a test that validates that the filtered packages are coming from the cache once the app targeting the cache is activated?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We only check for the layout in sdk repo.
528aee5
to
8d4abd4
Compare
|
||
protected override void ExecuteCore() | ||
{ | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: extra whitespace
|
||
foreach (var pkginfo in doc.Elements("Project").Elements("ItemGroup").Elements("PackageReference")) | ||
{ | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: extra whitepsace
using NuGet.Frameworks; | ||
using NuGet.ProjectModel; | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: extra whitespace
{ | ||
_hasallResolvedLibrariesBeenRecorded = true; | ||
return _allResolvedPackages.Keys; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: missing whitepsace
_doNotTrackPackageAsResolved = doNotTrackPackageAsResolved; | ||
return this; | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: extra whitespace
DependsOnTargets="RestoreForComposeCache; | ||
ComputeAndCopyFilesToCacheDirectory;" | ||
Condition="!Exists($(CacheWorkerWorkingDir))" /> | ||
<!-- |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: more whitespace issues. Avoid more than one blank line. And here the start of the comment for the next target looks like it's nested in the previous target as there is no blank line between them and the indentation is off.
<Target Name="PrepforRestoreForComposeCache"> | ||
|
||
<PropertyGroup> | ||
<_RPackageVersion>$(_RPackageVersion.Replace('*','-'))</_RPackageVersion> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What is RPackage? Again, Avoid abbreviations.
|
||
<MSBuild Projects="$(MSBuildProjectFullPath)" | ||
Targets="RunCrossGen" | ||
Properties="_CGenExe=$(_Crossgen); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's weird that we're passing properties to msbuild invocation that are underscore prefix (signals 'private' by convention only). This seems like a public contract with the RunCrossGenTarget and so its inputs should have good names and not be underscore prefixed.
Also, again avoid obscure abbreviations. There is no point saving 4 characters to say CGen instead of CrossGen.
|
||
|
||
cacheDirectory.Should().HaveFiles(files_on_disk); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: more extra whitespace
|
||
public ComposeCache(MSBuildTest msbuild, string projectPath) | ||
: base(msbuild, projectPath) | ||
public ComposeCache(MSBuildTest msbuild, string projectPath, string relativePathToProject= null) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: missing space before =
8d4abd4
to
b2149b6
Compare
@nguerrera i think i have fixed most of your concerns |
Waiting to understand if there's not a way to get rid of the manual split. "Outdated" thread: #890 (comment) |
Why are we not putting the full RID in the output directory? This would better align with "build" and "publish". Using Docker, I can easily spin up multiple types of machines (OSX, Ubuntu, etc) and use the same repo. I think we need to put the full RID in the output path. Refers to: test/Microsoft.NET.TestFramework/Commands/ComposeCache.cs:51 in b2149b6. [](commit_id = b2149b6, deletion_comment = False) |
</PackageReference> | ||
</ItemGroup> | ||
<ItemGroup> | ||
<Compile Include="$(RepositoryRootDirectory)src\Tasks\Microsoft.NET.Build.Tasks\PackageInfo.cs" Exclude="$(GlobalExclude)" /> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should be doing this.
We have 2 types of tests: unit and integration.
Unit tests can/should test each class in isolation and mock out things. Put those tests in the UnitTests project under https://github.com/dotnet/sdk/tree/master/src/Tasks/Microsoft.NET.Build.Tasks.UnitTests.
Integration tests should have 0 knowledge of the inner workings of the code. This test project is an integration test. It should only be doing things that users would see/interact with.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The logic is used to verify the output of the integration test. I can duplicate the logic, but @dsplaisted was ok with using this to build test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IMO - you should be baselining the file. It is easier to view the differences/changes when tests start to fail. You can see exactly what changed.
public class GivenThatWeWantToCacheAProjectWithDependencies : SdkTest | ||
{ | ||
private static string libPrefix; | ||
private static string runtimeos; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All these field names should conform to our coding standards. See https://github.com/dotnet/corefx/blob/master/Documentation/coding-guidelines/coding-style.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
|
||
public bool Equals(PackageInfo pkg) | ||
{ | ||
return Name.Equals(pkg.Name) && Version.Equals(pkg.Version); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you want case-insensitive comparison here for Name. NuGet package ids are case-insensitive.
|
||
namespace Microsoft.NET.Build.Tasks | ||
{ | ||
internal class PackageInfo : IEquatable<PackageInfo> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should use https://github.com/NuGet/NuGet.Client/blob/4cccb13833ad29d6a0bcff055460d964f1b49cfe/src/NuGet.Core/NuGet.Packaging.Core.Types/PackageIdentity.cs#L13 instead of making our own class. PackageIdentiy from NuGet already has all the correct semantics built into it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done
{ | ||
public class PublishAssembliesResolver | ||
{ | ||
private static ConcurrentDictionary<PackageInfo, byte> _allResolvedPackages = new ConcurrentDictionary<PackageInfo, byte>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bad idea. We should not be using a static anything here for communication between Tasks.
using NuGet.Versioning; | ||
using NuGet.Packaging.Core; | ||
|
||
#if PRODUCT |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I really don't think we should be reusing this code file in our tests. It is making the product code messier than it has to be.
The tests can just baseline the xml file and compare the result against the baseline. Or even use very simple Xml logic to ensure the file is correct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This logic would need to be duplicated in test sources, there is no other way out.
I believe this is the lesser of the evils
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I disagree. This is putting extra constraints on our product code that don't need to be there. For example these #ifs are unnecessary. If we want to add more logic to this file, we will have to continue adding these #ifs.
I still think baselining a couple xml files is the correct way for the tests to verify the output xml file is correct. It makes it super easy to see what has changed in the file when it has changed. And it catches things like when unexpected elements start showing up in the file.
In reply to: 103600599 [](ancestors = 103600599)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is not as trivial as that, the contents of the file are going to differ across platforms and we will end up with multiple such files and make the test fragile. Logic like
sdk/test/Microsoft.NET.Publish.Tests/GivenThatWeWantToCacheAProjectWithDependencies.cs
Line 156 in b3ee71d
if (!RuntimeInformation.IsOSPlatform(OSPlatform.Windows) && arch != "x86") |
If it becomes unbearable we should just fork the logic at that point
|
||
namespace Microsoft.NET.Build.Tasks | ||
{ | ||
//<!-- The following provides the logic to parse the output - artifact.xml --> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you misunderstood my previous ask.
In C#, class summary comments should look like the following:
/// <summary>
/// Represents a single diagnostic message, such as a compilation error or a project.json parsing error.
/// </summary>
internal class DiagnosticMessage
return results; | ||
} | ||
|
||
public IEnumerable<PackageIdentity> GetResolvedpackages() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(nit) capitalization.
|
||
|
||
<ItemGroup> | ||
<ResolvedPackagesPublished Include="@(PackagesThatWhereResolved)" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(nit) Where
should be Were
DependsOnTargets="CacheWorkerMain; | ||
_CopyResolvedOptimizedFiles"> | ||
|
||
<RemoveDuplicates |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From MSDN: "This task is case insensitive and does not compare item metadata when determining duplicates."
So what happens if we have multiple packages with the same ID but different versions? Will they get de-duped? Thus only 1 wins?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
great catch, will remove this logic for now. As i am handling duplicates when consuming them
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
spoke too soon, have fixed this with RemoveDuplicatePackageReferences
This resource doesn't exist in the .resx file. When I try to run
I am getting the following error:
Refers to: src/Tasks/Microsoft.NET.Build.Tasks/build/Microsoft.NET.ComposeCache.targets:203 in 67adb10. [](commit_id = 67adb10, deletion_comment = False) |
in response to #890 (comment) @nguerrera removed it in 75744ed |
@nguerrera and @eerhardt i believe i have addressed all your concerns |
namespace Microsoft.NET.Build.Tasks | ||
{ | ||
/// <summary> | ||
/// Resolves the assemblies to be published for a .NET app. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment seems like it was a copy-paste leftover.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
argh.. should stop doing this, will fix
|
||
protected override void ExecuteCore() | ||
{ | ||
var listofPackages = new HashSet<PackageIdentity>(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
listofPackages
is misleading since it is a hash set. Simply packages
would work, I think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
changed it to set
This error message doesn't make sense for this error:
That error message is strictly for desktop framework .exes. Here we are dealing with .NET Core, and this has nothing to do with .exes. Refers to: src/Tasks/Microsoft.NET.Build.Tasks/build/Microsoft.NET.ComposeCache.targets:201 in fb8836f. [](commit_id = fb8836f, deletion_comment = False) |
<PropertyGroup> | ||
<PathSeparator>$([System.IO.Path]::PathSeparator)</PathSeparator> | ||
<DirectorySeparatorChar>$([System.IO.Path]::DirectorySeparatorChar)</DirectorySeparatorChar> | ||
<TEMP Condition="'$(TEMP)' == ''">$([System.IO.Path]::GetTempPath())</TEMP> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this property used? I don't see where it is being used.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is used in the compose phase, will move it there
<_CacheArtifactContent> | ||
<![CDATA[ | ||
<CacheArtifacts> | ||
<ItemGroup> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need <ItemGroup>
in this xml file. That is a hold over from using MSBuild.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i will keep this for now. as it will also lead to an extensible format, if we want to add anything more for cache.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It doesn't make any sense for it to be named ItemGroup
.
Also, XML is already extensible. You can add any other <Element>
you want without having things nested inside collection elements.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As a shepherd of the MSBuild file format: don't use ItemGroup
. It's a bad name even for MSBuild.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok will get rid of it, yes if we need to extend we can add some other element
Just a few last things to clean up, and then I'll sign off. |
In reply to #890 (comment) changed the message |
@nguerrera @eerhardt any other concerns ? |
@@ -0,0 +1,4 @@ | |||
<CacheArtifacts> | |||
<PackageReference Include="Newtonsoft.Json" Version="9.0.1" /> | |||
<PackageReference Include="Newtonsoft.Json" Version ="9.0.2-beta2"/> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like Include
here since that is an msbuild concept, I think we should make it even more obvious that this is its own format, separate from csproj package references.
This is my suggestion:
<CacheArtifacts>
<Package Id="NewtonSoft.Json" Version="9.0.1" />
</CacheArtifacts>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't want to start doing something different just because we can.
I would stick to conventions as much as possible
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We talked offline, the problem with using any msbuild conventions is that it suggests other msbuild idioms might be possible. e.g. Can I use Remove? Exclude? Put common versions in props? etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@nguerrera changed them
{ | ||
throw new BuildErrorException(Strings.IncorrectFilterFormat, filterFile); | ||
} | ||
#endif |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I strongly disklike the ifdef precedent, it leads to spaghetti every time. You can leave it for now, though, and I'll look at fixing it by enabling IVTA from tests (#656)
I have just the one remaining concern about the file format left. |
- This allows for multiple versions of same package to be specified Parallelizing restore and crossgen operations Publish filter consumes the artifact produced by cache Removing temp copies during cache
@nguerrera @eerhardt thanks for your inputs |
Update the branding to rc1
With this change
dotnet-cache can cache packages with different versions at the same time, it further produces an artifact.xml which has the list of all the packages cached during dotnet-cache.
This artifact.xml should be used with publish filter option to remove the files which are already cached
@gkhanna79 @eerhardt @nguerrera PTAL