Skip to content

Conversation

@rk3rn3r
Copy link
Contributor

@rk3rn3r rk3rn3r commented Feb 21, 2013

  • added exclude function to composer / composer.json
  • added exclude documentation to composer-schema.json
  • added unit tests for the feature
  • updated to latest master version

example:

"autoload":
{
    "classmap" : ["src", "vendor"],
    "exclude-from-classmap": ["/test/", "/tests/", "/Tests/", "/test-suite/"]
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say from classmap generation

@stof
Copy link
Contributor

stof commented Feb 21, 2013

Please follow the PSR-2 coding standards which are used by Composer.

and your new feature should be tested.

Btw, you are breaking tests here as by default, you are excluding everything

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding .* before and after the excluding pattern is useless as your regex is not anchored anyway

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the packages/libraries with exclude in their composer.json I generate a blacklist pattern like
$LIBRARY_PATH$.%s. where %s is one exclude pattern. so I think it would be correct this way

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, at least the right is useless

@stof
Copy link
Contributor

stof commented Feb 21, 2013

you still need to add tests for the blacklist feature

@andygrunwald
Copy link

Hey @stof, hey @hosiplan,

@JohnDoe007 was so friendly to fix the CS and unit test issues. Thanks @JohnDoe007! (just fyi, he`s a team mate of me and we worked together on this feature).

Why we need this feature?
We use Composer to handle out dependencies with it.
We generate our autoloading map with -o (optimizing mode).
During this mode all available classes will be mapped into one big array for faster autoloading lookup,
In this array, some classes for unit tests are included.
In a big environments this could be a possible problem or a way to further optimizations.
This feature is a kind of optimization. It tries to exclude all UnitTest-Files with the introduction of an "exclude"-Folder.
@JohnDoe007 provided a small example how to use this feature in his own composer.json.
The named patterns will be executed during the autoload map creation.

Here you can find a minimal example how to reproduce this: https://gist.github.com/andygrunwald/4994295

I got a small chat with @Seldaek about this "feature" over twitter: https://twitter.com/andygrunwald/status/304163183541772288

If you got further notices / changes about this feature, let us know.
What do you think about this?

@fprochazka
Copy link
Contributor

I think this is counterproductive. There should be no tests in autoloader. The question is - how did they get there in the first place? I think excluding them, by not including them, is better approach.

But hey, if it won't broke any existing library... it's ok I guess.

@andygrunwald
Copy link

@hosiplan As you can see in the discussion from twitter with me and @Seldaek there is a regular expression to exclude all files which ends with Test.php (see https://github.com/composer/composer/blob/master/src/Composer/Autoload/AutoloadGenerator.php#L126)

This regular expression is not very "aggressiv" and in a Tests/-Folder there ar emuch more files as *Test.php
See example https://gist.github.com/andygrunwald/4994295 to reproduce :)

@stof
Copy link
Contributor

stof commented Feb 21, 2013

@hosiplan the test is not in the autoloader. In it in the autoloader generator (which is generating the autoloader configuration when you run composer install or composer dump-autoload)

@Seldaek
Copy link
Member

Seldaek commented Feb 21, 2013

@hosiplan is correct that you should not configure any autoload for the test files in your package definition, however in some cases like symfony bundles and others, tests are within the psr-0 mapped folder. In this case the ability to exclude them from generated classmaps would be a good thing.

@rk3rn3r
Copy link
Contributor Author

rk3rn3r commented Feb 22, 2013

Hello!
I guess I fixed all your issues regarding optimization of the blacklist pattern and the coding styleguide (psr-2) breaks and added a test for the exclude function.
In our case (symfony 2 app with in most cases sf2 related libraries added) we can save more then the half size of the classmap file (2500 entries instead of about 5300 by excluding tests-folders, sample-folders, etc).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to PSR-2, the curly brace should be on the same line than the if

@rk3rn3r
Copy link
Contributor Author

rk3rn3r commented Feb 25, 2013

@Parad0X probably this could be interesting for you

@rk3rn3r
Copy link
Contributor Author

rk3rn3r commented Feb 26, 2013

hmmm... any idea why there is no new travis build?

@patrickallaert
Copy link

Isn't it better to have both a whitelist and blacklist approach? Although the blacklist would at least solve issues we have with eZ Publish 5.

@rk3rn3r
Copy link
Contributor Author

rk3rn3r commented Mar 4, 2013

@patrickallaert if you say the blacklist solves your problem then it can't be a bad solution. I think using blacklist and whitelist together could be really confusing?
and which is the use case where a blacklist and whitelist is necessary?

@patrickallaert
Copy link

Don't get me wrong @JohnDoe007, I +1 this PR. Just that I see a use case for whitelisting: if you just want to cherry-pick some files or directories, it is just easier to select those rather than excluding everything else.

@Seldaek
Copy link
Member

Seldaek commented Mar 5, 2013

@patrickallaert if you just want a few files/directories, you can do it with "classmap":" ["file.php", "dir/"] already. And if you need to whitelist some things inside a blacklisted dir, I'd suggest you seriously need to rework your project structure :)

@patrickallaert
Copy link

@JohnDoe007 @Seldaek The need is indeed just to include some files/directories, so this is already implement. I was not aware of that since I'm not the guy touching that part of eZ Publish.
For sure, whitelisting inside blacklisting is just evil.

So +1.

@hason
Copy link
Contributor

hason commented Mar 8, 2013

I prefer integration of Finder component from symfony (#1406):

"classmap": ["src/*Bundle/", "src(/[^/]+(?<!test|Tests|foo/bar))*"]

and/or add support for complex configuration (http://symfony.com/doc/current/components/finder.html):

"classmap": ["src", {"in": ["vendor"], "depth": "2", "not-path": "..."}, {"in": ["lib"], "name": "*.cls"}]

@hason
Copy link
Contributor

hason commented Mar 8, 2013

Please rename it to a more descriptive name, eg. exclude-from-classmap. Current naming leads to confusion:

"autoload": {
    "psr-0": {"Acme": "src"},
    "exclude": ["Tests"]
}
$autoloader->findFile('Acme\Tests\FooTest'); // returns /.../src/Acme/Tests/FooTest.php

@Seldaek
Copy link
Member

Seldaek commented Mar 8, 2013

@hason I agree about the exclude name, it is not the clearest. Maybe classmap-exclude? Otherwise exclude-from-classmap is fairly clear too.

IMO it's easier to do this blacklist of explicit pathes than to integrate finder stuff because not everyone is familiar with it and not everyone can write lookahead regexes without shooting themselves in the foot :)

@Seldaek
Copy link
Member

Seldaek commented Mar 1, 2014

@rk3rn3r reviewed again and I wrote down one issue, apart from that it should also be rebased again if possible. But I think then it's good to merge. Sorry for the delay, many things to review.

@spezifanta
Copy link

Come on @rk3rn3r , go go go.

@klapifoch
Copy link

What happened with this feature?

@andygrunwald
Copy link

@klapifoch nothing yet. We (@trivago / @rk3rn3r / me) are now responsible.
We will try to take care of this feature, rebase it against the current master and push the new version.

Just give us a few more days to solve this. Ok?
Thanks and you will get new information soon :)

msiebeneicher added 4 commits February 11, 2015 17:54
…go/composer into add_exclude

* 'add_exclude' of https://github.com/trivago/composer:

# By Jordi Boggiano (239) and others
# Via Jordi Boggiano (184) and others
* 'master' of https://github.com/trivago/composer: (638 commits)
  Simplified syntax
  github deprecation changes
  fix bug in GitDriver::supports for remote repo
  strict check, testcase(s)
  Fix regex matching and add more tests for addSubNode, refs composer#3721, fixes composer#3716
  solve edge case for `composer remove vendor/pkg`
  chmod 644 src/Composer/Command/RemoveCommand.php
  Avoid failing on composer show of lazy providers
  Show more info when a download fails
  Add notion of autoloader skipping autoload-dev rules
  Satis grammar fix.
  remove unused statements
  removed needless output param
  + limit git ls-remote to heads + escape repo url
  add check for remote Repository in GitDriver::supports
  suppress the prefix
  Improve notice about /usr/local/bin
  Reuse current file permissions
  Add the P character to the regex pattern
  Added deprecated warning for the dev option
  ...

Conflicts:
	src/Composer/Autoload/AutoloadGenerator.php
	src/Composer/Autoload/ClassMapGenerator.php
…go/composer into add_exclude

* Resolve conflicts and update unit test
@andygrunwald
Copy link

Hey @stof, @Seldaek,
can you review it again? @msiebeneicher has updated this PR against the current master.
It would be cool if we can (finally) merge this feature. If you need anything else, please let us know.

@msiebeneicher
Copy link

The state was tested against the unit tests and the test case https://gist.github.com/andygrunwald/4994295 by @andygrunwald with the following composer-json:

{
  "require": {
    "doctrine/doctrine-bundle": "v1.0.0"
  },
  "autoload": {
    "exclude-from-classmap": ["/Tests/"]
  }
}

Result:

  • 790 classes in classmap with current composer version
  • 738 classes in classmap after this change

@klapifoch
Copy link

Not merged yet?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can see the removal of this block still has not been accounted for..

The problem is such, if you use psr-0, right now the dir is usually src/ and the namespace prefix Foo\Bar. So it will build up a classmap for all of src/ but exclude files not matching the whitelist (i.e. anything not in src/Foo/Bar).

Removing the whitelist is ok but then I think the $dir should be modified to have the namespace prefix appended if $psrType === 'psr-0' && strpos($namespace, '_') === false is true, that way we only scan the relevant files.

@Seldaek
Copy link
Member

Seldaek commented Feb 14, 2015

@andygrunwald @rk3rn3r Another thing that's missing for an easy merge is including it in the doc/04-schema.md in the autoload docs.

Unrelated: @klapifoch can you be more of an obnoxious and ungrateful person please?

@msiebeneicher
Copy link

@Seldaek If have updated the doc/04-schema.md and integrate a desciption.

I checked the whitelist handling again and i'm not sure if the logic make also sense for the blacklist handling:

With the "exclude-from-blacklist" you are free to define pattern to avoid classes in the classmap. If we extend the "user pattern" with the directories (and namespaces for psr-0), like the whitelist functionality, it's possbile to produce wrong or invalid patterns (if somebody use "^ - Start of string or line" for example).

It's looks like some of the unit tests failed because of temporary connections problems to "pear.phpmd.org"!?

@klapifoch
Copy link

@Seldaek It wasn't my intention to be annoying, it is just a feature i want to use in a project.
Sorry.

…d_exclude

# By Jordi Boggiano (30) and others
# Via Jordi Boggiano (37) and Morgan Campbell (1)
* 'master' of https://github.com/composer/composer: (83 commits)
  Update 01-basic-usage.md
  Revert 331425b as well, fixes composer#3612
  Revert "Disable overwrites when no-ansi is present, fixes composer#3612"
  Update deps
  Use justinrainbow/json-schema 1.4
  Improved wording
  Fix docs basic-auth => http-basic
  Add test for Generics class
  Single variable for traits and enums
  Use HHVM_VERSION instead of HPHP_VERSION
  Add support for using classmap to autoload Hack enums
  Re-use existing autoloader suffix if available, fixes composer#3701
  Report Travis CI build success early
  Test on HHVM nightly releases. Allow to fail.
  Make parseJson safer
  Use get home from Config instead of factory
  Fix env override regression, fixes composer#3820
  [create-project] Used no progress value for dependencies
  Add docBlock and fix CS
  Fix output of first line of progress when output is not decorated, refs composer#3818
  ...
@msiebeneicher
Copy link

@Seldaek: We reintegrated the current master again to our fork.

@andygrunwald
Copy link

Any update here?

@Seldaek Seldaek merged commit 7522a33 into composer:master Oct 30, 2015
Seldaek added a commit that referenced this pull request Oct 30, 2015
…ake sure they are all package-relative and do not leak to other packages, refs #1607
@Seldaek
Copy link
Member

Seldaek commented Oct 30, 2015

I guess 2.5y is a pretty long time, sorry about that, but we got there :) Finally took the time to merge this and try it out, tweaked things a bit because it was matching stuff a bit too greedily for my taste. Now you have to specify exact paths from package root to things you want to exclude, I see that it's not always the easiest think it's better that way rather than accidentally matching too much. We can always add support for wildcards or such later on if really needed. Thanks @rk3rn3r

@Seldaek
Copy link
Member

Seldaek commented Oct 30, 2015

So I added support for wildcards after all, see 6c16510

@Tobion
Copy link
Contributor

Tobion commented Oct 31, 2015

So when exluding test files from the autoload optimization, it means test classes might not be available anymore. So it is similar to using exclusion via gitattributes. But IMO using gitattributes is the better approach as this will actually also make the package considerably smaller. And developers still have a choice with --prefer-dist vs --prefer-source.
See symfony/symfony#16397 (comment)

@Seldaek
Copy link
Member

Seldaek commented Oct 31, 2015 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.