feat(reportImport): add ReportImport agent #882

maxhbr · 2017-07-26T14:30:24Z

NOTE: this is work in progress and this PR should be here as place to aggregate discussion related to the feature

corresponding issue: #669
related PR: maxhbr#2

maxhbr · 2017-08-11T14:34:46Z

Currently the UI look like:

maxhbr · 2017-08-17T19:31:27Z

NOTE: This PR currently also includes the changes from #889.

shaheemazmalmmd · 2017-08-22T04:27:21Z

Can you please add this new agent to debian packages ??

maxhbr · 2017-08-22T08:36:57Z

@shaheemazmalmmd: I have made the corresponding changes (they are untested)

shaheemazmalmmd · 2017-08-28T09:12:00Z

@maxhbr : There is a error while building a package.
dpkg-gencontrol: error: illegal package name 'fossology-reportImport': character 'I' not allowed dh_gencontrol: dpkg-gencontrol -pfossology-reportImport -ldebian/changelog -Tdebian/fossology-reportImport.substvars -Pdebian/fossology-reportImport returned exit code 255"

Can you please have a look into this

maxhbr · 2017-08-29T09:05:27Z

@shaheemazmalmmd the "illegal char"-problem should be fixed

maxhbr · 2017-09-04T08:00:12Z

@shaheemazmalmmd The code here has currently a bug which prevents the frontend arguments from beeing passed to the backend.

maxhbr · 2017-09-04T11:15:59Z

@shaheemazmalmmd @mcjaeger this branch is fixed again and should work

related to fossology#906 and fossology#882

mcjaeger · 2017-10-05T07:38:10Z

Hi, I think the point is true (given the links is displayed) let's skip the highlight, as this is not supported by the SPDX import format. regarding the radio buttono selection, it should be selected if there is a import and this import was set to already set decisions, this is the point?

regarding the error message issue with the missing copyright_ars table: So is this a serious issue in the way that the user does not see something in the UI or is it that the application throws an error message?

shaheemazmalmmd · 2017-10-03T13:04:29Z

src/www/ui/page/AdviceLicense.php

    if ($oldRow['rf_pk'] == -1)
    {
-      $oldRow['rf_pk'] = $licenseDao->insertUploadLicense($shortname, $rfText);
+      $oldRow['rf_pk'] = $licenseDao->insertUploadLicense($shortname, $rfText, $_SESSION['GroupId']);


You can use Auth::getGroupId for group id. Also you can use already initialized GroupId from Handle function as perm for saveInput.

shaheemazmalmmd · 2017-10-03T13:13:28Z

src/reportImport/ui/ReportImportPlugin.php

+      self::PERMISSION => Auth::PERM_WRITE,
+      self::REQUIRES_LOGIN => TRUE
+    ));
+    $this->userDao = $GLOBALS['container']->get('dao.user');


userDao is not used anywhere. you can remove this

shaheemazmalmmd · 2017-10-03T13:17:12Z

src/reportImport/ui/ReportImportAgentPlugin.php

@@ -0,0 +1,91 @@
+<?php
+/***********************************************************
+ * Copyright (C) 2015, Siemens AG


Please change the Year

shaheemazmalmmd · 2017-10-03T13:24:20Z

src/reportImport/ui/Makefile

@@ -0,0 +1,29 @@
+# Copyright Siemens AG 2014


Please change the Year

This file was not changed enough to be worth updating the copyright year. It is just copied from one of the other agents.

shaheemazmalmmd · 2017-10-03T13:25:32Z

src/reportImport/agent/services.php

@@ -0,0 +1,10 @@
+<?php
+/*
+ Copyright Siemens AG 2015


Please change the Year

This file is just a copy and thus should keep its header without changes

shaheemazmalmmd · 2017-10-03T13:31:10Z

src/reportImport/agent/XmlImportSource.php

+   * @param $fileid
+   * @return array
+   */
+  public function getHashesMap($fileid)


To return a empty array why is it required to pass $fileid ?

That is specified in the interface ImportSource

shaheemazmalmmd · 2017-10-03T13:37:28Z

src/reportImport/agent/ReportImportConfiguration.php

+    echo "\nDEBUG: \$addCopyrightInformation is: "             .$this->var_dump($this->addCopyrightInformation);
+    echo "\nDEBUG: \$concludeLicenseDecisionType is: "         .$this->var_dump($this->concludeLicenseDecisionType);
+    echo "\n";
+  }


Comment or remove the DEBUG code. which always gets printed in agent log.

shaheemazmalmmd · 2017-10-03T13:42:07Z

src/reportImport/agent/ReportImportSink.php

+      }
+      foreach ($removeLicenseIds as $licenseId)
+      {
+        echo "remove decision $licenseId from " . $pfile['uploadtree_pk'] . "\n";


Please remove if its used to debug.

shaheemazmalmmd · 2017-10-12T06:56:03Z

@maxhbr : Seems like there is some issue with existing database on this branch.

To reproduce

-> install master branch upload 2 components to fossology and do some clearing.
-> Now install report import branch upload a component. (In my case i have uploaded zlib).

-> Click on upload and import.
-> It schedules report import for previously uploaded component. Not the one which i have selected.

shaheemazmalmmd · 2017-10-12T10:41:13Z

src/reportImport/ui/ReportImportPlugin.php

+    else
+    {
+
+      $jobMetaData = $this->runImport($uploadId, $_FILES['report'], $request);


Here the $uploadId will be always the value of first upload deom the hidden field. Adding this line $uploadId = intval(GetArrayVal("uploadselect", $_POST)); in else case will give the value of selected upload.

- implement agent and ui - adopt license-explorer and others - allow to download the report which was uploaded by the user - add new dependency (EasyRdf) to composer

* they get saved in the DB and also exported * they are __not__ yet displayed by the copyright ui

maxhbr · 2017-10-12T17:44:57Z

@shaheemazmalmmd The mentioned bug is fixed and the value of "uploadselect" is used. But there is still the bug that one is redirected to the job-view of the wrong upload (where no job was scheduled).

mcjaeger · 2017-10-13T09:00:26Z

just for the files: I did a retest this morning using the vagrant deployment with the latest commit, everything looks fine.

maxhbr added the WIP label Jul 26, 2017

maxhbr force-pushed the dev/addSPDX2ImportAgent_dist branch 2 times, most recently from fcf77d7 to 5fe3c4d Compare July 27, 2017 08:57

maxhbr self-assigned this Jul 27, 2017

maxhbr changed the title ~~WIP: add SPDX2-Import agent~~ WIP: add ReportImport agent Aug 7, 2017

maxhbr changed the title ~~WIP: add ReportImport agent~~ WIP: feat(reportImport): add ReportImport agent Aug 7, 2017

maxhbr mentioned this pull request Aug 7, 2017

feat(spdx2): strip invalid characters from non-spdx-compatible licenses #884

Merged

maxhbr force-pushed the dev/addSPDX2ImportAgent_dist branch 3 times, most recently from bff47f4 to a80526f Compare August 10, 2017 16:56

maxhbr requested review from max-wittig, shaheemazmalmmd and steffen-weber August 11, 2017 08:11

maxhbr force-pushed the dev/addSPDX2ImportAgent_dist branch from a80526f to 1ad23bd Compare August 11, 2017 14:22

maxhbr force-pushed the dev/addSPDX2ImportAgent_dist branch 3 times, most recently from 1cdde79 to d7b50ca Compare August 17, 2017 19:30

maxhbr force-pushed the dev/addSPDX2ImportAgent_dist branch from d93c29f to f134af2 Compare August 29, 2017 09:00

mcjaeger self-assigned this Sep 21, 2017

mcjaeger removed the WIP label Sep 21, 2017

maxhbr force-pushed the dev/addSPDX2ImportAgent_dist branch from fa31891 to da592c0 Compare September 21, 2017 14:05

maxhbr added a commit to maxhbr/fossology that referenced this pull request Sep 25, 2017

feat(copyright): allow to have multiple copyright decisions

fda2ddb

related to fossology#906 and fossology#882

shaheemazmalmmd requested changes Oct 6, 2017

View reviewed changes

shaheemazmalmmd reviewed Oct 12, 2017

View reviewed changes

maxhbr and others added 21 commits October 12, 2017 18:29

feat(spdx2Import): inital commit

b7bd5b6

- implement agent and ui - adopt license-explorer and others - allow to download the report which was uploaded by the user - add new dependency (EasyRdf) to composer

feat(spdx2Import): also import copyright statements

26bff73

* they get saved in the DB and also exported * they are __not__ yet displayed by the copyright ui

feat(spdx2Import): conclusions

9e68781

feat(spdx2Import): compare only by sha1

26c4187

feat(spdx2Import): start to make conclusions optional

bd20cc7

feat(spdx2Import): menu entry at "Upload::..."

3d469b2

feat(spdx2Import): refactoring and splitup of files

0bc8788

feat(spdx2Import): splitup to support other formats

fa56a96

refactor(reportImport): spdx2Import -> reportImport

9cdf1d6

feat(reportImport): handle all arguments from UI

3f95181

feat(reportImport): add option to create real licenses

0d46873

feat(spdx2): also export ninka and import data

0fdba11

feat(reportImport): parse also xml files

74f6241

feat(reportImport): minor improvements

10a2a82

feat(reportImport): minor changes to satisfy older PHP versions

09b90a2

feat(reportImport): add corresponding debain definitions

5fdb4ce

fix(reportImport): fix bug in reportImport, refactor file matching

f5e65fb

chore(gitignore): add more entries to the blacklist

e268d89

feat(reportImport): add imported coyprights as decisions

025c4fe

feat(reportImport): handle orLaterOperator correctly

cb24345

feat(reportImport): some cleanup and minor improvements

05a3061

maxhbr force-pushed the dev/addSPDX2ImportAgent_dist branch from 291a2b3 to 05a3061 Compare October 12, 2017 17:33

mcjaeger merged commit 9e72e16 into fossology:master Oct 13, 2017

GMishx mentioned this pull request Aug 28, 2018

Is there a way to Migrate a Single Project/Upload from one instance/Server to another ? #1185

Closed

feat(reportImport): add ReportImport agent #882

feat(reportImport): add ReportImport agent #882

Uh oh!

Conversation

maxhbr commented Jul 26, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

maxhbr commented Aug 11, 2017

Uh oh!

maxhbr commented Aug 17, 2017

Uh oh!

shaheemazmalmmd commented Aug 22, 2017

Uh oh!

maxhbr commented Aug 22, 2017

Uh oh!

shaheemazmalmmd commented Aug 28, 2017

Uh oh!

maxhbr commented Aug 29, 2017

Uh oh!

maxhbr commented Sep 4, 2017

Uh oh!

maxhbr commented Sep 4, 2017

Uh oh!

mcjaeger commented Oct 5, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shaheemazmalmmd commented Oct 12, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxhbr commented Oct 12, 2017

Uh oh!

mcjaeger commented Oct 13, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

maxhbr commented Jul 26, 2017 •

edited

Loading

mcjaeger commented Oct 5, 2017 •

edited

Loading