[2/2] Circe-powered container scanning #1521

jssblck · 2025-03-10T21:41:59Z

Overview

Updates fossa container analyze to use circe reexport to normalize the container contents before scanning: fossas/circe#23

Acceptance criteria

This test case failed before: fossa container analyze nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0 -o
This test case works today: cabal run fossa -- container analyze nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0 -o

Testing plan

Manually validated the images specified here: https://github.com/fossas/circe/blob/main/integration/tests/it/reexport.rs#L12-L36

Before:

# Fails whether local or remote
fossa-cli on git circe-container-scan via λ 9.4.8 via rs v1.85.0
; fossa container analyze nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0
Discovered image for: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0 (of 171186315 bytes) via docker engine api.
Exporting docker image to temp file: /private/var/folders/rf/pv7g10r17zz_3f2rm6kmqdqc0000gn/T/fossa-docker-engine-tmp-cdcb4117f816d2d3/image.tar! This may take a while!
Analyzing exported docker archive: /private/var/folders/rf/pv7g10r17zz_3f2rm6kmqdqc0000gn/T/fossa-docker-engine-tmp-cdcb4117f816d2d3/image.tar
Searching for JARs in container image.
[ERROR] An issue occurred

  *** Relevant Errors ***

      Error: TarParserError: NotTarFormat :| [TarParserError: TruncatedArchive,TarParserError: NotTarFormat,TarParserError: NotTarFormat,TarParserError: TruncatedArchive,TarParserError: NotTarFormat,TarParserError: NotTarFormat,TarParserError: NotTarFormat,TarParserError: NotTarFormat,TarParserError: NotTarFormat,TarParserError: NotTarFormat,TarParserError: TruncatedArchive,TarParserError: NotTarFormat,TarParserError: NotTarFormat,TarParserError: TruncatedArchive]

fossa-cli on git circe-container-scan via λ 9.4.8 via rs v1.85.0
; fossa container analyze hellotest:latest
Discovered image for: hellotest:latest (of 3595690 bytes) via docker engine api.
Exporting docker image to temp file: /private/var/folders/rf/pv7g10r17zz_3f2rm6kmqdqc0000gn/T/fossa-docker-engine-tmp-bdcadc40643c36cb/image.tar! This may take a while!
Analyzing exported docker archive: /private/var/folders/rf/pv7g10r17zz_3f2rm6kmqdqc0000gn/T/fossa-docker-engine-tmp-bdcadc40643c36cb/image.tar
Searching for JARs in container image.
[ERROR] An issue occurred

  *** Relevant Errors ***

      Error: TarParserError: NotTarFormat :| [TarParserError: NotTarFormat,TarParserError: NotTarFormat]

fossa-cli on git circe-container-scan via λ 9.4.8 via rs v1.85.0
; fossa container analyze ~/projects/circe/scratch/changeset_example_docker.tar
Searching for JARs in container image.
[ERROR] An issue occurred

  *** Relevant Errors ***

      Error: TarParserError: NotTarFormat :| [TarParserError: TruncatedArchive,TarParserError: TruncatedArchive,TarParserError: TruncatedArchive,TarParserError: TruncatedArchive,TarParserError: TruncatedArchive,TarParserError: TruncatedArchive,TarParserError: TruncatedArchive]

After:

# In local container registry
fossa-cli on git circe-container-scan via λ 9.4.8 via rs v1.85.0
; cabal run fossa -- container analyze nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0
Discovered image for: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0 (of 171186315 bytes) via docker engine api.
Exporting normalized container image for: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0
Searching for JARs in container image.
Analyzing Base Layer
Analyzing sqlitedb project at var/lib/rpm/
Analyzing setuptools project at usr/lib/python3.9/site-packages/iniparse-0.4-py3.9.egg-info/
Analyzing setuptools project at usr/lib/python3.9/site-packages/python_dateutil-2.8.1-py3.9.egg-info/
Analyzing setuptools project at usr/lib/python3.9/site-packages/urllib3-1.26.5-py3.9.egg-info/
Analyzing setuptools project at usr/lib64/python3.9/site-packages/cloud_what/
Analyzing setuptools project at usr/lib64/python3.9/site-packages/subscription_manager-1.29.40-py3.9.egg-info/
Analyzing Other Layers
Analyzing sqlitedb project at var/lib/rpm/
Using project name: `nvcr.io/nvidia/cloud-native/gpu-operator-validator`
Using project revision: `sha256:70a0bd29259820d6257b04b0cdb6a175f9783d4dd19ccc4ec6599d407c359ba5`
Using branch: `No branch (detached HEAD)`
View FOSSA Report:
  https://app.fossa.com/projects/custom%2b24357%2fnvcr.io%2fnvidia%2fcloud-native%2fgpu-operator-validator/refs/branch/master/sha256:70a0bd29259820d6257b04b0cdb6a175f9783d4dd19ccc4ec6599d407c359ba5

# In remote repository, not local
fossa-cli on git circe-container-scan is pkg v0.4.0 via rs v1.85.0
; cabal run fossa -- container analyze nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0
Exporting normalized container image for: nvcr.io/nvidia/cloud-native/gpu-operator-validator:v24.9.0
Searching for JARs in container image.
Analyzing Base Layer
Analyzing sqlitedb project at var/lib/rpm/
Analyzing setuptools project at usr/lib64/python3.9/site-packages/subscription_manager-1.29.40-py3.9.egg-info/
Analyzing setuptools project at usr/lib64/python3.9/site-packages/cloud_what/
Analyzing setuptools project at usr/lib/python3.9/site-packages/urllib3-1.26.5-py3.9.egg-info/
Analyzing setuptools project at usr/lib/python3.9/site-packages/python_dateutil-2.8.1-py3.9.egg-info/
Analyzing setuptools project at usr/lib/python3.9/site-packages/iniparse-0.4-py3.9.egg-info/
Analyzing Other Layers
Analyzing sqlitedb project at var/lib/rpm/
Using project name: `nvcr.io/nvidia/cloud-native/gpu-operator-validator`
Using project revision: `sha256:78d5d811ce0779ed221ddd32eb205ddee8b9c100049c8ef7dacee927e88240a4`
Using branch: `No branch (detached HEAD)`
View FOSSA Report:
  https://app.fossa.com/projects/custom%2b24357%2fnvcr.io%2fnvidia%2fcloud-native%2fgpu-operator-validator/refs/branch/master/sha256:78d5d811ce0779ed221ddd32eb205ddee8b9c100049c8ef7dacee927e88240a4

fossa-cli on git circe-container-scan via λ 9.4.8 via rs v1.85.0
; cabal run fossa -- container analyze hellotest:latest
Discovered image for: hellotest:latest (of 3595690 bytes) via docker engine api.
Exporting normalized container image for: hellotest:latest
Searching for JARs in container image.
Analyzing Base Layer
Analyzing apkdb project at lib/apk/db/
Analyzing Other Layers
Using project name: `hellotest`
Using project revision: `sha256:3645fdb9d51137c0751c3d9a6112426b27bfcab9339d68c5b061818cc1d05377`
Using branch: `No branch (detached HEAD)`
View FOSSA Report:
  https://app.fossa.com/projects/custom%2b24357%2fhellotest/refs/branch/master/sha256:3645fdb9d51137c0751c3d9a6112426b27bfcab9339d68c5b061818cc1d05377

fossa-cli on git circe-container-scan via λ 9.4.8 via rs v1.85.0
; cabal run fossa -- container analyze ~/projects/circe/scratch/changeset_example_docker.tar
Exporting normalized container image for: /Users/jess/projects/circe/scratch/changeset_example_docker.tar
Searching for JARs in container image.
Analyzing Base Layer
Analyzing apkdb project at lib/apk/db/
Analyzing Other Layers
Using project name: `changeset_example_docker`
Using project revision: `sha256:1af7aa8d7fe18420f10b46a78c23c5c9cb01817d30a03a12c33e8a26555f7b4f`
Using branch: `No branch (detached HEAD)`
View FOSSA Report:
  https://app.fossa.com/projects/custom%2b24357%2fchangeset_example_docker/refs/branch/master/sha256:1af7aa8d7fe18420f10b46a78c23c5c9cb01817d30a03a12c33e8a26555f7b4f```

Risks

As of the current state of this PR this only affects remote images, not images in a local docker daemon or images which have been exported as tarballs.

The main effect of this risk is that users will still have issues scanning images in the local docker daemon.

Metrics

None

References

Finalizes https://fossa.atlassian.net/browse/ANE-2184

Checklist

I added tests for this PR's change (or explained in the PR description why tests don't make sense).
If this PR introduced a user-visible change, I added documentation into docs/.
If this PR added docs, I added links as appropriate to the user manual's ToC in docs/README.ms and gave consideration to how discoverable or not my documentation is.
If this change is externally visible, I updated Changelog.md. If this PR did not mark a release, I added my changes into an ## Unreleased section at the top.
If I made changes to .fossa.yml or fossa-deps.{json.yml}, I updated docs/references/files/*.schema.json AND I have updated example files used by fossa init command. You may also need to update these if you have added/removed new dependency type (e.g. pip) or analysis target type (e.g. poetry).
If I made changes to a subcommand's options, I updated docs/references/subcommands/<subcommand>.md.

jssblck · 2025-03-19T23:13:17Z

The integration test failed with a rate limit error due to all my rebuilds, I'm going ahead and requesting review but rest assured that if there are more errors than that I'll fix them before merging 😄

ETA: Since resolved

james-fossa · 2025-03-20T14:46:14Z

src/App/Fossa/Container/Sources/Registry.hs

+  Path Abs Dir ->
+  m (Maybe (Path Abs File))
+runWithCirceReexport img dir = do
+  let tarballPath = dir </> $(mkRelFile "image.tar")


Checking my understanding - $(mkRelFile "image.tar") gets expanded at compile time so that tarballPath points to a non-optional, non-errorful path to ./$dir/image.tar at runtime, is that correct?

I think it just converts from a FilePath to a Path Rel File. FilePath is an alias for String. Nothing fancy

FilePath: https://hackage.haskell.org/package/base-4.21.0.0/docs/Prelude.html#t:FilePath
mkRelFile: https://hackage.haskell.org/package/path-0.9.6/docs/OsPath-Posix.html#v:mkRelFile

So dir </> $(mkRelFile "image.tar") will just be <whatever dir is>/image.tar

james-fossa

Nicely done!

spatten

This looks great. 🚢

jssblck added 15 commits March 10, 2025 12:19

Skeleton

b7f7bc3

WIP

5fd5f32

Support using circe to normalize images

e472998

Fix reexport command

20029fd

Remove extra comment

3438e10

Implement circe 1.0.0

8b8d673

Undo rename of analyzeDockerArchive

a0e7e5d

Undo overzealous changes

4e79c37

Slightly simplify circeReexportCommand

ec94cce

Update changelog

99cfdfb

Update docs

9ecddf6

Fix how project names are calculated to match old behavior

c1c20ea

Update zip dependency for millhone

6783a06

Fix reporting project name for plain files

6d9c8dd

Fix reporting the project name for remote OCI repositories

a809b95

jssblck marked this pull request as ready for review March 19, 2025 23:13

jssblck requested a review from a team as a code owner March 19, 2025 23:13

jssblck requested review from james-fossa and spatten and removed request for a team March 19, 2025 23:13

james-fossa reviewed Mar 20, 2025

View reviewed changes

james-fossa approved these changes Mar 20, 2025

View reviewed changes

spatten approved these changes Mar 20, 2025

View reviewed changes

jssblck merged commit 7e5e5c4 into circe Mar 20, 2025
19 checks passed

jssblck deleted the circe-container-scan branch March 20, 2025 17:49

jssblck added a commit that referenced this pull request Mar 20, 2025

Circe-powered container scanning (#1521, #1514)

23fa998

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[2/2] Circe-powered container scanning #1521

[2/2] Circe-powered container scanning #1521

Uh oh!

jssblck commented Mar 10, 2025 •

edited

Loading

Uh oh!

jssblck commented Mar 19, 2025 •

edited

Loading

Uh oh!

james-fossa Mar 20, 2025

Uh oh!

spatten Mar 20, 2025 •

edited

Loading

Uh oh!

james-fossa left a comment

Uh oh!

spatten left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[2/2] Circe-powered container scanning #1521

[2/2] Circe-powered container scanning #1521

Uh oh!

Conversation

jssblck commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Acceptance criteria

Testing plan

Risks

Metrics

References

Checklist

Uh oh!

jssblck commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

james-fossa Mar 20, 2025

Choose a reason for hiding this comment

Uh oh!

spatten Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

james-fossa left a comment

Choose a reason for hiding this comment

Uh oh!

spatten left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jssblck commented Mar 10, 2025 •

edited

Loading

jssblck commented Mar 19, 2025 •

edited

Loading

spatten Mar 20, 2025 •

edited

Loading