Description:

This is a follow-up to #1652.

It attempts to 'intelligently' chunk inputs on subsequent newlines \n\n, rather than indiscriminately breaking text apart. The assumption is that it's safe(r) for a chunk to end at (1), rather than (2) or (3). Of course, the buffer cannot grow indefinitely or the original issue would regress; it will only search for consecutive newlines up to a maxPeekSize.

  // TODO: optimization could be introduced here
  if mimetype, err := filetype.Match(buf[:n]); err != nil {
    return nil
  } else if mimetype.MIME.Type == "application" {
    return nil // skip binary files
  }

>>> (1) <<<  // If the chunk doesn't end in a newli>>> (2) <<<ne, peek|maxPeekSize| until we find one.
  // This hopefully avoids splitting
>>> (3) <<< // See: https://github.com/gitleaks/gitleaks/issues/1651
  var (

The original issue with 60 private keys is solved by this change:

$ ./gitleaks dir /tmp/60keys.txt --no-color

10:22PM INF scan completed in 88ms
10:22PM WRN leaks found: 60

Thoughts

• Needs to be tested on Windows or files with CRLF.
• Reading one byte at a time might be slow (idk, seems pretty fast)
• It might make sense for whitespace not to reset the counter. That way empty lines can match the criteria.

    }
     
  // Foo
  

Checklist:

• Does your PR pass tests?
• Have you written new tests for your changes?
• Have you lint your code locally prior to submission?