If you are not familiar with fuzzing, this motivation document provides a quick 1-page introduction.
fzgo is a prototype of golang/go#19109 "cmd/go: make fuzzing a first class citizen, like tests or benchmarks".
fzgo supports some conveniences like fuzzing rich signatures and auto-generation of fuzzing functions.
The basic approach is that fzgo integrates dvyukov/go-fuzz
into go test, with the heavy lifting being done by go-fuzz, go-fuzz-build, and the go tool. The focus
is on step 1 of a tentative list of "Draft goals for a prototype" outlined in this
comment on #19109:
Step 1. Prototype proposed CLI, including interaction with existing 'go test'.
fzgo supports the -fuzz flag and several other related flags proposed in the March 2017
#19109 proposal document. fzgo also supports typical go commands
such as fzgo build, fzgo test, or fzgo env (which are implemented by wrapping the go tool).
Any and all feedback is welcome!
- Rich signatures like
FuzzRegexp(re string, input []byte, posix bool)are supported, as well as the classicFuzz(data []byte) intform used bygo-fuzz. - The corpus is automatically used as deterministic input to unit tests when running a normal
go test. - Individual corpus files can be unit tested via
fzgo test -fuzz=. -run=TestCorpus/<name>. go-fuzzrequires a two step process.fzgoeliminates the separate manual preparation step.fzgoautomatically caches instrumented binaries inGOPATH/pkg/fuzzand re-uses them if possible.- The fuzzing corpus defaults to
GOPATH/pkg/fuzz/corpus. - The
-fuzzdir=/some/pathflag allows the corpus to be stored elsewhere (e.g., a separate corpus repo);-fuzzdir=testdatastores the corpus under<pkgpath>/testdata/fuzz/fuzzname(hence typically in VCS with the code under test). fuzzandgofuzzbuild tags are allowed but not required.- An optional genfuzzfuncs utility can automatically create fuzzing functions for all of the public functions and methods in a package of interest. This makes it quicker and easier to start fuzzing.
Usage: fzgo test [build/test flags] [packages] [build/test flags]
Examples:
fzgo test # normal 'go test' of current package, plus run any corpus as unit tests
fzgo test -fuzz=. # fuzz the current package with a function starting with 'Fuzz'
fzgo test -fuzz=FuzzFoo # fuzz the current package with a function matching 'FuzzFoo'
fzgo test ./... -fuzz=FuzzFoo # fuzz a package in ./... with a function matching 'FuzzFoo'
fzgo test sample/pkg -fuzz=FuzzFoo # fuzz 'sample/pkg' with a function matching 'FuzzFoo'
Rich signatures like Fuzz(re string, input []byte, posix bool)` are supported, as well Fuzz(data []byte) int.
Fuzz functions must start with 'Fuzz'.
The following flags work with 'fzgo test -fuzz':
-fuzz regexp
fuzz at most one function matching regexp
-fuzzdir dir
store fuzz artifacts in dir (default pkgpath/testdata/fuzz)
-fuzztime d
fuzz for duration d (default unlimited)
-parallel n
start n fuzzing operations (default GOMAXPROCS)
-timeout d
fail an individual call to a fuzz function after duration d (default 10s, minimum 1s)
-c
compile the instrumented code but do not run it
-v
verbose: print additional output
$ go get -u github.com/thepudds/fzgo/...
$ go get -u github.com/dvyukov/go-fuzz/...
Note: if you already have an older dvyukov/go-fuzz, you might need to first delete it from GOPATH as described in
the dvyukov/go-fuzz repo.
The go-fuzz source code must be in your GOPATH, and the go-fuzz and go-fuzz-build binaries must be
in your path environment variable.
Note: Module-mode is not supported (#15), but you can fuzz a module with fzgo as long as the code under test is in GOPATH and you set GO111MODULE=off env variable.
This is a simple prototype. Don't expect great things. ;-)
That said, there is reasonable test coverage and fzgo is hopefully beta quality. Automatically generating fuzz functions is implemented in a separate genfuzzfuncs utility that is more alpha quality.
Testing is primarily done with the nice internal testscripts package used by the core Go team to test the go tool
and extracted at rogpeppe/go-internal/testscript.
The primary changes between the current fzgo prototype vs. the March 2017 proposal document:
- fzgo supports rich signatures.
- The corpus location does not default to
<pkgpath>/testdata/fuzz, but instead follows the approach outlined here and more precisely described in PR #7. - Initially, fzgo disallowed multiple fuzz functions to match (per the March 2017 proposal), but as an experiment fzgo now allows multiple fuzz functions to match in order to support something like 'go test -fuzz=. ./...' when there are multiple fuzz functions across multiple packages. Fuzzing happens in round-robin manner if multiple fuzz functions match.
- The proposal document suggested
GOPATH/pkg/GOOS_GOARCH_fuzz/for a cache, but the prototype instead usesGOPATH/pkg/fuzz/GOOS_GOARCH/. - The initial proposal document suggested generating new mutation-based inputs during
go testwhen-fuzzwas not specified. In order to keepgo testdeterministic,fzgodoes not do that, but now does use the corpus as a deterministic set of inputs duringgo testwhen-fuzzis not specified. Also, the proposal document suggested-fuzzinputas a way of specifying a file from the corpus to execute as a unit test.fzgoinstead uses the normal-runargument togo test. For example,fzgo test -run=TestCorpus/4fa128cf066f2a31 some/pkgruns the any file in thesome/pkgcorpus with a filename matching4fa128cf066f2a31. - Some of the commentators at #19109 suggested
-fuzztime durationas a way of controlling when to stop fuzzing. The proposal document does not include-fuzztimeandgo-fuzzdoes not support it, but it seems useful in general and-fuzztimeis in the prototype (and it proved useful while testing the prototype). This might be removed later. - For experimentation,
FZGOFLAGSBUILDandFZGOFLAGSFUZZenvironmental variables can optionally contain a space-separated list of arguments to pass togo-fuzz-buildandgo-fuzz, respectively.
fuzz.Fortesting.Fsignature for fuzzing function.- Allowing fuzzing functions to reside in
*_test.gofiles. - Anything to do with deeper integration with the compiler for more robust instrumentation. This prototype is not focused on that area.
- Any of a much larger set of preexisting build flags like
-ldflags,-coverprofile. - Areas covered in the March 2017 proposal document, outside of the direct user-facing behavior that this prototype focuses on. That said, the majority of user-facing behavior mentioned in the proposal document is either implemented in the prototype or explicitly mentioned in this list as not implemented.
The argument parsing in 'go test' is bespoke, and the argument parsing in fzgo is an approximation of that.
That might be OK for an early prototype. The right thing to do might be to extract
src/cmd/go/internal/test/testflag.go,
which includes this comment:
// The flag handling part of go test is large and distracting.
// We can't use the flag package because some of the flags from
// our command line are for us, and some are for 6.out, and
// some are for both.