Skip to content

Output is not deterministic #32

@jwalton

Description

@jwalton

If you run:

$ qbuild
$ cp build/foo_unknown.qpkg /tmp
$ qbuild
$ diff /tmp/foo_unknown.qpkg build/foo_unknown.qpkg
Binary files /tmp/foo_unknown.qpkg and build/foo/unknown.qpkg differ

This causes problems when using build systems like bazel, as we'll end up with a cache miss. There are also many other reasons why reproducible builds are a desireable thing.

I'm not sure if I've tracked down all the reasons this happens, but there are a couple I've found.

built_version

The built_version file contains the time at which the build happened. It would be nice if we could pass in a timestamp when we run qbuild and record this timestamp in the built_version file. This would let us pass in, for example, the timestamp of the most recent git commit, so the contents of this file would be predictable.

Timestamps in archives.

Threre are timestamps being recorded in various tar archives. and these timestamps are not stable. For example, when we call create_control_package, this copies a number of files into the build.$$ folder (which updates their mtimes to the current time) then runs tar to create the control.tar file. We should either be preserving the attributes of these files (with cp -a for example), or passing the --mtime option to tar.

The built_version file is another offender here, as we create it right before generating the control package. If we allow passing in a build timestamp, we could either touch this file with that timestamp or again use tar --mtime "${BUILD_TIMESTAMP}" ... to force the mtime of this file.

There are some places where we are implicitly recording timestamps too. For example, when creating the data archive with 7z we run:

/bin/tar $tar_verbose -cf - -C build.$$ . | /usr/local/sbin/7z a -si tmp.$$/data.tar.7z

and here 7z will record the current timestamp because of the -si option. We can work around this with:

/bin/tar $tar_verbose -cf tmp.$$/data.tar -C build.$$ . 
touch -d "${BUILD_TIME}" tmp.$$/data.tar
/usr/local/sbin/7z a tmp.$$/data.tar tmp.$$/data.tar.7z

which will explicitly set the timestamp in the 7z file. There's a similar problem with gzip compression, which can be worked around in much the same way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions