The code remains available for historic purposes.
The README as of the archival date remains unchanged below for historic purposes.
kaniko is a tool to build container images from a Dockerfile, inside a container or Kubernetes cluster.
kaniko doesn't depend on a Docker daemon and executes each command within a Dockerfile completely in userspace. This enables building container images in environments that can't easily or securely run a Docker daemon, such as a standard Kubernetes cluster.
kaniko is meant to be run as an image: gcr.io/kaniko-project/executor
. We do
not recommend running the kaniko executor binary in another image, as it
might not work as you expect - see Known Issues.
We'd love to hear from you! Join us on #kaniko Kubernetes Slack
📣 Please fill out our quick 5-question survey so that we can learn how satisfied you are with kaniko, and what improvements we should make. Thank you! 👯
If you are interested in contributing to kaniko, see DEVELOPMENT.md and CONTRIBUTING.md.
Table of Contents generated with DocToc
- kaniko - Build Images In Kubernetes
- 🚨NOTE: kaniko is not an officially supported Google product🚨
- Community
- How does kaniko work?
- Known Issues
- Demo
- Tutorial
- Using kaniko
- kaniko Build Contexts
- Using Azure Blob Storage
- Using Private Git Repository
- Using Standard Input
- Running kaniko
- Caching
- Pushing to Different Registries
- Additional Flags
- Flag
--build-arg
- Flag
--cache
- Flag
--cache-dir
- Flag
--cache-repo
- Flag
--cache-copy-layers
- Flag
--cache-run-layers
- Flag
--cache-ttl duration
- Flag
--cleanup
- Flag
--compressed-caching
- Flag
--context-sub-path
- Flag
--custom-platform
- Flag
--digest-file
- Flag
--dockerfile
- Flag
--force
- Flag
--git
- Flag
--image-name-with-digest-file
- Flag
--image-name-tag-with-digest-file
- Flag
--insecure
- Flag
--insecure-pull
- Flag
--insecure-registry
- Flag
--label
- Flag
--log-format
- Flag
--log-timestamp
- Flag
--no-push
- Flag
--no-push-cache
- Flag
--oci-layout-path
- Flag
--push-retry
- Flag
--registry-certificate
- Flag
--registry-client-cert
- Flag
--registry-map
- Flag
--registry-mirror
- Flag
--skip-default-registry-fallback
- Flag
--reproducible
- Flag
--single-snapshot
- Flag
--skip-push-permission-check
- Flag
--skip-tls-verify
- Flag
--skip-tls-verify-pull
- Flag
--skip-tls-verify-registry
- Flag
--skip-unused-stages
- Flag
--snapshot-mode
- Flag
--tar-path
- Flag
--target
- Flag
--use-new-run
- Flag
--verbosity
- Flag
--ignore-var-run
- Flag
--ignore-path
- Flag
--image-fs-extract-retry
- Flag
--image-download-retry
- Flag
- Debug Image
- Security
- Kaniko Builds - Profiling
- Creating Multi-arch Container Manifests Using Kaniko and Manifest-tool
- Comparison with Other Tools
- Community
- Limitations
- References
We'd love to hear from you! Join #kaniko on Kubernetes Slack
The kaniko executor image is responsible for building an image from a Dockerfile and pushing it to a registry. Within the executor image, we extract the filesystem of the base image (the FROM image in the Dockerfile). We then execute the commands in the Dockerfile, snapshotting the filesystem in userspace after each one. After each command, we append a layer of changed files to the base image (if there are any) and update image metadata.
- kaniko does not support building Windows containers.
- Running kaniko in any Docker image other than the official kaniko image is not
supported due to implementation details.
- This includes copying the kaniko executables from the official image into another image (e.g. a Jenkins CI agent).
- In particular, it cannot use chroot or bind-mount because its container must not require privilege, so it unpacks directly into its own container root and may overwrite anything already there.
- kaniko does not support the v1 Registry API (Registry v1 API Deprecation)
For a detailed example of kaniko with local storage, please refer to a getting started tutorial.
Please see References for more docs & video tutorials
To use kaniko to build and push an image for you, you will need:
- A build context, aka something to build
- A running instance of kaniko
kaniko's build context is very similar to the build context you would send your
Docker daemon for an image build; it represents a directory containing a
Dockerfile which kaniko will use to build your image. For example, a COPY
command in your Dockerfile should refer to a file in the build context.
You will need to store your build context in a place that kaniko can access. Right now, kaniko supports these storage solutions:
- GCS Bucket
- S3 Bucket
- Azure Blob Storage
- Local Directory
- Local Tar
- Standard Input
- Git Repository
Note about Local Directory: this option refers to a directory within the kaniko container. If you wish to use this option, you will need to mount in your build context into the container as a directory.
Note about Local Tar: this option refers to a tar gz file within the kaniko container. If you wish to use this option, you will need to mount in your build context into the container as a file.
Note about Standard Input: the only Standard Input allowed by kaniko is in
.tar.gz
format.
If using a GCS or S3 bucket, you will first need to create a compressed tar of your build context and upload it to your bucket. Once running, kaniko will then download and unpack the compressed tar of the build context before starting the image build.
To create a compressed tar, you can run:
tar -C <path to build context> -zcvf context.tar.gz .
Then, copy over the compressed tar into your bucket. For example, we can copy over the compressed tar to a GCS bucket with gsutil:
gsutil cp context.tar.gz gs://<bucket name>
When running kaniko, use the --context
flag with the appropriate prefix to
specify the location of your build context:
Source | Prefix | Example |
---|---|---|
Local Directory | dir://[path to a directory in the kaniko container] | dir:///workspace |
Local Tar Gz | tar://[path to a .tar.gz in the kaniko container] | tar:///path/to/context.tar.gz |
Standard Input | tar://[stdin] | tar://stdin |
GCS Bucket | gs://[bucket name]/[path to .tar.gz] | gs://kaniko-bucket/path/to/context.tar.gz |
S3 Bucket | s3://[bucket name]/[path to .tar.gz] | s3://kaniko-bucket/path/to/context.tar.gz |
Azure Blob Storage | https://[account].[azureblobhostsuffix]/[container]/[path to .tar.gz] | https://myaccount.blob.core.windows.net/container/path/to/context.tar.gz |
Git Repository | git://[repository url][#reference][#commit-id] | git://github.com/acme/myproject.git#refs/heads/mybranch#<desired-commit-id> |
If you don't specify a prefix, kaniko will assume a local directory. For
example, to use a GCS bucket called kaniko-bucket
, you would pass in
--context=gs://kaniko-bucket/path/to/context.tar.gz
.
If you are using Azure Blob Storage for context file, you will need to pass
Azure Storage Account Access Key
as an environment variable named AZURE_STORAGE_ACCESS_KEY
through Kubernetes
Secrets
You can use Personal Access Tokens
for Build Contexts from Private
Repositories from
GitHub.
You can either pass this in as part of the git URL (e.g.,
git://[email protected]/acme/myproject.git#refs/heads/mybranch
) or using the
environment variable GIT_TOKEN
.
You can also pass GIT_USERNAME
and GIT_PASSWORD
(password being the token)
if you want to be explicit about the username.
If running kaniko and using Standard Input build context, you will need to add
the docker or kubernetes -i, --interactive
flag. Once running, kaniko will
then get the data from STDIN
and create the build context as a compressed tar.
It will then unpack the compressed tar of the build context before starting the
image build. If no data is piped during the interactive run, you will need to
send the EOF signal by yourself by pressing Ctrl+D
.
Complete example of how to interactively run kaniko with .tar.gz
Standard
Input data, using docker:
echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | docker run \
--interactive -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest \
--context tar://stdin \
--destination=<gcr.io/$project/$image:$tag>
Complete example of how to interactively run kaniko with .tar.gz
Standard
Input data, using Kubernetes command line with a temporary container and
completely dockerless:
echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | kubectl run kaniko \
--rm --stdin=true \
--image=gcr.io/kaniko-project/executor:latest --restart=Never \
--overrides='{
"apiVersion": "v1",
"spec": {
"containers": [
{
"name": "kaniko",
"image": "gcr.io/kaniko-project/executor:latest",
"stdin": true,
"stdinOnce": true,
"args": [
"--dockerfile=Dockerfile",
"--context=tar://stdin",
"--destination=gcr.io/my-repo/my-image"
],
"volumeMounts": [
{
"name": "cabundle",
"mountPath": "/kaniko/ssl/certs/"
},
{
"name": "docker-config",
"mountPath": "/kaniko/.docker/"
}
]
}
],
"volumes": [
{
"name": "cabundle",
"configMap": {
"name": "cabundle"
}
},
{
"name": "docker-config",
"configMap": {
"name": "docker-config"
}
}
]
}
}'
There are several different ways to deploy and run kaniko:
Requirements:
- Standard Kubernetes cluster (e.g. using GKE)
- Kubernetes Secret
- A build context
To run kaniko in a Kubernetes cluster, you will need a standard running Kubernetes cluster and a Kubernetes secret, which contains the auth required to push the final image.
To create a secret to authenticate to Google Cloud Registry, follow these steps:
- Create a service account in the Google Cloud Console project you want to push
the final image to with
Storage Admin
permissions. - Download a JSON key for this service account
- Rename the key to
kaniko-secret.json
- To create the secret, run:
kubectl create secret generic kaniko-secret --from-file=<path to kaniko-secret.json>
Note: If using a GCS bucket in the same GCP project as a build context, this service account should now also have permissions to read from that bucket.
The Kubernetes Pod spec should look similar to this, with the args parameters filled in:
apiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args:
- "--dockerfile=<path to Dockerfile within the build context>"
- "--context=gs://<GCS bucket>/<path to .tar.gz>"
- "--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>"
volumeMounts:
- name: kaniko-secret
mountPath: /secret
env:
- name: GOOGLE_APPLICATION_CREDENTIALS
value: /secret/kaniko-secret.json
restartPolicy: Never
volumes:
- name: kaniko-secret
secret:
secretName: kaniko-secret
This example pulls the build context from a GCS bucket. To use a local directory build context, you could consider using configMaps to mount in small build contexts.
Running kaniko in gVisor provides an
additional security boundary. You will need to add the --force
flag to run
kaniko in gVisor, since currently there isn't a way to determine whether or not
a container is running in gVisor.
docker run --runtime=runsc -v $(pwd):/workspace -v ~/.config:/root/.config \
gcr.io/kaniko-project/executor:latest \
--dockerfile=<path to Dockerfile> --context=/workspace \
--destination=gcr.io/my-repo/my-image --force
We pass in --runtime=runsc
to use gVisor. This example mounts the current
directory to /workspace
for the build context and the ~/.config
directory
for GCR credentials.
Requirements:
To run kaniko in GCB, add it to your build config as a build step:
steps:
- name: gcr.io/kaniko-project/executor:latest
args:
[
"--dockerfile=<path to Dockerfile within the build context>",
"--context=dir://<path to build context>",
"--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>",
]
kaniko will build and push the final image in this build step.
Requirements:
We can run the kaniko executor image locally in a Docker daemon to build and push an image from a Dockerfile.
For example, when using gcloud and GCR you could run kaniko as follows:
docker run \
-v "$HOME"/.config/gcloud:/root/.config/gcloud \
-v /path/to/context:/workspace \
gcr.io/kaniko-project/executor:latest \
--dockerfile /workspace/Dockerfile \
--destination "gcr.io/$PROJECT_ID/$IMAGE_NAME:$TAG" \
--context dir:///workspace/
There is also a utility script run_in_docker.sh
that can
be used as follows:
./run_in_docker.sh <path to Dockerfile> <path to build context> <destination of final image>
NOTE: run_in_docker.sh
expects a path to a Dockerfile relative to the
absolute path of the build context.
An example run, specifying the Dockerfile in the container directory
/workspace
, the build context in the local directory
/home/user/kaniko-project
, and a Google Container Registry as a remote image
destination:
./run_in_docker.sh /workspace/Dockerfile /home/user/kaniko-project gcr.io/$PROJECT_ID/$TAG
kaniko can cache layers created by RUN
(configured by flag
--cache-run-layers
) and COPY
(configured by flag --cache-copy-layers
)
commands in a remote repository. Before executing a command, kaniko checks the
cache for the layer. If it exists, kaniko will pull and extract the cached layer
instead of executing the command. If not, kaniko will execute the command and
then push the newly created layer to the cache.
Note that kaniko cannot read layers from the cache after a cache miss: once a layer has not been found in the cache, all subsequent layers are built locally without consulting the cache.
Users can opt into caching by setting the --cache=true
flag. A remote
repository for storing cached layers can be provided via the --cache-repo
flag. If this flag isn't provided, a cached repo will be inferred from the
--destination
provided.
kaniko can cache images in a local directory that can be volume mounted into the
kaniko pod. To do so, the cache must first be populated, as it is read-only. We
provide a kaniko cache warming image at gcr.io/kaniko-project/warmer
:
docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:latest --cache-dir=/workspace/cache --image=<image to cache> --image=<another image to cache>
docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:latest --cache-dir=/workspace/cache --dockerfile=<path to dockerfile>
docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:latest --cache-dir=/workspace/cache --dockerfile=<path to dockerfile> --build-arg version=1.19
--image
can be specified for any number of desired images. --dockerfile
can
be specified for the path of dockerfile for cache.These command will combined to
cache those images by digest in a local directory named cache
. Once the cache
is populated, caching is opted into with the same --cache=true
flag as above.
The location of the local cache is provided via the --cache-dir
flag,
defaulting to /cache
as with the cache warmer. See the examples
directory
for how to use with kubernetes clusters and persistent cache volumes.
kaniko uses Docker credential helpers to push images to a registry.
kaniko comes with support for GCR, Docker config.json
and Amazon ECR, but
configuring another credential helper should allow pushing to a different
registry.
Get your docker registry user and password encoded in base64
echo -n USER:PASSWORD | base64
Create a config.json
file with your Docker registry url and the previous
generated base64 string
Note: Please use v1 endpoint. See #1209 for more details
{
"auths": {
"https://index.docker.io/v1/": {
"auth": "xxxxxxxxxxxxxxx"
}
}
}
Run kaniko with the config.json
inside /kaniko/.docker/config.json
docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:latest --dockerfile=Dockerfile --destination=yourimagename
To create a credentials to authenticate to Google Cloud Registry, follow these steps:
- Create a
service account
or in the Google Cloud Console project you want to push the final image to
with
Storage Admin
permissions. - Download a JSON key for this service account
- (optional) Rename the key to
kaniko-secret.json
, if you don't rename, you have to change the name used the command(in the volume part) - Run the container adding the path in GOOGLE_APPLICATION_CREDENTIALS env var
docker run -ti --rm -e GOOGLE_APPLICATION_CREDENTIALS=/kaniko/config.json \
-v `pwd`:/workspace -v `pwd`/kaniko-secret.json:/kaniko/config.json:ro gcr.io/kaniko-project/executor:latest \
--dockerfile=Dockerfile --destination=yourimagename
If you have enabled Workload Identity on your GKE cluster then you can use the
workload identity to push built images to GCR without adding a
GOOGLE_APPLICATION_CREDENTIALS
in your kaniko pod specification.
Learn more on how to enable and migrate existing apps to workload identity.
To authenticate using workload identity you need to run the kaniko pod using the
Kubernetes Service Account (KSA) bound to Google Service Account (GSA) which has
Storage.Admin
permissions to push images to Google Container registry.
Please follow the detailed steps here to create a Kubernetes Service Account, Google Service Account and create an IAM policy binding between the two to allow the Kubernetes Service account to act as the Google service account.
To grant the Google Service account the right permission to push to GCR, run the following GCR command
gcloud projects add-iam-policy-binding $PROJECT \
--member=serviceAccount:[gsa-name]@${PROJECT}.iam.gserviceaccount.com \
--role=roles/storage.objectAdmin
Please ensure, kaniko pod is running in the namespace and with a Kubernetes Service Account.
The Amazon ECR credential helper is built into the kaniko executor image.
-
Configure credentials
-
You can use instance roles when pushing to ECR from a EC2 instance or from EKS, by configuring the instance role permissions (the AWS managed policy
EC2InstanceProfileForImageBuilderECRContainerBuilds
provides broad permissions to upload ECR images and may be used as configuration baseline). Additionally, setAWS_SDK_LOAD_CONFIG=true
as environment variable within the kaniko pod. If running on an EC2 instance with an instance profile, you may also need to setAWS_EC2_METADATA_DISABLED=true
for kaniko to pick up the correct credentials. -
Or you can create a Kubernetes secret for your
~/.aws/credentials
file so that credentials can be accessed within the cluster. To create the secret, run:shell kubectl create secret generic aws-secret --from-file=<path to .aws/credentials>
-
The Kubernetes Pod spec should look similar to this, with the args parameters
filled in. Note that aws-secret
volume mount and volume are only needed when
using AWS credentials from a secret, not when using instance roles.
apiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args:
- "--dockerfile=<path to Dockerfile within the build context>"
- "--context=s3://<bucket name>/<path to .tar.gz>"
- "--destination=<aws_account_id.dkr.ecr.region.amazonaws.com/my-repository:my-tag>"
volumeMounts:
# when not using instance role
- name: aws-secret
mountPath: /root/.aws/
restartPolicy: Never
volumes:
# when not using instance role
- name: aws-secret
secret:
secretName: aws-secret
An ACR credential helper is built into the kaniko executor image, which can be used to authenticate with well-known Azure environmental information.
To configure credentials, you will need to do the following:
- Update the
credStore
section ofconfig.json
:
{ "credsStore": "acr" }
A downside of this approach is that ACR authentication will be used for all
registries, which will fail if you also pull from DockerHub, GCR, etc. Thus, it
is better to configure the credential tool only for your ACR registries by using
credHelpers
instead of credsStore
:
{ "credHelpers": { "mycr.azurecr.io": "acr-env" } }
You can mount in the new config as a configMap:
kubectl create configmap docker-config --from-file=<path to config.json>
- Configure credentials
You can create a Kubernetes secret with environment variables required for Service Principal authentication and expose them to the builder container.
AZURE_CLIENT_ID=<clientID>
AZURE_CLIENT_SECRET=<clientSecret>
AZURE_TENANT_ID=<tenantId>
If the above are not set then authentication falls back to managed service identities and the MSI endpoint is attempted to be contacted which will work in various Azure contexts such as App Service and Azure Kubernetes Service where the MSI endpoint will authenticate the MSI context the service is running under.
The Kubernetes Pod spec should look similar to this, with the args parameters
filled in. Note that azure-secret
secret is only needed when using Azure
Service Principal credentials, not when using a managed service identity.
apiVersion: v1
kind: Pod
metadata:
name: kaniko
spec:
containers:
- name: kaniko
image: gcr.io/kaniko-project/executor:latest
args:
- "--dockerfile=<path to Dockerfile within the build context>"
- "--context=s3://<bucket name>/<path to .tar.gz>"
- "--destination=mycr.azurecr.io/my-repository:my-tag"
envFrom:
# when authenticating with service principal
- secretRef:
name: azure-secret
volumeMounts:
- name: docker-config
mountPath: /kaniko/.docker/
volumes:
- name: docker-config
configMap:
name: docker-config
restartPolicy: Never
Kaniko can be used with both JFrog Container Registry and JFrog Artifactory.
Get your JFrog Artifactory registry user and password encoded in base64
echo -n USER:PASSWORD | base64
Create a config.json
file with your Artifactory Docker local registry URL and
the previous generated base64 string
{
"auths": {
"artprod.company.com": {
"auth": "xxxxxxxxxxxxxxx"
}
}
}
For example, for Artifactory cloud users, the docker registry should be:
<company>.<local-repository-name>.io
.
Run kaniko with the config.json
inside /kaniko/.docker/config.json
docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:latest --dockerfile=Dockerfile --destination=yourimagename
After the image is uploaded, using the JFrog CLI, you can collect and publish the build information to Artifactory and trigger build vulnerabilities scanning using JFrog Xray.
To collect and publish the image's build information using the Jenkins Artifactory plugin, see instructions for scripted pipeline and declarative pipeline.
This flag allows you to pass in ARG values at build time, similarly to Docker. You can set it multiple times for multiple arguments.
Note that passing values that contain spaces is not natively supported - you
need to ensure that the IFS is set to null before your executor command. You can
set this by adding export IFS=''
before your executor call. See the following
example
export IFS=''
/kaniko/executor --build-arg "MY_VAR='value with spaces'" ...
Set this flag as --cache=true
to opt into caching with kaniko.
Set this flag to specify a local directory cache for base images. Defaults to
/cache
.
This flag must be used in conjunction with the --cache=true
flag.
Set this flag to specify a remote repository that will be used to store cached layers.
If this flag is not provided, a cache repo will be inferred from the
--destination
flag. If --destination=gcr.io/kaniko-project/test
, then cached
layers will be stored in gcr.io/kaniko-project/test/cache
.
This flag must be used in conjunction with the --cache=true
flag.
Set this flag to cache copy layers.
Set this flag to cache run layers (default=true).
Cache timeout in hours. Defaults to two weeks.
Set this flag to clean the filesystem at the end of the build.
Set this to false in order to prevent tar compression for cached layers. This
will increase the runtime of the build, but decrease the memory usage especially
for large builds. Try to use --compressed-caching=false
if your build fails
with an out of memory error. Defaults to true.
Set a sub path within the given --context
.
Its particularly useful when your context is, for example, a git repository, and you want to build one of its subfolders instead of the root folder.
Allows to build with another default platform than the host, similarly to docker
build --platform xxx the value has to be on the form
--custom-platform=linux/arm
, with acceptable values listed here:
GOOS/GOARCH.
It's also possible specifying CPU variants adding it as a third parameter (like
--custom-platform=linux/arm/v5
). Currently CPU variants are only known to be
used for the ARM architecture as listed here:
GOARM
The resulting images cannot provide any metadata about CPU variant due to a limitation of the OCI-image specification.
This is not virtualization and cannot help to build an architecture not natively supported by the build host. This is used to build i386 on an amd64 Host for example, or arm32 on an arm64 host.
Set this flag to specify a file in the container. This file will receive the digest of a built image. This can be used to automatically track the exact image built by kaniko.
For example, setting the flag to --digest-file=/dev/termination-log
will write
the digest to that file, which is picked up by Kubernetes automatically as the
{{.state.terminated.message}}
of the container.
Path to the dockerfile to be built. (default "Dockerfile")
Force building outside of a container
Branch to clone if build context is a git repository (default branch=,single-branch=false,recurse-submodules=false,insecure-skip-tls=false)
Specify a file to save the image name w/ digest of the built image to.