Image Policies

The GitOps Toolkit Custom Resource Definitions documentation.

The ImagePolicies API defines rules for selecting a “latest” image from ImageRepositories.

Example

The following is an example of an ImagePolicy. It queries the referred ImageRepository for the image name of the repository, reads all the tags in the repository and selects the latest tag based on the defined policy rules.

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: podinfo
  namespace: default
spec:
  imageRepositoryRef:
    name: podinfo
  digestReflectionPolicy: IfNotPresent
  policy:
    semver:
      range: 5.1.x

In the above example:

  • An ImagePolicy named podinfo is created, indicated by the .metadata.name field.
  • The image-reflector-controller applies the latest tag selection policy every time there’s an update in the referred ImageRepository, indicated by the .spec.imageRepositoryRef.name field.
  • It fetches the canonical image name of the referred ImageRepository and reads the scanned tags from the internal database for the image name. The read tags are then used to select the latest tag based on the policy defined in .spec.policy.
  • The latest image’s name is derived from the ImageRepository image and reported together with the selected tag and digest in the .status.latestRef object.

This example can be run by saving the manifest into imagepolicy.yaml.

  1. Apply the resource on the cluster:
kubectl apply -f imagepolicy.yaml
  1. Run kubectl get imagepolicy to see the ImagePolicy:
NAME      IMAGE                          TAG     READY   STATUS                                                                  AGE
podinfo   ghcr.io/stefanprodan/podinfo   5.1.4   True    Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4   5m
  1. Run kubectl describe imagepolicy podinfo to see the Latest Ref and Conditions in the ImagePolicy’s Status:
Status:
  Conditions:
    Last Transition Time:  2022-09-20T07:09:56Z
    Message:               Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4
    Observed Generation:   1
    Reason:                Succeeded
    Status:                True
    Type:                  Ready
  Latest Ref:
    Digest:                 sha256:2d9a00b3981628a533ff43352193b1838b0a4bf6b0033444286f563205e51a2c
    Image:                  ghcr.io/stefanprodan/podinfo
    Tag:                    5.1.4
  Observed Generation:     1
Events:
  Type    Reason     Age              From                        Message
  ----    ------     ----             ----                        -------
  Normal  Succeeded  7s (x3 over 8s)  image-reflector-controller  Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4

Writing an ImagePolicy spec

As with all other Kubernetes config, an ImagePolicy needs apiVersion, kind, and metadata fields. The name of an ImagePolicy object must be a valid DNS subdomain name.

An ImagePolicy also needs a .spec section.

Image Repository Reference

.spec.imageRepositoryRef is a required field that specifies the ImageRepository for which the latest image has to be selected. The value must be a namespaced object reference. For ImageRepository in the same namespace as the ImagePolicy, no namespace needs to be provided. For ImageRepository in a different namespace than the namespace of the ImagePolicy, namespace name has to be provided. For example:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: podinfo
  namespace: default
spec:
  imageRepositoryRef:
    name: podinfo
    namespace: flux-system
...

The ImageRepository access is determied by its ACL for cross-namespace reference. For more details on how to allow cross-namespace references see the ImageRepository docs.

Policy

.spec.policy is a required field that specifies how to choose a latest image given the image metadata. There are three image policy choices:

  • SemVer
  • Alphabetical
  • Numerical

SemVer

SemVer policy interprets all the tags as semver versions and chooses the highest version available that fits the given semver constraints. The constraint is set in the .spec.policy.semver.range field.

Example of a SemVer image policy choice:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: podinfo
spec:
  imageRepositoryRef:
    name: podinfo
  policy:
    semver:
      range: '>=1.0.0'

This will select the latest stable version tag.

Alphabetical

Alphabetical policy chooses the last tag when all the tags are sorted alphabetically (in either ascending or descending order). The sort order is set in the .spec.policy.alphabetical.order field. The value could be asc for ascending order or desc for descending order. The default value is asc.

Example of an Alphabetical policy choice:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: podinfo
spec:
  imageRepositoryRef:
    name: podinfo
  policy:
    alphabetical:
      order: asc

This will select the last tag when all the tags are sorted alphabetically in ascending order.

Numerical

Numerical policy chooses the last tag when all the tags are sorted numerically (in either ascending or descending order). The sort order is set in the .spec.policy.numerical.order field. The value could be asc for ascending order or desc for descending order. The default value is asc.

Example of a Numerical policy choice:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: podinfo
spec:
  imageRepositoryRef:
    name: podinfo
  policy:
    numerical:
      order: asc

This will select the last tag when all the tags are sorted numerically in ascending order.

Filter Tags

.spec.filterTags is an optional field to specify a filter on the image tags before they are considered by the policy rule.

The filter pattern is a regular expression, set in the .spec.filterTags.pattern field. Only tags that match the pattern are considered by the policy rule.

The .spec.filterTags.extract is an optional field used to extract a value from the matching tags which is supplied to the policy rule instead of the original tags. If unspecified, the tags that match the pattern will be used as they are.

Example of selecting the latest release candidate (semver):

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: podinfo
spec:
  imageRepositoryRef:
    name: podinfo
  filterTags:
    pattern: '.*-rc.*'
  policy:
    semver:
      range: '^1.x-0'

Example of selecting the latest release tagged as RELEASE.<RFC3339-TIMESTAMP> (alphabetical):

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: minio
spec:
  imageRepositoryRef:
    name: minio
  filterTags:
    pattern: '^RELEASE\.(?P<timestamp>.*)Z$'
    extract: '$timestamp'
  policy:
    alphabetical:
      order: asc

In the above example, the timestamp value from the tag pattern is extracted and used in the policy rule to determine the latest tag.

Digest Reflection

.spec.digestReflectionPolicy is a field that governs the reflection of the selected image’s digest in the ImagePolicy’s .status.latestRef.digest field. The field has three possible values:

  • Never: If the field is set to Never (the default) the digest will not be reflected at all.
  • Always: This value leads to the digest of the latest tag to always be reflected in .status.latestRef.digest. An existing, potentially different digest will be overwritten with the most recent value retrieved from the image registry even if the tag didn’t change. This may be useful to track mutable tags like latest.
  • IfNotPresent: This value will only store the digest of the latest tag once and never overwrite an existing value unless the tag has changed as well. This is the safest option to track immutable tags.

Interval

.spec.interval specifies the interval at which the ImagePolicy must refresh the digest of the latest tag. The value must be in a Go recognized duration string format, e.g. 10m0s to reconcile the object every 10 minutes. This field must and can only be specified when .spec.digestReflectionPolicy is set to Always.

Working with ImagePolicy

Triggering a reconcile

ImagePolicy is reconciled automatically when the associated ImageRepository is updated. Whenever ImageRepository gets updated, ImagePolicy will be triggered and have the policy result based on the latest values of ImageRepository. To manually tell the image-reflector-controller to reconcile an ImagePolicy, the associated ImageRepository can be annotated with reconcile.fluxcd.io/requestedAt: <arbitrary value>. See triggering a reconcile for more details about reconciling ImageRepository.

Waiting for Ready

When a change is applied, it is possible to wait for the ImagePolicy to reach a ready state using kubectl:

kubectl wait imagepolicy/<policy-name> --for=condition=ready --timeout=1m

Debugging an ImagePolicy

There are several ways to gather information about an ImagePolicy for debugging purposes.

Describe the ImagePolicy

Describing an ImagePolicy using kubectl describe imagepolicy <policy-name> displays the latest recorded information for the resource in the Status and Events sections:

...
Status:
  Conditions:
    Last Transition Time:  2022-10-06T12:07:35Z
    Message:               accessing ImageRepository
    Observed Generation:   1
    Reason:                AccessingRepository
    Status:                True
    Type:                  Reconciling
    Last Transition Time:  2022-10-06T12:07:35Z
    Message:               failed to get the referred ImageRepository: referenced ImageRepository does not exist: ImageRepository.image.toolkit.fluxcd.io "podinfo" not found
    Observed Generation:   1
    Reason:                DependencyNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type     Reason              Age                From                        Message
  ----     ------              ----               ----                        -------
  Warning  DependencyNotReady  2s (x4 over 5s)    image-reflector-controller  failed to get the referred ImageRepository: referenced ImageRepository does not exist: ImageRepository.image.toolkit.fluxcd.io "podinfo" not found

Trace emitted Events

To view events for specific ImagePolicy(s), kubectl events can be used in combination with --for to list the Events for specific objects. For example, running

kubectl events --for ImagePolicy/<policy-name>

lists

LAST SEEN   TYPE      REASON               OBJECT                      MESSAGE
4m44s       Normal    Succeeded            imagepolicy/<policy-name>   Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.1.4
95s         Warning   DependencyNotReady   imagepolicy/<policy-name>   failed to get the referred ImageRepository: referenced ImageRepository does not exist: ImageRepository.image.toolkit.fluxcd.io "podinfo" not found

Besides being reported in Events, the reconciliation errors are also logged by the controller. The Flux CLI offer commands for filtering the logs for a specific ImagePolicy, e.g. flux logs --level=error --kind=ImagePolicy --name=<policy-name>.

ImagePolicy Status

Latest Ref

The ImagePolicy reports the latest selected image from the ImageRepository tags in .status.latestRef for the resource. The field .status.latestRef.digest is dependent on the chosen digest reflection policy and is only set for the Always or IfNotPresent policies.

Example:

---
apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: <policy-name>
status:
  latestRef:
    image: ghcr.io/stefanprodan/podinfo
    tag: 5.1.4
    digest: sha256:2d9a00b3981628a533ff43352193b1838b0a4bf6b0033444286f563205e51a2c
[...]

Observed Previous Ref

The ImagePolicy reports the previously observed latest image in .status.observedPreviousRef for the resource. This is used by the ImagePolicy to determine an upgrade path of an ImagePolicy update. This field is reset when the ImagePolicy fails due to some reason to be able to distinguish between a failure recovery and a genuine latest image upgrade.

Example:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
  name: <policy-name>
status:
  latestRef:
    image: ghcr.io/stefanprodan/podinfo
    tag: 6.2.1
  observedPreviousRef:
    image: ghcr.io/stefanprodan/podinfo
    tag: 5.1.4

Conditions

An ImagePolicy enters various states during its lifecycle, reflected as Kubernetes Conditions. It can be reconciling while reading the tags from ImageRepository scan results, it can be ready, or it can fail during reconciliation.

The ImagePolicy API is compatible with the kstatus specification, and reports Reconciling and Stalled conditions where applicable to provide better (timeout) support to solutions polling the ImagePolicy to become Ready.

Reconciling ImagePolicy

The image-reflector-controller marks an ImagePolicy as reconciling when one of the following is true:

  • The generation of the ImagePolicy is newer than the Observed Generation.
  • The ImagePolicy is accessing the provided ImageRepository reference.
  • The ImagePolicy is being applied to the tags read from an ImageRepository.

When the ImagePolicy is “reconciling”, the Ready Condition status becomes False, and the controller adds a Condition with the following attributes to the ImagePolicy’s .status.conditions:

  • type: Reconciling
  • status: "True"
  • reason: NewGeneration | reason:AccessingRepository | reason: ApplyingPolicy

It has a “negative polarity”, and is only present on the ImagePolicy while its status value is "True".

Ready ImagePolicy

The image-reflector-controller marks an ImagePolicy as ready when it has the following characteristics:

  • The ImagePolicy reports a Latest Image
  • The referenced ImageRepository is accessible and the internal tags database contains the tags that ImagePolicy needs to apply the policy on.

When the ImagePolicy is “ready”, the controller sets a Condition with the following attributes in the ImagePolicy’s .status.conditions.

  • type: Ready
  • status: "True"
  • reason: Succeeded

This Ready Condition will retain a status value of "True" until the ImagePolicy is marked as reconciling, or e.g. a transient error occurs due to a temporary network issue.

Failed ImagePolicy

The image-reflector-controller may get stuck trying to apply a policy without completing. This can occur due to some of the following factors:

  • The referenced ImageRepository is temporarily unavailable.
  • The referenced ImageRepository does not exist.
  • The referenced ImageRepository is not accessible in a different namespace.
  • The ImagePolicy spec contains a generic misconfiguration.
  • The ImagePolicy could not select the latest tag based on the given rules and the available tags.
  • A database related failure when reading or writing the scanned tags.

When this happens, the controller sets the Ready condition status to False wit the following reason:

  • reason: Failure | reason: AccessDenied | reason: DependencyNotReady

While the ImagePolicy is in failing state, the controller will continue to attempt to get the referenced ImageRepository for the resource and apply the policy rules with an exponential backoff, until it succeeds and the ImagePolicy is marked as ready.

Note that an ImagePolicy can be reconcilcing while failing at the same time, for example due to a newly introduced configuration issue in the ImagePolicy spec.

Observed Generation

The image-reflector-controller reports an observed generation in the ImagePolicy’s .status.observedGeneration. The observed generation is the latest .metadata.generation which resulted in either a ready state, or stalled due to error it can not recover from without human intervention.