Kubernetes Contributors – Code Organization

Docs: CRI API version skew policy

Mon, 01 Jan 0001 00:00:00 +0000

CRI is a plugin interface which enables the kubelet to use a wide variety of container runtimes, without the need to recompile. CRI consists of a protocol buffers and gRPC API. Read more about CRI API at kubernetes docs.

The CRI API is only intended to be used for the kubelet to container runtime interactions, or for node-level troubleshooting using a tool such as crictl. It is not a common purpose container runtime API for general use, and is intended to be Kubernetes-centric. This is why there may be an undocumented logic within a container runtimes that assumes the order or specific parameters of call(s) that the kubelet makes. Attempts to call CRI API in a different order by a client different than the kubelet, might result in unrecoverable error. Whenever discovered, this logic is being documented and avoided.

Version skew on a node

On a single Node there may be installed multiple components implementing different versions of CRI API.

For example, on a single node there might be:

Kubelet may call into Container Runtime (e.g. containerd) and Image Service Proxy (e.g. stargz-snapshotter). Container Runtime may be versioned with the OS Image, Kubelet is installed by system administrator and Image Service proxy is versioned by the third party vendor.
Image Service Proxy calls into Container Runtime.
CRI tools (e.g. crictl) may be installed by end user to troubleshoot, same as a third party daemonsets. All of them are used to call into the Container Runtime to collect container information.

So on a single node it may happen that Container Runtime is serving a newer version’d kubelet and older versioned crictl. This is a supported scenario within the version skew policy.

Version Skew Policy for CRI API

CRI API has two versions:

Major semantic version (known versions are v1alpha2 (removed in 1.26), v1).
Kubernetes version (for example: @1.23). Note, the cri-api Golang library is versioned as 0.23 as it doesn’t guarantee Go types backward compatibility.

Major semantic version (e.g. v1) is used to introduce breaking changes and major new features that are incompatible with the current API.

Kubernetes version is used to indicate a specific feature set implemented on top of the major semantic version. All changes made without the change of a major semantic version API must be backward and forward compatible.

Kubelet must work with the older Container Runtime if it implements the same semantic version of CRI API (e.g. v1) of up to three Kubernetes minor versions back. New features implemented in CRI API must be gracefully degraded. For example, Kubelet of version 1.26 must work with Container Runtime implementing k8s.io/cri-api@v0.23.0+.
Kubelet must work with Container Runtime if it implements the same semantic version of CRI API (e.g. v1) of up to three minor versions up. New features implemented in CRI API must not change behavior of old method calls and response values. For example, Kubelet of version 1.22 must work with Container Runtime implementing k8s.io/cri-api@v0.25.5.

Versioning

This library contains go classes generated from the CRI API protocol buffers and gRPC API.

The library versioned as 0.XX as Kubernetes doesn’t provide any guarantees on backward compatibility of Go wrappers between versions. However CRI API itself (protocol buffers and gRPC API) is marked as stable v1 version and it is backward compatible between versions.

Versions like v0.<minor>.<patch> (e.g. v0.25.5) are considered stable. It is discouraged to introduce CRI API changes in patch releases and recommended to use versions like v0.<minor>.0.

All alpha and beta versions (e.g. k8s.io/cri-api@v0.26.0-beta.0) should be backward and forward compatible.

Feature development

Some features development requires changes in CRI API and corresponding changes in Container Runtime. Coordinating between Kubernetes branches and release versions and Container Runtime versions is not always trivial.

The recommended feature development flow is following:

Review proposed CRI API changes during the KEP review stage. Some field names and types may not be spelled out exactly at this stage.
Locally implement a prototype that implement changes in both - Kubernetes and Container Runtime.
Submit a Pull Request for Kubernetes implementing CRI API changes alongside the feature code. Feature must be developed to degrade gracefully when used with older Container Runtime according to the Version Skew policy.
Once PR is merged, wait for the next Kubernetes release tag being produced. Find the corresponding CRI API tag (e.g. k8s.io/cri-api@v0.26.0-beta.0).
This tag can be used to implement the feature in Container Runtime. It is recommended to switch to the stable tag like (k8s.io/cri-api@v0.26.0) once available.

Designing new CRI APIs

The following are considerations to take into account designing new features:

The intended behavior, expectations, and call sequence, must be documented directly in the protocol definition to simplify runtime adoption.
The CRI API change must be as simple as possible. Choosing between simplicity and expressiveness, simplicity has a preference.
Existing fields must be reused only if their logical meaning allows it and does not interfere with the existing features. Changing the expected value format or call sequence may break things in a way that is hard to test and should be avoided.

Feature testing

It is highly encouraged to add critest to every new CRI API. Read about CRI API validation.

What’s next

Docs: Kubernetes feature development and container runtimes

Mon, 01 Jan 0001 00:00:00 +0000

The mechanics of a feature development that requires new CRI APIs is covered in documentation on CRI API feature-development. This article defines policies for developing new Kubernetes features that require CRI API changes. The goal of these policies is to ensure great user experience for people trying the new feature early, adopting it when it is enabled by default, and relying on it as a GA functionality.

Notice

This document establishes development policies for new features requiring adherence to the policy by two currently-participating runtime. This is not a commitment from the Kubernetes community to support these features. Also the list of participating container runtimes may change depending on the community engagement.

Participating container runtimes

Features and CRI API are supposed to be portable and generic and not limited to a specific container runtime. However at this moment SIG Node requires every feature to work on two container runtimes: containerd and CRI-O. These are two runtimes that are tested as part of a Kubernetes development and release process to refer features portability. When this document refers to two container runtimes, it assumes both - containerd and CRI-O. If any other container runtimes begin working actively with the Kubernetes community, or one of existing container runtimes will stop engagement, this document will be updated.

Same maturity level (for beta and GA)

Implementation of an API needed for a Kubernetes feature in a container runtime must be at least the same maturity as in k8s at a moment of Kubernetes release. This is similar to the deprecation policy when one feature is replacing another.

With containerd and CRI-O it means that for the GA features, there should be a release of containerd and CRI-O implementing APIs needed for the feature in those container runtimes. And those APIs must not be marked as experimental features (containerd experimental features). For the beta, neither container runtime has a notion of beta feature and realistically the same maturity criteria applies as for GA.

Note, beta releases of containerd and CRI-O have a different semantic than the beta feature in Kubernetes. Beta releases on container runtimes today are referring to the release readiness rather than feature completeness.

Same maturity level (for alpha)

There should be at least one implementation of an API needed for the Kubernetes feature merged into the container runtime development branch (main) or marked as experimental. An e2e test which demonstrates that the feature is working should be merged alongside the code. Note, tests may run against the default branch of a container runtime and the feature can be still not shipped.

The actual container runtime release may be delayed to the later stage, but Kubernetes highly encourages fast availability of a release of a container runtime that can be tested by early adopters.

Minimal number of implementations

Both containerd and CRI-O must have a GA release with the implementations of an API needed for a Kubernetes feature before this Kubernetes feature can be promoted to GA.

Safe Kubernetes defaults

The feature cannot be enabled by default in Kubernetes as a beta feature before the required APIs are implemented in both container runtimes (containerd and CRI-O) and there is a GA release of a container runtime for each. The feature can be marked as beta, but disabled by default, if there is only one container runtime implementation of a required API that is released as GA. Note, as for any Kubernetes features, at least one release with the beta feature enabled by default is required before it is progressing to GA.

Guarantee portability

The feature design (KEP PR) MUST be lgtm-ed by container runtime maintainers of CRI-O and containerd.

The feature can only be merged as alpha in Kubernetes, if there is an agreement from both container runtime maintainers on the feature design in general and API shape. Gaining this agreement will often involve authoring the pull request demonstrating an API implementation to the both container runtime repositories or an alternative way for container runtime maintainers to confirm viability of suggested CRI APIs.

Guaranteed implementation

CRI API can only be merged if there is a PR in both - Kubernetes repository and container runtime repository (at least one) utilizing this API and demoing the feature working end to end. See CRI API feature-development documentation.

Features discoverability

Kubernetes features that depend on the environment or special container runtime capabilities must have its own explicit API configuration (like Pod API or Node API) and must not depend on the cluster or node configuration that is not clearly exposed via these APIs. For example, it is OK to have Windows specific features that are configured via Pod API. But it is not OK to design a feature that will work on one container runtime and incompatible with the other container runtime. There are three exceptions to this case:

there will be a different behavior during the feature adoption period while older runtime versions do not support the API yet. In those cases, attempting to try the feature must result in failing as fast as possible.
LTS and older versions of container runtimes may not have an implementation of an API and still be widely used by Kubernetes end users.
If any of container runtime underlying systems cannot support the feature in-principle (e.g. kata containers with CRI-O may have limitations), while CRI-O still supports the feature without these systems configured, this must be designed as part of a normal operation. In this case, Pod or Node APIs must handle these cases gracefully and those must be documented clearly.