Kubernetes Contributors – Contributor Blog

Blog: Spotlight on SIG Architecture: API Governance

Thu, 12 Feb 2026 00:00:00 +0000

This is the fifth interview of a SIG Architecture Spotlight series that covers the different subprojects, and we will be covering SIG Architecture: API Governance.

In this SIG Architecture spotlight we talked with Jordan Liggitt, lead of the API Governance sub-project.

Introduction

FM: Hello Jordan, thank you for your availability. Tell us a bit about yourself, your role and how you got involved in Kubernetes.

JL: My name is Jordan Liggitt. I’m a Christian, husband, father of four, software engineer at Google by day, and amateur musician by stealth. I was born in Texas (and still like to claim it as my point of origin), but I’ve lived in North Carolina for most of my life.

I’ve been working on Kubernetes since 2014. At that time, I was working on authentication and authorization at Red Hat, and my very first pull request to Kubernetes attempted to add an OAuth server to the Kubernetes API server. It never exited work-in-progress status. I ended up going with a different approach that layered on top of the core Kubernetes API server in a different project (spoiler alert: this is foreshadowing), and I closed it without merging six months later.

Undeterred by that start, I stayed involved, helped build Kubernetes authentication and authorization capabilities, and got involved in the definition and evolution of the core Kubernetes APIs from early beta APIs, like v1beta3 to v1. I got tagged as an API reviewer in 2016 based on those contributions, and was added as an API approver in 2017.

Today, I help lead the API Governance and code organization subprojects for SIG Architecture, and I am a tech lead for SIG Auth.

FM: And when did you get specifically involved in the API Governance project?

JL: Around 2019.

Goals and scope of API Governance

FM: How would you describe the main goals and areas of intervention of the subproject?

The surface area includes all the various APIs Kubernetes has, and there are APIs that people do not always realize are APIs: command-line flags, configuration files, how binaries are run, how they talk to back-end components like the container runtime, and how they persist data. People often think of “the API” as only the REST API… that is the biggest and most obvious one, and the one with the largest audience, but all of these other surfaces are also APIs. Their audiences are narrower, so there is more flexibility there, but they still require consideration.

The goals are to be stable while still enabling innovation. Stability is easy if you never change anything, but that contradicts the goal of evolution and growth. So we balance “be stable” with “allow change”.

FM: Speaking of changes, in terms of ensuring consistency and quality (which is clearly one of the reasons this project exists), what are the specific quality gates in the lifecycle of a Kubernetes change? Does API Governance get involved during the release cycle, prior to it through guidelines, or somewhere in between? At what points do you ensure the intended role is fulfilled?

JL: We have guidelines and conventions, both for APIs in general and for how to change an API. These are living documents that we update as we encounter new scenarios. They are long and dense, so we also support them with involvement at either the design stage or the implementation stage.

Sometimes, due to bandwidth constraints, teams move ahead with design work without feedback from API Review. That’s fine, but it means that when implementation begins, the API review will happen then, and there may be substantial feedback. So we get involved when a new API is created or an existing API is changed, either at design or implementation.

FM: Is this during the Kubernetes Enhancement Proposal (KEP) process? Since KEPs are mandatory for enhancements, I assume part of the work intersects with API Governance?

JL: It can. KEPs vary in how detailed they are. Some include literal API definitions. When they do, we can perform an API review at the design stage. Then implementation becomes a matter of checking fidelity to the design.

Getting involved early is ideal. But some KEPs are conceptual and leave details to the implementation. That’s not wrong; it just means the implementation will be more exploratory. Then API Review gets involved later, possibly recommending structural changes.

There’s a trade-off regardless: detailed design upfront versus iterative discovery during implementation. People and teams work differently, and we’re flexible and happy to consult early or at implementation time.

FM: This reminds me of what Fred Brooks wrote in “The Mythical Man-Month” about conceptual integrity being central to product quality… No matter how you structure the process, there must be a point where someone looks at what is coming and ensures conceptual integrity. Kubernetes uses APIs everywhere – externally and internally – so API Governance is critical to maintaining that integrity. How is this captured?

JL: Yes, the conventions document captures patterns we’ve learned over time: what to do in various situations. We also have automated linters and checks to ensure correctness around patterns like spec/status semantics. These automated tools help catch issues even when humans miss them.

As new scenarios arise – and they do constantly – we think through how to approach them and fold the results back into our documentation and tools. Sometimes it takes a few attempts before we settle on an approach that works well.

FM: Exactly. Each new interaction improves the guidelines.

JL: Right. And sometimes the first approach turns out to be wrong. It may take two or three iterations before we land on something robust.

The impact of Custom Resource Definitions

FM: Is there any particular change, episode, or domain that stands out as especially noteworthy, complex, or interesting in your experience?

JL: The watershed moment was Custom Resources. Prior to that, every API was handcrafted by us and fully reviewed. There were inconsistencies, but we understood and controlled every type and field.

When Custom Resources arrived, anyone could define anything. The first version did not even require a schema. That made it extremely powerful – it enabled change immediately – but it left us playing catch-up on stability and consistency.

When Custom Resources graduated to General Availability (GA), schemas became required, but escape hatches still existed for backward compatibility. Since then, we’ve been working on giving CRD authors validation capabilities comparable to built-ins. Built-in validation rules for CRDs have only just reached GA in the last few releases.

So CRDs opened the “anything is possible” era. Built-in validation rules are the second major milestone: bringing consistency back.

The three major themes have been defining schemas, validating data, and handling pre-existing invalid data. With ratcheting validation (allowing data to improve without breaking existing objects), we can now guide CRD authors toward conventions without breaking the world.

API Governance in context

FM: How does API Governance relate to SIG Architecture and API Machinery?

JL: API Machinery provides the actual code and tools that people build APIs on. They don’t review APIs for storage, networking, scheduling, etc.

SIG Architecture sets the overall system direction and works with API Machinery to ensure the system supports that direction. API Governance works with other SIGs building on that foundation to define conventions and patterns, ensuring consistent use of what API Machinery provides.

FM: Thank you. That clarifies the flow. Going back to release cycles: do release phases – enhancements freeze, code freeze – change your workload? Or is API Governance mostly continuous?

JL: We get involved in two places: design and implementation. Design involvement increases before enhancements freeze; implementation involvement increases before code freeze. However, many efforts span multiple releases, so there is always some design and implementation happening, even for work targeting future releases. Between those intense periods, we often have time to work on long-term design work.

An anti-pattern we see is teams thinking about a large feature for months and then presenting it three weeks before enhancements freeze, saying, “Here is the design, please review.” For big changes with API impact, it’s much better to involve API Governance early.

And there are good times in the cycle for this – between freezes – when people have bandwidth. That’s when long-term review work fits best.

Getting involved

FM: Clearly. Now, regarding team dynamics and new contributors: how can someone get involved in API Governance? What should they focus on?

JL: It’s usually best to follow a specific change rather than trying to learn everything at once. Pick a small API change, perhaps one someone else is making or one you want to make, and observe the full process: design, implementation, review.

High-bandwidth review – live discussion over video – is often very effective. If you’re making or following a change, ask whether there’s a time to go over the design or PR together. Observing those discussions is extremely instructive.

Start with a small change. Then move to a bigger one. Then maybe a new API. That builds understanding of conventions as they are applied in practice.

FM: Excellent. Any final comments, or anything we missed?

JL: Yes… the reason we care so much about compatibility and stability is for our users. It’s easy for contributors to see those requirements as painful obstacles preventing cleanup or requiring tedious work… but users integrated with our system, and we made a promise to them: we want them to trust that we won’t break that contract. So even when it requires more work, moves slower, or involves duplication, we choose stability.

We are not trying to be obstructive; we are trying to make life good for users.

A lot of our questions focus on the future: you want to do something now… how will you evolve it later without breaking it? We assume we will know more in the future, and we want the design to leave room for that.

We also assume we will make mistakes. The question then is: how do we leave ourselves avenues to improve while keeping compatibility promises?

FM: Exactly. Jordan, thank you, I think we’ve covered everything. This has been an insightful view into the API Governance project and its role in the wider Kubernetes project.

JL: Thank you.

Blog: Announcing the Checkpoint/Restore Working Group

Wed, 21 Jan 2026 00:00:00 +0000

The community around Kubernetes includes a number of Special Interest Groups (SIGs) and Working Groups (WGs) facilitating discussions on important topics between interested contributors. Today we would like to announce the new Kubernetes Checkpoint Restore WG focusing on the integration of Checkpoint/Restore functionality into Kubernetes.

Motivation and use cases

There are several high-level scenarios discussed in the working group:

Optimizing resource utilization for interactive workloads, such as Jupyter notebooks and AI chatbots
Accelerating startup of applications with long initialization times, including Java applications and LLM inference services
Using periodic checkpointing to enable fault-tolerance for long-running workloads, such as distributed model training
Providing interruption-aware scheduling with transparent checkpoint/restore, allowing lower-priority Pods to be preempted while preserving the runtime state of applications
Facilitating Pod migration across nodes for load balancing and maintenance, without disrupting workloads.
Enabling forensic checkpointing to investigate and analyze security incidents such as cyberattacks, data breaches, and unauthorized access.

Across these scenarios, the goal is to help facilitate discussions of ideas between the Kubernetes community and the growing Checkpoint/Restore in Userspace (CRIU) ecosystem. The CRIU community includes several projects that support these use cases, including:

CRIU - A tool for checkpointing and restoring running applications and containers
checkpointctl - A tool for in-depth analysis of container checkpoints
criu-coordinator - A tool for coordinated checkpoint/restore of distributed applications with CRIU
checkpoint-restore-operator - A Kubernetes operator for managing checkpoints

More information about the checkpoint/restore integration with Kubernetes is also available here.

Following our presentation about transparent checkpointing at KubeCon EU 2025, we are excited to welcome you to our panel discussion and AI + ML session at KubeCon + CloudNativeCon Europe 2026.

Connect with us

If you are interested in contributing to Kubernetes or CRIU, there are several ways to participate:

Join our meeting every second Thursday at 17:00 UTC via the Zoom link in our meeting notes; recordings of our prior meetings are available here.
Chat with us on the Kubernetes Slack: #wg-checkpoint-restore
Email us at the wg-checkpoint-restore mailing list

Blog: Ingress NGINX Retirement: What You Need to Know

Wed, 12 Nov 2025 12:00:00 -0500

To prioritize the safety and security of the ecosystem, Kubernetes SIG Network and the Security Response Committee are announcing the upcoming retirement of Ingress NGINX. Best-effort maintenance will continue until March 2026. Afterward, there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered. Existing deployments of Ingress NGINX will continue to function and installation artifacts will remain available.

We recommend migrating to one of the many alternatives. Consider migrating to Gateway API, the modern replacement for Ingress. If you must continue using Ingress, many alternative Ingress controllers are listed in the Kubernetes documentation. Continue reading for further information about the history and current state of Ingress NGINX, as well as next steps.

About Ingress NGINX

Ingress is the original user-friendly way to direct network traffic to workloads running on Kubernetes. (Gateway API is a newer way to achieve many of the same goals.) In order for an Ingress to work in your cluster, there must be an Ingress controller running. There are many Ingress controller choices available, which serve the needs of different users and use cases. Some are cloud-provider specific, while others have more general applicability.

Ingress NGINX was an Ingress controller, developed early in the history of the Kubernetes project as an example implementation of the API. It became very popular due to its tremendous flexibility, breadth of features, and independence from any particular cloud or infrastructure provider. Since those days, many other Ingress controllers have been created within the Kubernetes project by community groups, and by cloud native vendors. Ingress NGINX has continued to be one of the most popular, deployed as part of many hosted Kubernetes platforms and within innumerable independent users’ clusters.

History and Challenges

The breadth and flexibility of Ingress NGINX has caused maintenance challenges. Changing expectations about cloud native software have also added complications. What were once considered helpful options have sometimes come to be considered serious security flaws, such as the ability to add arbitrary NGINX configuration directives via the “snippets” annotations. Yesterday’s flexibility has become today’s insurmountable technical debt.

Despite the project’s popularity among users, Ingress NGINX has always struggled with insufficient or barely-sufficient maintainership. For years, the project has had only one or two people doing development work, on their own time, after work hours and on weekends. Last year, the Ingress NGINX maintainers announced their plans to wind down Ingress NGINX and develop a replacement controller together with the Gateway API community. Unfortunately, even that announcement failed to generate additional interest in helping maintain Ingress NGINX or develop InGate to replace it. (InGate development never progressed far enough to create a mature replacement; it will also be retired.)

Current State and Next Steps

Currently, Ingress NGINX is receiving best-effort maintenance. SIG Network and the Security Response Committee have exhausted our efforts to find additional support to make Ingress NGINX sustainable. To prioritize user safety, we must retire the project.

In March 2026, Ingress NGINX maintenance will be halted, and the project will be retired. After that time, there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered. The GitHub repositories will be made read-only and left available for reference.

Existing deployments of Ingress NGINX will not be broken. Existing project artifacts such as Helm charts and container images will remain available.

In most cases, you can check whether you use Ingress NGINX by running kubectl get pods \--all-namespaces \--selector app.kubernetes.io/name=ingress-nginx with cluster administrator permissions.

We would like to thank the Ingress NGINX maintainers for their work in creating and maintaining this project–their dedication remains impressive. This Ingress controller has powered billions of requests in datacenters and homelabs all around the world. In a lot of ways, Kubernetes wouldn’t be where it is without Ingress NGINX, and we are grateful for so many years of incredible effort.

SIG Network and the Security Response Committee recommend that all Ingress NGINX users begin migration to Gateway API or another Ingress controller immediately. Many options are listed in the Kubernetes documentation: Gateway API, Ingress. Additional options may be available from vendors you work with.

Blog: Announcing the 2025 Steering Committee Election Results

Sun, 09 Nov 2025 15:10:00 -0500

The 2025 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2025. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

The Steering Committee oversees the governance of the entire Kubernetes project. With that great power comes great responsibility. You can learn more about the steering committee’s role in their charter.

Thank you to everyone who voted in the election; your participation helps support the community’s continued health and success.

Results

Congratulations to the elected committee members whose two year terms begin immediately (listed in alphabetical order by GitHub handle):

Kat Cosgrove (@katcosgrove), Minimus
Paco Xu (@pacoxu), DaoCloud
Rita Zhang (@ritazh), Microsoft
Maciej Szulik (@soltysh), Defense Unicorns

They join continuing members:

Antonio Ojea (@aojea), Google
Benjamin Elder (@BenTheElder), Google
Sascha Grunert (@saschagrunert), Red Hat

Maciej Szulik and Paco Xu are returning Steering Committee Members.

Big thanks!

Thank you and congratulations on a successful election to this round’s election officers:

Christoph Blecker (@cblecker)
Nina Polshakova (@npolshakova)
Sreeram Venkitesh (@sreeram-venkitesh)

Thanks to the Emeritus Steering Committee Members. Your service is appreciated by the community:

Stephen Augustus (@justaugustus), Bloomberg
Patrick Ohly (@pohly), Intel

And thank you to all the candidates who came forward to run for election.

Get involved with the Steering Committee

This governing body, like all of Kubernetes, is open to all. You can follow along with Steering Committee meeting notes and weigh in by filing an issue or creating a PR against their repo. They have an open meeting on the first Wednesday at 8am PT of every month. They can also be contacted at their public mailing list steering@kubernetes.io.

You can see what the Steering Committee meetings are all about by watching past meetings on the YouTube Playlist.

This post was adapted from one written by the Contributor Comms Subproject. If you want to write stories about the Kubernetes community, learn more about us.

This article was revised in November 2025 to update the information about when the steering committee meets.

Blog: Spotlight on Policy Working Group

Sat, 18 Oct 2025 00:00:00 +0000

(Note: The Policy Working Group has completed its mission and is no longer active. This article reflects its work, accomplishments, and insights into how a working group operates.)

In the complex world of Kubernetes, policies play a crucial role in managing and securing clusters. But have you ever wondered how these policies are developed, implemented, and standardized across the Kubernetes ecosystem? To answer that, let’s take a look back at the work of the Policy Working Group.

The Policy Working Group was dedicated to a critical mission: providing an overall architecture that encompasses both current policy-related implementations and future policy proposals in Kubernetes. Their goal was both ambitious and essential: to develop a universal policy architecture that benefits developers and end-users alike.

Through collaborative methods, this working group strove to bring clarity and consistency to the often complex world of Kubernetes policies. By focusing on both existing implementations and future proposals, they ensured that the policy landscape in Kubernetes remains coherent and accessible as the technology evolves.

This blog post dives deeper into the work of the Policy Working Group, guided by insights from its former co-chairs:

Interviewed by Arujjwal Negi.

These co-chairs explained what the Policy Working Group was all about.

Introduction

Hello, thank you for the time! Let’s start with some introductions, could you tell us a bit about yourself, your role, and how you got involved in Kubernetes?

Jim Bugwadia: My name is Jim Bugwadia, and I am a co-founder and the CEO at Nirmata which provides solutions that automate security and compliance for cloud-native workloads. At Nirmata, we have been working with Kubernetes since it started in 2014. We initially built a Kubernetes policy engine in our commercial platform and later donated it to CNCF as the Kyverno project. I joined the CNCF Kubernetes Policy Working Group to help build and standardize various aspects of policy management for Kubernetes and later became a co-chair.

Andy Suderman: My name is Andy Suderman and I am the CTO of Fairwinds, a managed Kubernetes-as-a-Service provider. I began working with Kubernetes in 2016 building a web conferencing platform. I am an author and/or maintainer of several Kubernetes-related open-source projects such as Goldilocks, Pluto, and Polaris. Polaris is a JSON-schema-based policy engine, which started Fairwinds’ journey into the policy space and my involvement in the Policy Working Group.

Poonam Lamba: My name is Poonam Lamba, and I currently work as a Product Manager for Google Kubernetes Engine (GKE) at Google. My journey with Kubernetes began back in 2017 when I was building an SRE platform for a large enterprise, using a private cloud built on Kubernetes. Intrigued by its potential to revolutionize the way we deployed and managed applications at the time, I dove headfirst into learning everything I could about it. Since then, I’ve had the opportunity to build the policy and compliance products for GKE. I lead and contribute to GKE CIS benchmarks. I am involved with the Gatekeeper project as well as I have contributed to Policy-WG for over 2 years and served as a co-chair for the group.

Responses to the following questions represent an amalgamation of insights from the former co-chairs.

About Working Groups

One thing even I am not aware of is the difference between a working group and a SIG. Can you help us understand what a working group is and how it is different from a SIG?

Unlike SIGs, working groups are temporary and focused on tackling specific, cross-cutting issues or projects that may involve multiple SIGs. Their lifespan is defined, and they disband once they’ve achieved their objective. Generally, working groups don’t own code or have long-term responsibility for managing a particular area of the Kubernetes project.

(To know more about SIGs, visit the list of Special Interest Groups)

You mentioned that Working Groups involve multiple SIGS. What SIGS was the Policy WG closely involved with, and how did you coordinate with them?

The group collaborated closely with Kubernetes SIG Auth throughout our existence, and more recently, the group also worked with SIG Security since its formation. Our collaboration occurred in a few ways. We provided periodic updates during the SIG meetings to keep them informed of our progress and activities. Additionally, we utilize other community forums to maintain open lines of communication and ensured our work aligned with the broader Kubernetes ecosystem. This collaborative approach helped the group stay coordinated with related efforts across the Kubernetes community.

Policy WG

Why was the Policy Working Group created?

To enable a broad set of use cases, we recognize that Kubernetes is powered by a highly declarative, fine-grained, and extensible configuration management system. We’ve observed that a Kubernetes configuration manifest may have different portions that are important to various stakeholders. For example, some parts may be crucial for developers, while others might be of particular interest to security teams or address operational concerns. Given this complexity, we believe that policies governing the usage of these intricate configurations are essential for success with Kubernetes.

Our Policy Working Group was created specifically to research the standardization of policy definitions and related artifacts. We saw a need to bring consistency and clarity to how policies are defined and implemented across the Kubernetes ecosystem, given the diverse requirements and stakeholders involved in Kubernetes deployments.

Can you give me an idea of the work you did in the group?

We worked on several Kubernetes policy-related projects. Our initiatives included:

We worked on a Kubernetes Enhancement Proposal (KEP) for the Kubernetes Policy Reports API. This aims to standardize how policy reports are generated and consumed within the Kubernetes ecosystem.
We conducted a CNCF survey to better understand policy usage in the Kubernetes space. This helped gauge the practices and needs across the community at the time.
We wrote a paper that will guide users in achieving PCI-DSS compliance for containers. This is intended to help organizations meet important security standards in their Kubernetes environments.
We also worked on a paper highlighting how shifting security down can benefit organizations. This focuses on the advantages of implementing security measures earlier in the development and deployment process.

Can you tell us what were the main objectives of the Policy Working Group and some of your key accomplishments?

The charter of the Policy WG was to help standardize policy management for Kubernetes and educate the community on best practices.

To accomplish this we updated the Kubernetes documentation (Policies | Kubernetes), produced several whitepapers (Kubernetes Policy Management, Kubernetes GRC), and created the Policy Reports API (API reference) which standardizes reporting across various tools. Several popular tools such as Falco, Trivy, Kyverno, kube-bench, and others support the Policy Report API. A major milestone for the Policy WG was promoting the Policy Reports API to a SIG-level API or finding it a stable home.

Beyond that, as ValidatingAdmissionPolicy and MutatingAdmissionPolicy approached GA in Kubernetes, a key goal of the WG was to guide and educate the community on the tradeoffs and appropriate usage patterns for these built-in API objects and other CNCF policy management solutions like OPA/Gatekeeper and Kyverno.

Challenges

What were some of the major challenges that the Policy Working Group worked on?

During our work in the Policy Working Group, we encountered several challenges:

One of the main issues we faced was finding time to consistently contribute. Given that many of us have other professional commitments, it can be difficult to dedicate regular time to the working group’s initiatives.
Another challenge we experienced was related to our consensus-driven model. While this approach ensures that all voices are heard, it can sometimes lead to slower decision-making processes. We valued thorough discussion and agreement, but this can occasionally delay progress on our projects.
We’ve also encountered occasional differences of opinion among group members. These situations require careful navigation to ensure that we maintain a collaborative and productive environment while addressing diverse viewpoints.
Lastly, we’ve noticed that newcomers to the group may find it difficult to contribute effectively without consistent attendance at our meetings. The complex nature of our work often requires ongoing context, which can be challenging for those who aren’t able to participate regularly.

Can you tell me more about those challenges? How did you discover each one? What has the impact been? What were some strategies you used to address them?

There are no easy answers, but having more contributors and maintainers greatly helps! Overall the CNCF community is great to work with and is very welcoming to beginners. So, if folks out there are hesitating to get involved, I highly encourage them to attend a WG or SIG meeting and just listen in.

It often takes a few meetings to fully understand the discussions, so don’t feel discouraged if you don’t grasp everything right away. We made a point to emphasize this and encouraged new members to review documentation as a starting point for getting involved.

Additionally, differences of opinion were valued and encouraged within the Policy-WG. We adhered to the CNCF core values and resolve disagreements by maintaining respect for one another. We also strove to timebox our decisions and assign clear responsibilities to keep things moving forward.

This is where our discussion about the Policy Working Group ends. The working group, and especially the people who took part in this article, hope this gave you some insights into the group’s aims and workings. You can get more info about Working Groups here.

Blog: Spotlight on the Kubernetes Steering Committee

Mon, 22 Sep 2025 00:00:00 +0000

This interview was conducted in August 2024, and due to the dynamic nature of the Steering Committee membership and election process it might not represent the actual composition accurately. The topics covered are, however, overwhelmingly relevant to understand its scope of work. As we approach the Steering Committee elections, it provides useful insights into the workings of the Committee.

The Kubernetes Steering Committee is the backbone of the Kubernetes project, ensuring that its vibrant community and governance structures operate smoothly and effectively. While the technical brilliance of Kubernetes is often spotlighted through its Special Interest Groups (SIGs) and Working Groups (WGs), the unsung heroes quietly steering the ship are the members of the Steering Committee. They tackle complex organizational challenges, empower contributors, and foster the thriving open source ecosystem that Kubernetes is celebrated for.

But what does it really take to lead one of the world’s largest open source communities? What are the hidden challenges, and what drives these individuals to dedicate their time and effort to such an impactful role? In this exclusive conversation, we sit down with current Steering Committee (SC) members — Ben, Nabarun, Paco, Patrick, and Maciej — to uncover the rewarding, and sometimes demanding, realities of steering Kubernetes. From their personal journeys and motivations to the committee’s vital responsibilities and future outlook, this Spotlight offers a rare behind-the-scenes glimpse into the people who keep Kubernetes on course.

Introductions

Sandipan: Can you tell us a little bit about yourself?

Ben: Hi, I’m Benjamin Elder, also known as BenTheElder. I started in Kubernetes as a Google Summer of Code student in 2015 and have been working at Google in the space since 2017. I have contributed a lot to many areas but especially build, CI, test tooling, etc. My favorite project so far was building KIND. I have been on the release team, a chair of SIG Testing, and currently a tech lead of SIG Testing and SIG K8s Infra.

Nabarun: Hi, I am Nabarun from India. I have been working on Kubernetes since 2019. I have been contributing across multiple areas in Kubernetes: SIG ContribEx (where I am also a chair), API Machinery, Architecture, and SIG Release, where I contributed to several releases including being the Release Team Lead of Kubernetes 1.21.

Paco: I am Paco from China. I worked as an open source team lead in DaoCloud, Shanghai. In the community, I participate mainly in kubeadm, SIG Node and SIG Testing. Besides, I helped in KCD China and was co-chair of the recent KubeCon+CloudNativeCon China 2024 in Hong Kong.

Patrick: Hello! I’m Patrick. I’ve contributed to Kubernetes since 2018. I started in SIG Storage and then got involved in more and more areas. Nowadays, I am a SIG Testing tech lead, logging infrastructure maintainer, organizer of the Structured Logging and Device Management working groups, contributor in SIG Scheduling, and of course member of the Steering Committee. My main focus area currently is Dynamic Resource Allocation (DRA), a new API for accelerators.

Maciej: Hey, my name is Maciej and I’ve been working on Kubernetes since late 2014 in various areas, including controllers, apiserver and kubectl. Aside from being part of the Steering Committee, I’m also helping guide SIG CLI, SIG Apps and WG Batch.

About the Steering Committee

Sandipan: What does Steering do?

Ben: The charter is the definitive answer, but I see Steering as helping resolve Kubernetes-organization-level “people problems” (as opposed to technical problems), such as clarifying project governance and liaising with the Cloud Native Computing Foundation (for example, to request additional resources and support) and other CNCF projects.

Maciej: Our charter nicely describes all the responsibilities. In short, we make sure the project runs smoothly by supporting our maintainers and contributors in their daily tasks.

Patrick: Ideally, we don’t do anything 😀 All of the day-to-day business has been delegated to SIGs and WGs. Steering gets involved when something pops up where it isn’t obvious who should handle it or when conflicts need to be resolved.

**Sandipan: And how is Steering different from SIGs?

Ben: From a governance perspective: Steering delegates all of the ownership of subprojects to the SIGs and/or committees (Security Response, Code Of Conduct, etc.). They’re very different. The SIGs own pieces of the project, and Steering handles some of the overarching people and policy issues. You’ll find all of the software development, releasing, communications and documentation work happening in the SIGs and committees.

Maciej: SIGs or WGs are primarily concerned with the technical direction of a particular area in Kubernetes. Steering, on the other hand, is primarily concerned with ensuring all the SIGs, WGs, and most importantly maintainers have everything they need to run the project smoothly. This includes anything from ensuring financing of our CI systems, through governance structures and policies all the way to supporting individual maintainers in various inquiries.

**Sandipan: You’ve mentioned projects, could you give us an example of a project Steering has worked on recently?

Ben: We’ve been discussing the logistics to sync a better definition of the project’s official maintainers to the CNCF, which are used, for example, to vote for the Technical Oversight Committee (TOC). Currently that list is the Steering Committee, with SIG Contributor Experience and Infra + Release leads having access to the CNCF service desk. This isn’t well standardized yet across CNCF projects but I think it’s important.

Maciej: For the past year I’ve been sitting on the SC, I believe the majority of tasks we’ve been involved in were around providing letters supporting visa applications. Also, like every year, we’ve been helping all the SIGs and WGs with their annual reports.

Patrick: Apparently it has been a quiet year since Maciej and I joined the Steering Committee at the end of 2023. That’s exactly how it should be.

Sandipan: Do you have any examples of projects that came to Steering, which you then redirected to SIGs?

Ben: We often get requests for test/build related resources that we redirect to SIG K8s Infra + SIG Testing, or more specifically about releasing for subprojects that we redirect to SIG K8s Infra / SIG Release.

The road to the Steering Committee

Sandipan: What motivated you to be part of the Steering Committee? What has your journey been like?

Ben: I had a few people reach out and prompt me to run, but I was motivated by my passion for this community and the project. I think we have something really special going here and I care deeply about the ongoing success. I’ve been involved in this space my whole career and while there’s always rough edges, this community has been really supportive and I hope we can keep it that way.

Paco: After my journey to the Kubernetes Contributor Summit EU 2023, I met and chatted with many maintainers and members there, and attended the steering AMA there for the first time as there hadn’t been a contributor summit in China since 2019, and I started to connect with contributors in China to make it later the year. Through conversations at KCS EU and with local contributors, I realized that it is quite important to make it easy to start a contributor journey for APAC contributors and want to attract more contributors to the community. Then, I was elected just after the KCS CN 2023.

Patrick: I had done a lot of technical work, of which some affects and (hopefully) benefits all contributors to Kubernetes (linting and testing) and users (better log output). I saw joining the Steering Committee as an opportunity to help also with the organizational aspects of running a big open source project.

Maciej: I’ve been going through the idea of running for SC for a while now. My biggest drive was conversations with various members of our community. Eventually last year, I decided to follow their advice, and got elected :-)

Sandipan: What is your favorite part of being part of Steering?

Ben: When we get to help contributors directly. For example, sometimes extensive contributors reach out for an official letter from Steering explaining their contribution and its value for visa support. When we get to just purely help out Kubernetes contributors, that’s my favorite part.

Patrick: It’s a good place to learn more about how the project is actually run, directly from the other great people who are doing it.

Maciej: The same thing as with the project — it’s always the people that surround us, that give us opportunities to collaborate and create something interesting and exciting.

Sandipan: What do you think is most challenging about being part of Steering?

Ben: I think we’ve all spent a lot of time grappling with the sustainability issues in the project and not having a single great answer to solve them. A lot of people are working on these problems but we have limited time and resources. We’ve officially delegated most of this (for example, to SIGs Contributor Experience and K8s Infra), but I think we all still consider it very important and deserving of more time and energy, yet we only have so much and the answers are not obvious. The balancing act is hard.

Paco: Sustainability of contributors and maintainers is one of the most challenging aspects to me. I am constantly advocating for OSS users and employers to join the community. Community is a place that developers can learn from each other, discuss issues they encounter, and share their experience or solutions. Ensuring everyone in the community to feel supported and valued is crucial for the long-term health of the project.

Patrick: There is documentation about how things are done, but it’s not exhaustive. There are parts which are intentionally not documented, perhaps because they cannot be public, change too often, or simply need to be handled on a case-by-case basis. Luckily we have overlapping terms, so there is an opportunity to learn from more experienced members. We also have a list of former members and they are happy to respond to questions if needed.

Maciej: The unknown unknowns :-) After I got elected to SC I’ve tried to talk to various folks from current and past SC. The biggest challenge that came from all those discussions is that no matter how hard you try and what you prepare for, there will always be something new that none of the previous SC had to deal with, so far.

Sandipan: For folks who might want to run for Steering in the future, what are the most important things you think they should know?

Ben: A lot of what Steering does is “interrupt driven”… something comes up and needs resolution – Make sure you’re committed and prepared to set aside the time. Otherwise I hope you think calmly about issues and listen to our community with empathy.

Paco: This is a quote from the survey to all previous SCs: we should make sure that “everyone’s voice was heard and respected” from Clayton. For the community, the most important part is about the people here.

Maciej: Once you decide to run and get elected, make sure to reserve a constant time per week for your Steering duties. There will be times when nothing will need to happen, and others when you will overflow so reserving that timeframe will ensure consistency in you being a Steering member.

Conclusion

Behind every successful open source project is a group of dedicated people who ensure things run smoothly, and the Kubernetes Steering Committee does just that. They work quietly but effectively, tackling challenges, supporting contributors, and ensuring the community remains inclusive and vibrant.

What makes them stand out is their focus on people — empowering contributors, resolving governance issues, and creating an environment where innovation can thrive. It’s not always easy, but as they’ve shared, it’s deeply rewarding.

Whether you’re a long-time contributor or thinking about getting involved, the Kubernetes community is open to you. At its heart, Kubernetes is about more than just technology — it’s about the people who make it all happen. There’s always room for one more voice to help shape the future.

Blog: Post-Quantum Cryptography in Kubernetes

Fri, 18 Jul 2025 00:00:00 +0000

The world of cryptography is on the cusp of a major shift with the advent of quantum computing. While powerful quantum computers are still largely theoretical for many applications, their potential to break current cryptographic standards is a serious concern, especially for long-lived systems. This is where Post-Quantum Cryptography (PQC) comes in. In this article, I'll dive into what PQC means for TLS and, more specifically, for the Kubernetes ecosystem. I’ll explain what the (suprising) state of PQC in Kubernetes is and what the implications are for current and future clusters.

What is Post-Quantum Cryptography

Post-Quantum Cryptography refers to cryptographic algorithms that are thought to be secure against attacks by both classical and quantum computers. The primary concern is that quantum computers, using algorithms like Shor's Algorithm, could efficiently break widely used public-key cryptosystems such as RSA and Elliptic Curve Cryptography (ECC), which underpin much of today's secure communication, including TLS. The industry is actively working on standardizing and adopting PQC algorithms. One of the first to be standardized by NIST is the Module-Lattice Key Encapsulation Mechanism (ML-KEM), formerly known as Kyber, and now standardized as FIPS-203 (PDF download).

It is difficult to predict when quantum computers will be able to break classical algorithms. However, it is clear that we need to start migrating to PQC algorithms now, as the next section shows. To get a feeling for the predicted timeline we can look at a NIST report covering the transition to post-quantum cryptography standards. It declares that system with classical crypto should be deprecated after 2030 and disallowed after 2035.

Key exchange vs. digital signatures: different needs, different timelines

In TLS, there are two main cryptographic operations we need to secure:

Key Exchange: This is how the client and server agree on a shared secret to encrypt their communication. If an attacker records encrypted traffic today, they could decrypt it in the future, if they gain access to a quantum computer capable of breaking the key exchange. This makes migrating KEMs to PQC an immediate priority.

Digital Signatures: These are primarily used to authenticate the server (and sometimes the client) via certificates. The authenticity of a server is verified at the time of connection. While important, the risk of an attack today is much lower, because the decision of trusting a server cannot be abused after the fact. Additionally, current PQC signature schemes often come with significant computational overhead and larger key/signature sizes compared to their classical counterparts.

Another significant hurdle in the migration to PQ certificates is the upgrade of root certificates. These certificates have long validity periods and are installed in many devices and operating systems as trust anchors.

Given these differences, the focus for immediate PQC adoption in TLS has been on hybrid key exchange mechanisms. These combine a classical algorithm (such as Elliptic Curve Diffie-Hellman Ephemeral (ECDHE)) with a PQC algorithm (such as ML-KEM). The resulting shared secret is secure as long as at least one of the component algorithms remains unbroken. The X25519MLKEM768 hybrid scheme is the most widely supported one.

State of PQC key exchange mechanisms (KEMs) today

Support for PQC KEMs is rapidly improving across the ecosystem.

Go: The Go standard library's crypto/tls package introduced support for X25519MLKEM768 in version 1.24 (released February 2025). Crucially, it's enabled by default when there is no explicit configuration, i.e., Config.CurvePreferences is nil.

Browsers & OpenSSL: Major browsers like Chrome (version 131, November 2024) and Firefox (version 135, February 2025), as well as OpenSSL (version 3.5.0, April 2025), have also added support for the ML-KEM based hybrid scheme.

Apple is also rolling out support for X25519MLKEM768 in version 26 of their operating systems. Given the proliferation of Apple devices, this will have a significant impact on the global PQC adoption.

For a more detailed overview of the state of PQC in the wider industry, see this blog post by Cloudflare.

Post-quantum KEMs in Kubernetes: an unexpected arrival

So, what does this mean for Kubernetes? Kubernetes components, including the API server and kubelet, are built with Go.

As of Kubernetes v1.33, released in April 2025, the project uses Go 1.24. A quick check of the Kubernetes codebase reveals that Config.CurvePreferences is not explicitly set. This leads to a fascinating conclusion: Kubernetes v1.33, by virtue of using Go 1.24, supports hybrid post-quantum X25519MLKEM768 for TLS connections by default!

You can test this yourself. If you set up a Minikube cluster running Kubernetes v1.33.0, you can connect to the API server using a recent OpenSSL client:

$ minikube start --kubernetes-version=v1.33.0
$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:<PORT>
$ kubectl config view --minify --raw -o jsonpath=\'{.clusters[0].cluster.certificate-authority-data}\' | base64 -d > ca.crt
$ openssl version
OpenSSL 3.5.0 8 Apr 2025 (Library: OpenSSL 3.5.0 8 Apr 2025)
$ echo -n "Q" | openssl s_client -connect 127.0.0.1:<PORT> -CAfile ca.crt
[...]
Negotiated TLS1.3 group: X25519MLKEM768
[...]
DONE

Lo and behold, the negotiated group is X25519MLKEM768! This is a significant step towards making Kubernetes quantum-safe, seemingly without a major announcement or dedicated KEP (Kubernetes Enhancement Proposal).

The Go version mismatch pitfall

An interesting wrinkle emerged with Go versions 1.23 and 1.24. Go 1.23 included experimental support for a draft version of ML-KEM, identified as X25519Kyber768Draft00. This was also enabled by default if Config.CurvePreferences was nil. Kubernetes v1.32 used Go 1.23. However, Go 1.24 removed the draft support and replaced it with the standardized version X25519MLKEM768.

What happens if a client and server are using mismatched Go versions (one on 1.23, the other on 1.24)? They won't have a common PQC KEM to negotiate, and the handshake will fall back to classical ECC curves (e.g., X25519). How could this happen in practice?

Consider a scenario:

A Kubernetes cluster is running v1.32 (using Go 1.23 and thus X25519Kyber768Draft00). A developer upgrades their kubectl to v1.33, compiled with Go 1.24, only supporting X25519MLKEM768. Now, when kubectl communicates with the v1.32 API server, they no longer share a common PQC algorithm. The connection will downgrade to classical cryptography, silently losing the PQC protection that has been in place. This highlights the importance of understanding the implications of Go version upgrades, and the details of the TLS stack.

Limitations: packet size

One practical consideration with ML-KEM is the size of its public keys with encoded key sizes of around 1.2 kilobytes for ML-KEM-768. This can cause the initial TLS ClientHello message not to fit inside a single TCP/IP packet, given the typical networking constraints (most commonly, the standard Ethernet frame size limit of 1500 bytes). Some TLS libraries or network appliances might not handle this gracefully, assuming the Client Hello always fits in one packet. This issue has been observed in some Kubernetes-related projects and networking components, potentially leading to connection failures when PQC KEMs are used. More details can be found at tldr.fail.

State of Post-Quantum Signatures

While KEMs are seeing broader adoption, PQC digital signatures are further behind in terms of widespread integration into standard toolchains. NIST has published standards for PQC signatures, such as ML-DSA (FIPS-204) and SLH-DSA (FIPS-205). However, implementing these in a way that's broadly usable (e.g., for PQC Certificate Authorities) presents challenges:

Larger Keys and Signatures: PQC signature schemes often have significantly larger public keys and signature sizes compared to classical algorithms like Ed25519 or RSA. For instance, Dilithium2 keys can be 30 times larger than Ed25519 keys, and certificates can be 12 times larger.

Performance: Signing and verification operations can be substantially slower. While some algorithms are on par with classical algorithms, others may have a much higher overhead, sometimes on the order of 10x to 1000x worse performance. To improve this situation, NIST is running a second round of standardization for PQC signatures.

Toolchain Support: Mainstream TLS libraries and CA software do not yet have mature, built-in support for these new signature algorithms. The Go team, for example, has indicated that ML-DSA support is a high priority, but the soonest it might appear in the standard library is Go 1.26 (as of May 2025).

Cloudflare's CIRCL (Cloudflare Interoperable Reusable Cryptographic Library) library implements some PQC signature schemes like variants of Dilithium, and they maintain a fork of Go (cfgo) that integrates CIRCL. Using cfgo, it's possible to experiment with generating certificates signed with PQC algorithms like Ed25519-Dilithium2. However, this requires using a custom Go toolchain and is not yet part of the mainstream Kubernetes or Go distributions.

Conclusion

The journey to a post-quantum secure Kubernetes is underway, and perhaps further along than many realize, thanks to the proactive adoption of ML-KEM in Go. With Kubernetes v1.33, users are already benefiting from hybrid post-quantum key exchange in many TLS connections by default.

However, awareness of potential pitfalls, such as Go version mismatches leading to downgrades and issues with Client Hello packet sizes, is crucial. While PQC for KEMs is becoming a reality, PQC for digital signatures and certificate hierarchies is still in earlier stages of development and adoption for mainstream use. As Kubernetes maintainers and contributors, staying informed about these developments will be key to ensuring the long-term security of the platform.

Blog: Changes to Kubernetes Slack

Mon, 16 Jun 2025 00:00:00 +0000

UPDATE: We’ve received notice from Salesforce that our Slack workspace WILL NOT BE DOWNGRADED on June 20th. Stand by for more details, but for now, there is no urgency to back up private channels or direct messages.

~~Kubernetes Slack will lose its special status and will be changing into a standard free Slack on June 20, 2025~~. Sometime later this year, our community may move to a new platform. If you are responsible for a channel or private channel, or a member of a User Group, you will need to take some actions as soon as you can.

For the last decade, Slack has supported our project with a free customized enterprise account. They have let us know that they can no longer do so, particularly since our Slack is one of the largest and more active ones on the platform. As such, they will be downgrading it to a standard free Slack while we decide on, and implement, other options.

On Friday, June 20, we will be subject to the feature limitations of free Slack. The primary ones which will affect us will be only retaining 90 days of history, and having to disable several apps and workflows which we are currently using. The Slack Admin team will do their best to manage these limitations.

Responsible channel owners, members of private channels, and members of User Groups should take some actions to prepare for the upgrade and preserve information as soon as possible.

The CNCF projects staff have proposed that our community look at migrating to Discord. Because of existing issues where we have been pushing the limits of Slack, they have already explored what a Kubernetes Discord would look like. Discord would allow us to implement new tools and integrations which would help the community, such as GitHub group membership synchronization. The Steering Committee will discuss and decide on our future platform.

Please see our FAQ, and check the kubernetes-dev mailing list and the #announcements channel for further news. If you have specific feedback on our Slack status join the discussion on GitHub.

Blog: Spotlight on SIG Apps

Wed, 12 Mar 2025 00:00:00 +0000

In our ongoing SIG Spotlight series, we dive into the heart of the Kubernetes project by talking to the leaders of its various Special Interest Groups (SIGs). This time, we focus on SIG Apps, the group responsible for everything related to developing, deploying, and operating applications on Kubernetes. Sandipan Panda (DevZero) had the opportunity to interview Maciej Szulik (Defense Unicorns) and Janet Kuo (Google), the chairs and tech leads of SIG Apps. They shared their experiences, challenges, and visions for the future of application management within the Kubernetes ecosystem.

Introductions

Sandipan: Hello, could you start by telling us a bit about yourself, your role, and your journey within the Kubernetes community that led to your current roles in SIG Apps?

Maciej: Hey, my name is Maciej, and I’m one of the leads for SIG Apps. Aside from this role, you can also find me helping SIG CLI and also being one of the Steering Committee members. I’ve been contributing to Kubernetes since late 2014 in various areas, including controllers, apiserver, and kubectl.

Janet: Certainly! I’m Janet, a Staff Software Engineer at Google, and I’ve been deeply involved with the Kubernetes project since its early days, even before the 1.0 launch in 2015. It’s been an amazing journey!

My current role within the Kubernetes community is one of the chairs and tech leads of SIG Apps. My journey with SIG Apps started organically. I started with building the Deployment API and adding rolling update functionalities. I naturally gravitated towards SIG Apps and became increasingly involved. Over time, I took on more responsibilities, culminating in my current leadership roles.

About SIG Apps

All following answers were jointly provided by Maciej and Janet.

Sandipan: For those unfamiliar, could you provide an overview of SIG Apps’ mission and objectives? What key problems does it aim to solve within the Kubernetes ecosystem?

As described in our charter, we cover a broad area related to developing, deploying, and operating applications on Kubernetes. That, in short, means we’re open to each and everyone showing up at our bi-weekly meetings and discussing the ups and downs of writing and deploying various applications on Kubernetes.

Sandipan: What are some of the most significant projects or initiatives currently being undertaken by SIG Apps?

At this point in time, the main factors driving the development of our controllers are the challenges coming from running various AI-related workloads. It’s worth giving credit here to two working groups we’ve sponsored over the past years:

The Batch Working Group, which is looking at running HPC, AI/ML, and data analytics jobs on top of Kubernetes.
The Serving Working Group, which is focusing on hardware-accelerated AI/ML inference.

Best practices and challenges

Sandipan: SIG Apps plays a crucial role in developing application management best practices for Kubernetes. Can you share some of these best practices and how they help improve application lifecycle management?

Implementing health checks and readiness probes ensures that your applications are healthy and ready to serve traffic, leading to improved reliability and uptime. The above, combined with comprehensive logging, monitoring, and tracing solutions, will provide insights into your application’s behavior, enabling you to identify and resolve issues quickly.
Auto-scale your application based on resource utilization or custom metrics, optimizing resource usage and ensuring your application can handle varying loads.
Use Deployment for stateless applications, StatefulSet for stateful applications, Job and CronJob for batch workloads, and DaemonSet for running a daemon on each node. Use Operators and CRDs to extend the Kubernetes API to automate the deployment, management, and lifecycle of complex applications, making them easier to operate and reducing manual intervention.

Sandipan: What are some of the common challenges SIG Apps faces, and how do you address them?

The biggest challenge we’re facing all the time is the need to reject a lot of features, ideas, and improvements. This requires a lot of discipline and patience to be able to explain the reasons behind those decisions.

Sandipan: How has the evolution of Kubernetes influenced the work of SIG Apps? Are there any recent changes or upcoming features in Kubernetes that you find particularly relevant or beneficial for SIG Apps?

The main benefit for both us and the whole community around SIG Apps is the ability to extend kubernetes with Custom Resource Definitions and the fact that users can build their own custom controllers leveraging the built-in ones to achieve whatever sophisticated use cases they might have and we, as the core maintainers, haven’t considered or weren’t able to efficiently resolve inside Kubernetes.

Contributing to SIG Apps

Sandipan: What opportunities are available for new contributors who want to get involved with SIG Apps, and what advice would you give them?

We get the question, “What good first issue might you recommend we start with?” a lot :-) But unfortunately, there’s no easy answer to it. We always tell everyone that the best option to start contributing to core controllers is to find one you are willing to spend some time with. Read through the code, then try running unit tests and integration tests focusing on that controller. Once you grasp the general idea, try breaking it and the tests again to verify your breakage. Once you start feeling confident you understand that particular controller, you may want to search through open issues affecting that controller and either provide suggestions, explaining the problem users have, or maybe attempt your first fix.

Like we said, there are no shortcuts on that road; you need to spend the time with the codebase to understand all the edge cases we’ve slowly built up to get to the point where we are. Once you’re successful with one controller, you’ll need to repeat that same process with others all over again.

Sandipan: How does SIG Apps gather feedback from the community, and how is this feedback integrated into your work?

We always encourage everyone to show up and present their problems and solutions during our bi-weekly meetings. As long as you’re solving an interesting problem on top of Kubernetes and you can provide valuable feedback about any of the core controllers, we’re always happy to hear from everyone.

Looking ahead

Sandipan: Looking ahead, what are the key focus areas or upcoming trends in application management within Kubernetes that SIG Apps is excited about? How is the SIG adapting to these trends?

Definitely the current AI hype is the major driving factor; as mentioned above, we have two working groups, each covering a different aspect of it.

Sandipan: What are some of your favorite things about this SIG?

Without a doubt, the people that participate in our meetings and on Slack, who tirelessly help triage issues, pull requests and invest a lot of their time (very frequently their private time) into making kubernetes great!

SIG Apps is an essential part of the Kubernetes community, helping to shape how applications are deployed and managed at scale. From its work on improving Kubernetes’ workload APIs to driving innovation in AI/ML application management, SIG Apps is continually adapting to meet the needs of modern application developers and operators. Whether you’re a new contributor or an experienced developer, there’s always an opportunity to get involved and make an impact.

If you’re interested in learning more or contributing to SIG Apps, be sure to check out their SIG README and join their bi-weekly meetings.

Blog: Spotlight on SIG etcd

Tue, 04 Mar 2025 00:00:00 +0000

In this SIG etcd spotlight we talked with James Blair, Marek Siarkowicz, Wenjia Zhang, and Benjamin Wang to learn a bit more about this Kubernetes Special Interest Group.

Introducing SIG etcd

Frederico: Hello, thank you for the time! Let’s start with some introductions, could you tell us a bit about yourself, your role and how you got involved in Kubernetes.

Benjamin: Hello, I am Benjamin. I am a SIG etcd Tech Lead and one of the etcd maintainers. I work for VMware, which is part of the Broadcom group. I got involved in Kubernetes & etcd & CSI (Container Storage Interface) because of work and also a big passion for open source. I have been working on Kubernetes & etcd (and also CSI) since 2020.

James: Hey team, I’m James, a co-chair for SIG etcd and etcd maintainer. I work at Red Hat as a Specialist Architect helping people adopt cloud native technology. I got involved with the Kubernetes ecosystem in 2019. Around the end of 2022 I noticed how the etcd community and project needed help so started contributing as often as I could. There is a saying in our community that “you come for the technology, and stay for the people”: for me this is absolutely real, it’s been a wonderful journey so far and I’m excited to support our community moving forward.

Marek: Hey everyone, I’m Marek, the SIG etcd lead. At Google, I lead the GKE etcd team, ensuring a stable and reliable experience for all GKE users. My Kubernetes journey began with SIG Instrumentation, where I created and led the Kubernetes Structured Logging effort.
I’m still the main project lead for Kubernetes Metrics Server, providing crucial signals for autoscaling in Kubernetes. I started working on etcd 3 years ago, right around the 3.5 release. We faced some challenges, but I’m thrilled to see etcd now the most scalable and reliable it’s ever been, with the highest contribution numbers in the project’s history. I’m passionate about distributed systems, extreme programming, and testing.

Wenjia: Hi there, my name is Wenjia, I am the co-chair of SIG etcd and one of the etcd maintainers. I work at Google as an Engineering Manager, working on GKE (Google Kubernetes Engine) and GDC (Google Distributed Cloud). I have been working in the area of open source Kubernetes and etcd since the Kubernetes v1.10 and etcd v3.1 releases. I got involved in Kubernetes because of my job, but what keeps me in the space is the charm of the container orchestration technology, and more importantly, the awesome open source community.

Becoming a Kubernetes Special Interest Group (SIG)

Frederico: Excellent, thank you. I’d like to start with the origin of the SIG itself: SIG etcd is a very recent SIG, could you quickly go through the history and reasons behind its creation?

Marek: Absolutely! SIG etcd was formed because etcd is a critical component of Kubernetes, serving as its data store. However, etcd was facing challenges like maintainer turnover and reliability issues. Creating a dedicated SIG allowed us to focus on addressing these problems, improving development and maintenance processes, and ensuring etcd evolves in sync with the cloud-native landscape.

Frederico: And has becoming a SIG worked out as expected? Better yet, are the motivations you just described being addressed, and to what extent?

Marek: It’s been a positive change overall. Becoming a SIG has brought more structure and transparency to etcd’s development. We’ve adopted Kubernetes processes like KEPs (Kubernetes Enhancement Proposals and PRRs (Production Readiness Reviews, which has improved our feature development and release cycle.

Frederico: On top of those, what would you single out as the major benefit that has resulted from becoming a SIG?

Marek: The biggest benefits for me was adopting Kubernetes testing infrastructure, tools like Prow and TestGrid. For large projects like etcd there is just no comparison to the default GitHub tooling. Having known, easy to use, clear tools is a major boost to the etcd as it makes it much easier for Kubernetes contributors to also help etcd.

Wenjia: Totally agree, while challenges remain, the SIG structure provides a solid foundation for addressing them and ensuring etcd’s continued success as a critical component of the Kubernetes ecosystem.

The positive impact on the community is another crucial aspect of SIG etcd’s success that I’d like to highlight. The Kubernetes SIG structure has created a welcoming environment for etcd contributors, leading to increased participation from the broader Kubernetes community. We have had greater collaboration with other SIGs like SIG API Machinery, SIG Scalability, SIG Testing, SIG Cluster Lifecycle, etc.

This collaboration helps ensure etcd’s development aligns with the needs of the wider Kubernetes ecosystem. The formation of the etcd Operator Working Group under the joint effort between SIG etcd and SIG Cluster Lifecycle exemplifies this successful collaboration, demonstrating a shared commitment to improving etcd’s operational aspects within Kubernetes.

Frederico: Since you mentioned collaboration, have you seen changes in terms of contributors and community involvement in recent months?

James: Yes – as showing in our unique PR author data we recently hit an all time high in March and are trending in a positive direction:

Additionally, looking at our overall contributions across all etcd project repositories we are also observing a positive trend showing a resurgence in etcd project activity:

The road ahead

Frederico: That’s quite telling, thank you. In terms of the near future, what are the current priorities for SIG etcd?

Marek: Reliability is always top of mind -– we need to make sure etcd is rock-solid. We’re also working on making etcd easier to use and manage for operators. And we have our sights set on making etcd a viable standalone solution for infrastructure management, not just for Kubernetes. Oh, and of course, scaling -– we need to ensure etcd can handle the growing demands of the cloud-native world.

Benjamin: I agree that reliability should always be our top guiding principle. We need to ensure not only correctness but also compatibility. Additionally, we should continuously strive to improve the understandability and maintainability of etcd. Our focus should be on addressing the pain points that the community cares about the most.

Frederico: Are there any specific SIGs that you work closely with?

Marek: SIG API Machinery, for sure – they own the structure of the data etcd stores, so we’re constantly working together. And SIG Cluster Lifecycle – etcd is a key part of Kubernetes clusters, so we collaborate on the newly created etcd operator Working group.

Wenjia: Other than SIG API Machinery and SIG Cluster Lifecycle that Marek mentioned above, SIG Scalability and SIG Testing is another group that we work closely with.

Frederico: In a more general sense, how would you list the key challenges for SIG etcd in the evolving cloud native landscape?

Marek: Well, reliability is always a challenge when you’re dealing with critical data. The cloud-native world is evolving so fast that scaling to meet those demands is a constant effort.

Getting involved

Frederico: We’re almost at the end of our conversation, but for those interested in in etcd, how can they get involved?

Marek: We’d love to have them! The best way to start is to join our SIG etcd meetings, follow discussions on the etcd-dev mailing list, and check out our GitHub issues. We’re always looking for people to review proposals, test code, and contribute to documentation.

Wenjia: I love this question 😀 . There are numerous ways for people interested in contributing to SIG etcd to get involved and make a difference. Here are some key areas where you can help:

Code Contributions:

Bug Fixes: Tackle existing issues in the etcd codebase. Start with issues labeled “good first issue” or “help wanted” to find tasks that are suitable for newcomers.
Feature Development: Contribute to the development of new features and enhancements. Check the etcd roadmap and discussions to see what’s being planned and where your skills might fit in.
Testing and Code Reviews: Help ensure the quality of etcd by writing tests, reviewing code changes, and providing feedback.
Documentation: Improve etcd’s documentation by adding new content, clarifying existing information, or fixing errors. Clear and comprehensive documentation is essential for users and contributors.
Community Support: Answer questions on forums, mailing lists, or Slack channels. Helping others understand and use etcd is a valuable contribution.

Getting Started:

Join the community: Start by joining the etcd community on Slack, attending SIG meetings, and following the mailing lists. This will help you get familiar with the project, its processes, and the people involved.
Find a mentor: If you’re new to open source or etcd, consider finding a mentor who can guide you and provide support. Stay tuned! Our first cohort of mentorship program was very successful. We will have a new round of mentorship program coming up.
Start small: Don’t be afraid to start with small contributions. Even fixing a typo in the documentation or submitting a simple bug fix can be a great way to get involved.

By contributing to etcd, you’ll not only be helping to improve a critical piece of the cloud-native ecosystem but also gaining valuable experience and skills. So, jump in and start contributing!

Frederico: Excellent, thank you. Lastly, one piece of advice that you’d like to give to other newly formed SIGs?

Marek: Absolutely! My advice would be to embrace the established processes of the larger community, prioritize collaboration with other SIGs, and focus on building a strong community.

Wenjia: Here are some tips I myself found very helpful in my OSS journey:

Be patient: Open source development can take time. Don’t get discouraged if your contributions aren’t accepted immediately or if you encounter challenges.
Be respectful: The etcd community values collaboration and respect. Be mindful of others’ opinions and work together to achieve common goals.
Have fun: Contributing to open source should be enjoyable. Find areas that interest you and contribute in ways that you find fulfilling.

Frederico: A great way to end this spotlight, thank you all!

For more information and resources, please take a look at :

etcd website: https://etcd.io/
etcd GitHub repository: https://github.com/etcd-io/etcd
etcd community: https://etcd.io/community/

Blog: Spotlight on SIG Architecture: Enhancements

Tue, 21 Jan 2025 00:00:00 +0000

This is the fourth interview of a SIG Architecture Spotlight series that will cover the different subprojects, and we will be covering SIG Architecture: Enhancements.

In this SIG Architecture spotlight we talked with Kirsten Garrison, lead of the Enhancements subproject.

The Enhancements subproject

Frederico (FSM): Hi Kirsten, very happy to have the opportunity to talk about the Enhancements subproject. Let’s start with some quick information about yourself and your role.

Kirsten Garrison (KG): I’m a lead of the Enhancements subproject of SIG-Architecture and currently work at Google. I first got involved by contributing to the service-catalog project with the help of Carolyn Van Slyck. With time, I joined the Release team, eventually becoming the Enhancements Lead and a Release Lead shadow. While on the release team, I worked on some ideas to make the process better for the SIGs and Enhancements team (the opt-in process) based on my team’s experiences. Eventually, I started attending Subproject meetings and contributing to the Subproject’s work.

FSM: You mentioned the Enhancements subproject: how would you describe its main goals and areas of intervention?

KG: The Enhancements Subproject primarily concerns itself with the Kubernetes Enhancement Proposal (KEP for short)—the “design” documents required for all features and significant changes to the Kubernetes project.

The KEP and its impact

FSM: The improvement of the KEP process was (and is) one in which SIG Architecture was heavily involved. Could you explain the process to those that aren’t aware of it?

KG: Every release, the SIGs let the Release Team know which features they intend to work on to be put into the release. As mentioned above, the prerequisite for these changes is a KEP - a standardized design document that all authors must fill out and approve in the first weeks of the release cycle. Most features will move through 3 phases: alpha, beta and finally GA so approving a feature represents a significant commitment for the SIG.

The KEP serves as the full source of truth of a feature. The KEP template has different requirements based on what stage a feature is in, but it generally requires a detailed discussion of the design and the impact as well as providing artifacts of stability and performance. The KEP takes quite a bit of iterative work between authors, SIG reviewers, api review team and the Production Readiness Review team¹ before it is approved. Each set of reviewers is looking to make sure that the proposal meets their standards in order to have a stable and performant Kubernetes release. Only after all approvals are secured, can an author go forth and merge their feature in the Kubernetes code base.

FSM: I see, quite a bit of additional structure was added. Looking back, what were the most significant improvements of that approach?

KG: In general, I think that the improvements with the most impact had to do with focusing on the core intent of the KEP. KEPs exist not just to memorialize designs, but provide a structured way to discuss and come to an agreement about different facets of the change. At the core of the KEP process is communication and consideration.

To that end, some of the significant changes revolve around a more detailed and accessible KEP template. A significant amount of work was put in over time to get the k/enhancements repo into its current form – a directory structure organized by SIG with the contours of the modern KEP template (with Proposal/Motivation/Design Details subsections). We might take that basic structure for granted today, but it really represents the work of many people trying to get the foundation of this process in place over time.

As Kubernetes matures, we’ve needed to think about more than just the end goal of getting a single feature merged. We need to think about things like: stability, performance, setting and meeting user expectations. And as we’ve thought about those things the template has grown more detailed. The addition of the Production Readiness Review was major as well as the enhanced testing requirements (varying at different stages of a KEP’s lifecycle).

Current areas of focus

FSM: Speaking of maturing, we’ve recently released Kubernetes v1.31, and work on v1.32 has started. Are there any areas that the Enhancements sub-project is currently addressing that might change the way things are done?

KG: We’re currently working on two things:

Creating a Process KEP template. Sometimes people want to harness the KEP process for significant changes that are more process oriented rather than feature oriented. We want to support this because memorializing changes is important and giving people a better tool to do so will only encourage more discussion and transparency.
KEP versioning. While our template changes aim to be as non-disruptive as possible, we believe that it will be easier to track and communicate those changes to the community better with a versioned KEP template and the policies that go alongside such versioning.

Both features will take some time to get right and fully roll out (just like a KEP feature) but we believe that they will both provide improvements that will benefit the community at large.

FSM: You mentioned improvements: I remember when project boards for Enhancement tracking were introduced in recent releases, to great effect and unanimous applause from release team members. Was this a particular area of focus for the subproject?

KG: The Subproject provided support to the Release Team’s Enhancement team in the migration away from using the spreadsheet to a project board. The collection and tracking of enhancements has always been a logistical challenge. During my time on the Release Team, I helped with the transition to an opt-in system of enhancements, whereby the SIG leads “opt-in” KEPs for release tracking. This helped to enhance communication between authors and SIGs before any significant work was undertaken on a KEP and removed toil from the Enhancements team. This change used the existing tools to avoid introducing too many changes at once to the community. Later, the Release Team approached the Subproject with an idea of leveraging GitHub Project Boards to further improve the collection process. This was to be a move away from the use of complicated spreadsheets to using repo-native labels on k/enhancement issues and project boards.

FSM: That surely adds an impact on simplifying the workflow…

KG: Removing sources of friction and promoting clear communication is very important to the Enhancements Subproject. At the same time, it’s important to give careful consideration to decisions that impact the community as a whole. We want to make sure that changes are balanced to give an upside and while not causing any regressions and pain in the rollout. We supported the Release Team in ideation as well as through the actual migration to the project boards. It was a great success and exciting to see the team make high impact changes that helped everyone involved in the KEP process!

Getting involved

FSM: For those reading that might be curious and interested in helping, how would you describe the required skills for participating in the sub-project?

KG: Familiarity with KEPs either via experience or taking time to look through the kubernetes/enhancements repo is helpful. All are welcome to participate if interested - we can take it from there.

FSM: Excellent! Many thanks for your time and insight – any final comments you would like to share with our readers?

KG: The Enhancements process is one of the most important parts of Kubernetes and requires enormous amounts of coordination and collaboration of people and teams across the project to make it successful. I’m thankful and inspired by everyone’s continued hard work and dedication to making the project great. This is truly a wonderful community.

For more information, check the Production Readiness Review spotlight interview in this series. ↩︎

Blog: Spotlight on Kubernetes Upstream Training in Japan

Mon, 28 Oct 2024 00:00:00 +0000

We are organizers of Kubernetes Upstream Training in Japan. Our team is composed of members who actively contribute to Kubernetes, including individuals who hold roles such as member, reviewer, approver, and chair.

Our goal is to increase the number of Kubernetes contributors and foster the growth of the community. While Kubernetes community is friendly and collaborative, newcomers may find the first step of contributing to be a bit challenging. Our training program aims to lower that barrier and create an environment where even beginners can participate smoothly.

What is Kubernetes upstream training in Japan?

Our training started in 2019 and is held 1 to 2 times a year. Initially, Kubernetes Upstream Training was conducted as a co-located event of KubeCon (Kubernetes Contributor Summit), but we launched Kubernetes Upstream Training in Japan with the aim of increasing Japanese contributors by hosting a similar event in Japan.

Before the pandemic, the training was held in person, but since 2020, it has been conducted online. The training offers the following content for those who have not yet contributed to Kubernetes:

Introduction to Kubernetes community
Overview of Kubernetes codebase and how to create your first PR
Tips and encouragement to lower participation barriers, such as language
How to set up the development environment
Hands-on session using kubernetes-sigs/contributor-playground

At the beginning of the program, we explain why contributing to Kubernetes is important and who can contribute. We emphasize that contributing to Kubernetes allows you to make a global impact and that Kubernetes community is looking forward to your contributions!

We also explain Kubernetes community, SIGs, and Working Groups. Next, we explain the roles and responsibilities of Member, Reviewer, Approver, Tech Lead, and Chair. Additionally, we introduce the communication tools we primarily use, such as Slack, GitHub, and mailing lists. Some Japanese speakers may feel that communicating in English is a barrier. Additionally, those who are new to the community need to understand where and how communication takes place. We emphasize the importance of taking that first step, which is the most important aspect we focus on in our training!

We then go over the structure of Kubernetes codebase, the main repositories, how to create a PR, and the CI/CD process using Prow. We explain in detail the process from creating a PR to getting it merged.

After several lectures, participants get to experience hands-on work using kubernetes-sigs/contributor-playground, where they can create a simple PR. The goal is for participants to get a feel for the process of contributing to Kubernetes.

At the end of the program, we also provide a detailed explanation of setting up the development environment for contributing to the kubernetes/kubernetes repository, including building code locally, running tests efficiently, and setting up clusters.

Interview with participants

We conducted interviews with those who participated in our training program. We asked them about their reasons for joining, their impressions, and their future goals.

Keita Mochizuki (NTT DATA Group Corporation)

Keita Mochizuki is a contributor who consistently contributes to Kubernetes and related projects. Keita is also a professional in container security and has recently published a book. Additionally, he has made available a Roadmap for New Contributors, which is highly beneficial for those new to contributing.

Junya: Why did you decide to participate in Kubernetes Upstream Training?

Keita: Actually, I participated twice, in 2020 and 2022. In 2020, I had just started learning about Kubernetes and wanted to try getting involved in activities outside of work, so I signed up after seeing the event on Twitter by chance. However, I didn’t have much knowledge at the time, and contributing to OSS felt like something beyond my reach. As a result, my understanding after the training was shallow, and I left with more of a “hmm, okay” feeling.

In 2022, I participated again when I was at a stage where I was seriously considering starting contributions. This time, I did prior research and was able to resolve my questions during the lectures, making it a very productive experience.

Junya: How did you feel after participating?

Keita: I felt that the significance of this training greatly depends on the participant’s mindset. The training itself consists of general explanations and simple hands-on exercises, but it doesn’t mean that attending the training will immediately lead to contributions.

Junya: What is your purpose for contributing?

Keita: My initial motivation was to “gain a deep understanding of Kubernetes and build a track record,” meaning “contributing itself was the goal.” Nowadays, I also contribute to address bugs or constraints I discover during my work. Additionally, through contributing, I’ve become less hesitant to analyze undocumented features directly from the source code.

Junya: What has been challenging about contributing?

Keita: The most difficult part was taking the first step. Contributing to OSS requires a certain level of knowledge, and leveraging resources like this training and support from others was essential. One phrase that stuck with me was, “Once you take the first step, it becomes easier to move forward.” Also, in terms of continuing contributions as part of my job, the most challenging aspect is presenting the outcomes as achievements. To keep contributing over time, it’s important to align it with business goals and strategies, but upstream contributions don’t always lead to immediate results that can be directly tied to performance. Therefore, it’s crucial to ensure mutual understanding with managers and gain their support.

Junya: What are your future goals?

Keita: My goal is to contribute to areas with a larger impact. So far, I’ve mainly contributed by fixing smaller bugs as my primary focus was building a track record, but moving forward, I’d like to challenge myself with contributions that have a greater impact on Kubernetes users or that address issues related to my work. Recently, I’ve also been working on reflecting the changes I’ve made to the codebase into the official documentation, and I see this as a step toward achieving my goals.

Junya: Thank you very much!

Yoshiki Fujikane (CyberAgent, Inc.)

Yoshiki Fujikane is one of the maintainers of PipeCD, a CNCF Sandbox project. In addition to developing new features for Kubernetes support in PipeCD, Yoshiki actively participates in community management and speaks at various technical conferences.

Junya: Why did you decide to participate in the Kubernetes Upstream Training?

Yoshiki: At the time I participated, I was still a student. I had only briefly worked with EKS, but I thought Kubernetes seemed complex yet cool, and I was casually interested in it. Back then, OSS felt like something out of reach, and upstream development for Kubernetes seemed incredibly daunting. While I had always been interested in OSS, I didn’t know where to start. It was during this time that I learned about the Kubernetes Upstream Training and decided to take the challenge of contributing to Kubernetes.

Junya: What were your impressions after participating?

Yoshiki: I found it extremely valuable as a way to understand what it’s like to be part of an OSS community. At the time, my English skills weren’t very strong, so accessing primary sources of information felt like a big hurdle for me. Kubernetes is a very large project, and I didn’t have a clear understanding of the overall structure, let alone what was necessary for contributing. The upstream training provided a Japanese explanation of the community structure and allowed me to gain hands-on experience with actual contributions. Thanks to the guidance I received, I was able to learn how to approach primary sources and use them as entry points for further investigation, which was incredibly helpful. This experience made me realize the importance of organizing and reviewing primary sources, and now I often dive into GitHub issues and documentation when something piques my interest. As a result, while I am no longer contributing to Kubernetes itself, the experience has been a great foundation for contributing to other projects.

Junya: What areas are you currently contributing to, and what are the other projects you’re involved in?

Yoshiki: Right now, I’m no longer working with Kubernetes, but instead, I’m a maintainer of PipeCD, a CNCF Sandbox project. PipeCD is a CD tool that supports GitOps-style deployments for various application platforms. The tool originally started as an internal project at CyberAgent. With different teams adopting different platforms, PipeCD was developed to provide a unified CD platform with a consistent user experience. Currently, it supports Kubernetes, AWS ECS, Lambda, Cloud Run, and Terraform.

Junya: What role do you play within the PipeCD team?

Yoshiki: I work full-time on improving and developing Kubernetes-related features within the team. Since we provide PipeCD as a SaaS internally, my main focus is on adding new features and improving existing ones as part of that support. In addition to code contributions, I also contribute by giving talks at various events and managing community meetings to help grow the PipeCD community.

Junya: Could you explain what kind of improvements or developments you are working on with regards to Kubernetes?

Yoshiki: PipeCD supports GitOps and Progressive Delivery for Kubernetes, so I’m involved in the development of those features. Recently, I’ve been working on features that streamline deployments across multiple clusters.

Junya: Have you encountered any challenges while contributing to OSS?

Yoshiki: One challenge is developing features that maintain generality while meeting user use cases. When we receive feature requests while operating the internal SaaS, we first consider adding features to solve those issues. At the same time, we want PipeCD to be used by a broader audience as an OSS tool. So, I always think about whether a feature designed for one use case could be applied to another, ensuring the software remains flexible and widely usable.

Junya: What are your goals moving forward?

Yoshiki: I want to focus on expanding PipeCD’s functionality. Currently, we are developing PipeCD under the slogan “One CD for All.” As I mentioned earlier, it supports Kubernetes, AWS ECS, Lambda, Cloud Run, and Terraform, but there are many other platforms out there, and new platforms may emerge in the future. For this reason, we are currently developing a plugin system that will allow users to extend PipeCD on their own, and I want to push this effort forward. I’m also working on features for multi-cluster deployments in Kubernetes, and I aim to continue making impactful contributions.

Junya: Thank you very much!

Future of Kubernetes upstream training

We plan to continue hosting Kubernetes Upstream Training in Japan and look forward to welcoming many new contributors. Our next session is scheduled to take place at the end of November during CloudNative Days Winter 2024.

Moreover, our goal is to expand these training programs not only in Japan but also around the world. Kubernetes celebrated its 10th anniversary this year, and for the community to become even more active, it’s crucial for people across the globe to continue contributing. While Upstream Training is already held in several regions, we aim to bring it to even more places.

We hope that as more people join Kubernetes community and contribute, our community will become even more vibrant!

Blog: Announcing the 2024 Steering Committee Election Results

Wed, 02 Oct 2024 15:10:00 -0500

The 2024 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 3 of which were up for election in 2024. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

This community body is significant since it oversees the governance of the entire Kubernetes project. With that great power comes great responsibility. You can learn more about the steering committee’s role in their charter.

Thank you to everyone who voted in the election; your participation helps support the community’s continued health and success.

Results

Congratulations to the elected committee members whose two year terms begin immediately (listed in alphabetical order by GitHub handle):

Antonio Ojea (@aojea), Google
Benjamin Elder (@BenTheElder), Google
Sascha Grunert (@saschagrunert), Red Hat

They join continuing members:

Stephen Augustus (@justaugustus), Cisco
Paco Xu 徐俊杰 (@pacoxu), DaoCloud
Patrick Ohly (@pohly), Intel
Maciej Szulik (@soltysh), Defense Unicorns

Benjamin Elder is a returning Steering Committee Member.

Big thanks!

Thank you and congratulations on a successful election to this round’s election officers:

Bridget Kromhout (@bridgetkromhout)
Christoph Blecker (@cblecker)
Priyanka Saggu (@Priyankasaggu11929)

Thanks to the Emeritus Steering Committee Members. Your service is appreciated by the community:

Bob Killen (@mrbobbytables)
Nabarun Pal (@palnabarun)

And thank you to all the candidates who came forward to run for election.

Get involved with the Steering Committee

This governing body, like all of Kubernetes, is open to all. You can follow along with Steering Committee meeting notes and weigh in by filing an issue or creating a PR against their repo. They have an open meeting on the first Monday at 8am PT of every month. They can also be contacted at their public mailing list steering@kubernetes.io.

You can see what the Steering Committee meetings are all about by watching past meetings on the YouTube Playlist.

If you want to meet some of the newly elected Steering Committee members, join us for the Steering AMA at the Kubernetes Contributor Summit North America 2024 in Salt Lake City.

This post was adapted from one written by the Contributor Comms Subproject. If you want to write stories about the Kubernetes community, learn more about us.

Blog: Spotlight on CNCF Deaf and Hard-of-hearing Working Group (DHHWG)

Mon, 30 Sep 2024 00:00:00 +0000

In recognition of Deaf Awareness Month and the importance of inclusivity in the tech community, we are spotlighting Catherine Paganini, facilitator and one of the founding members of CNCF Deaf and Hard-of-Hearing Working Group (DHHWG). In this interview, Sandeep Kanabar, a deaf member of the DHHWG and part of the Kubernetes SIG ContribEx Communications team, sits down with Catherine to explore the impact of the DHHWG on cloud native projects like Kubernetes.

Sandeep’s journey is a testament to the power of inclusion. Through his involvement in the DHHWG, he connected with members of the Kubernetes community who encouraged him to join SIG ContribEx - the group responsible for sustaining the Kubernetes contributor experience. In an ecosystem where open-source projects are actively seeking contributors and maintainers, this story highlights how important it is to create pathways for underrepresented groups, including those with disabilities, to contribute their unique perspectives and skills.

In this interview, we delve into Catherine’s journey, the challenges and triumphs of establishing the DHHWG, and the vision for a more inclusive future in cloud native. We invite Kubernetes contributors, maintainers, and community members to reflect on the significance of empathy, advocacy, and community in fostering a truly inclusive environment for all, and to think about how they can support efforts to increase diversity and accessibility within their own projects.

Introduction

Sandeep Kanabar (SK): Hello Catherine, could you please introduce yourself, share your professional background, and explain your connection to the Kubernetes ecosystem?

Catherine Paganini (CP): I’m the Head of Marketing at Buoyant, the creator of Linkerd, the CNCF-graduated service mesh, and 5th CNCF project. Four years ago, I started contributing to open source. The initial motivation was to make cloud native concepts more accessible to newbies and non-technical people. Without a technical background, it was hard for me to understand what Kubernetes, containers, service meshes, etc. mean. All content was targeted at engineers already familiar with foundational concepts. Clearly, I couldn’t be the only one struggling with wrapping my head around cloud native.

My first contribution was the CNCF Landscape Guide, which I co-authored with my former colleague Jason Morgan. Next, we started the CNCF Glossary, which explains cloud native concepts in simple terms. Today, the glossary has been (partially) localised into 14 languages!

Currently, I’m the co-chair of the TAG Contributor Strategy and the Facilitator of the Deaf and Hard of Hearing Working Group (DHHWG) and Blind and Visually Impaired WG (BVIWG), which is still in formation. I’m also working on a new Linux Foundation (LF) initiative called ABIDE (Accessibility and Belonging through Inclusion, Diversity, and Equity), so stay tuned to learn more about it!

Motivation and early milestones

SK: That’s inspiring! Building on your passion for accessibility, what motivated you to facilitate the creation of the DHHWG? Was there a speecifc moment or experience that sparked this initiative?

CP: Last year at KubeCon Amsterdam, I learned about a great initiative by Jay Tihema that creates pathways for Maori youth into cloud native and open source. While telling my CODA (children of deaf adults) high school friend about it, I thought it’d be great to create something similar for deaf folks. A few months later, I posted about it in a LinkedIn post that the CNCF shared. Deaf people started to reach out, wanting to participate. And the rest is history.

SK: Speaking of history, since its launch, how has the DHHWG evolved? Could you highlight some of the key milestones or achievements the group has reached recently?

CP: Our WG is about a year old. It started with a few deaf engineers and me brainstorming how to make KubeCon more accessible. We published an initial draft of Best practices for an inclusive conference and shared it with the LF events team. KubeCon Chicago was two months later, and we had a couple of deaf attendees. It was the first KubeCon accessible to deaf signers. Destiny, one of our co-chairs, even participated in a keynote panel. It was incredible how quickly everything happened!

DHHWG members at KubeCon Chicago

The team has grown since then, and we’ve been able to do much more. With a kiosk in the project pavilion, an open space discussion, a sign language crash course, and a few media interviews, KubeCon Paris had a stronger advocacy and outreach focus. Check out this video of our team in Paris to get a glimpse of all the different KubeCon activities — it was such a great event! The team also launched the first CNCF Community Group in sign language, Deaf in Cloud Native, a glossary team that creates sign language videos for each technical term to help standardize technical signs across the globe. It’s crazy to think that it all happened within one year!

Overcoming challenges and addressing misconceptions

SK: That’s remarkable progress in just a year! Building such momentum must have come with its challenges. What barriers have you encountered in facilitating the DHHWG, and how did you and the group work to overcome them?

CP: The support from the community, LF, and CNCF has been incredible. The fact that we achieved so much is proof of it. The challenges are more in helping some team members overcome their fear of contributing. Most are new to open source, and it can be intimidating to put your work out there for everyone to see. The fear of being criticized in public is real; however, as they will hopefully realize over time, our community is incredibly supportive. Instead of criticizing, people tend to help improve the work, leading to better outcomes.

SK: Are there any misconceptions about the deaf and hard-of-hearing community in tech that you’d like to address?

CP: Deaf and hard of hearing individuals are very diverse — there is no one-size-fits-all. Some deaf people are oral (speak), others sign, while some lip read or prefer captions. It generally depends on how people grew up. While some people come from deaf families and sign language is their native language, others were born into hearing families who may or may not have learned how to sign. Some deaf people grew up surrounded by hearing people, while others grew up deeply embedded in Deaf culture. Hard-of-hearing individuals, on the other hand, typically can communicate well with hearing peers one-on-one in quiet settings, but loud environments or conversations with multiple people can make it hard to follow the conversation. Most rely heavily on captions. Each background and experience will shape their communication style and preferences. In short, what works for one person, doesn’t necessarily work for others. So never assume and always ask about accessibility needs and preferences.

Impact and the role of allies

SK: Can you share some key impacts/outcomes of the conference best practices document?

CP: Here are the two most important ones: Captions should be on the monitor, not in an app. That’s especially important during technical talks with live demos. Deaf and hard of hearing attendees will miss important information switching between captions on their phone and code on the screen.

Interpreters are most valuable during networking, not in talks (with captions). Most people come to conferences for the hallway track. That is no different for deaf attendees. If they can’t network, they are missing out on key professional connections, affecting their career prospects.

SK: In your view, how crucial is the role of allies within the DHHWG, and what contributions have they made to the group’s success?

CP: Deaf and hard of hearing individuals are a minority and can only do so much. Allies are the key to any diversity and inclusion initiative. As a majority, allies can help spread the word and educate their peers, playing a key role in scaling advocacy efforts. They also have the power to demand change. It’s easy for companies to ignore minorities, but if the majority demands that their employers be accessible, environmentally conscious, and good citizens, they will ultimately be pushed to adapt to new societal values.

Expanding DEI efforts and future vision

SK: The importance of allies in driving change is clear. Beyond the DHHWG, are you involved in any other DEI groups or initiatives within the tech community?

CP: As mentioned above, I’m working on an initiative called ABIDE, which is still work in progress. I don’t want to share too much about it yet, but what I can say is that the DHHWG will be part of it and that we just started a Blind and Visually Impaired WG (BVIWG). ABIDE will start by focusing on accessibility, so if anyone reading this has an idea for another WG, please reach out to me via the CNCF Slack @Catherine Paganini.

SK: What does the future hold for the DHHWG? Can you share details about any ongoing or upcoming initiatives?

CP: I think we’ve been very successful in terms of visibility and awareness so far. We can’t stop, though. Awareness work is ongoing, and most people in our community haven’t heard about us or met anyone on our team yet, so a lot of work still lies ahead.

DHHWG members at KubeCon Paris

The next step is to refocus on advocacy. The same thing we did with the conference best practices but for other areas. The goal is to help educate the community about what real accessibility looks like, how projects can be more accessible, and why employers should seriously consider deaf candidates while providing them with the tools they need to conduct successful interviews and employee onboarding. We need to capture all that in documents, publish it, and then get the word out. That last part is certainly the most challenging, but it’s also where everyone can get involved.

Call to action

SK: Thank you for sharing your insights, Catherine. As we wrap up, do you have any final thoughts or a call to action for our readers?

CP: As we build our accessibility page, check in regularly to see what’s new. Share the docs with your team, employer, and network — anyone, really. The more people understand what accessibility really means and why it matters, the more people will recognize when something isn’t accessible, and be able to call out marketing-BS, which, unfortunately, is more often the case than not. We need allies to help push for change. No minority can do this on their own. So please learn about accessibility, keep an eye out for it, and call it out when something isn’t accessible. We need your help!

Wrapping up

Catherine and the DHHWG’s work exemplify the power of community and advocacy. As we celebrate Deaf Awareness Month, let’s reflect on her role as an ally and consider how we can all contribute to building a more inclusive tech community, particularly within open-source projects like Kubernetes.

Together, we can break down barriers, challenge misconceptions, and ensure that everyone feels welcome and valued. By advocating for accessibility, supporting initiatives like the DHHWG, and fostering a culture of empathy, we can create a truly inclusive and welcoming space for all.

Blog: Spotlight on SIG Scheduling

Tue, 24 Sep 2024 00:00:00 +0000

In this SIG Scheduling spotlight we talked with Kensei Nakada, an approver in SIG Scheduling.

Introductions

Arvind: Hello, thank you for the opportunity to learn more about SIG Scheduling! Would you like to introduce yourself and tell us a bit about your role, and how you got involved with Kubernetes?

Kensei: Hi, thanks for the opportunity! I’m Kensei Nakada (@sanposhiho), a software engineer at Tetrate.io. I have been contributing to Kubernetes in my free time for more than 3 years, and now I’m an approver of SIG Scheduling in Kubernetes. Also, I’m a founder/owner of two SIG subprojects, kube-scheduler-simulator and kube-scheduler-wasm-extension.

About SIG Scheduling

AP: That’s awesome! You’ve been involved with the project since a long time. Can you provide a brief overview of SIG Scheduling and explain its role within the Kubernetes ecosystem?

KN: As the name implies, our responsibility is to enhance scheduling within Kubernetes. Specifically, we develop the components that determine which Node is the best place for each Pod. In Kubernetes, our main focus is on maintaining the kube-scheduler, along with other scheduling-related components as part of our SIG subprojects.

AP: I see, got it! That makes me curious–what recent innovations or developments has SIG Scheduling introduced to Kubernetes scheduling?

KN: From a feature perspective, there have been several enhancements to PodTopologySpread recently. PodTopologySpread is a relatively new feature in the scheduler, and we are still in the process of gathering feedback and making improvements.

Most recently, we have been focusing on a new internal enhancement called QueueingHint which aims to enhance scheduling throughput. Throughput is one of our crucial metrics in scheduling. Traditionally, we have primarily focused on optimizing the latency of each scheduling cycle. QueueingHint takes a different approach, optimizing when to retry scheduling, thereby reducing the likelihood of wasting scheduling cycles.

A: That sounds interesting! Are there any other interesting topics or projects you are currently working on within SIG Scheduling?

KN: I’m leading the development of QueueingHint which I just shared. Given that it’s a big new challenge for us, we’ve been facing many unexpected challenges, especially around the scalability, and we’re trying to solve each of them to eventually enable it by default.

And also, I believe kube-scheduler-wasm-extension (a SIG subproject) that I started last year would be interesting to many people. Kubernetes has various extensions from many components. Traditionally, extensions are provided via webhooks (extender in the scheduler) or Go SDK (Scheduling Framework in the scheduler). However, these come with drawbacks - performance issues with webhooks and the need to rebuild and replace schedulers with Go SDK, posing difficulties for those seeking to extend the scheduler but lacking familiarity with it. The project is trying to introduce a new solution to this general challenge - a WebAssembly based extension. Wasm allows users to build plugins easily, without worrying about recompiling or replacing their scheduler, and sidestepping performance concerns.

Through this project, SIG Scheduling has been learning valuable insights about WebAssembly’s interaction with large Kubernetes objects. And I believe the experience that we’re gaining should be useful broadly within the community, beyond SIG Scheduling.

A: Definitely! Now, there are 8 subprojects inside SIG Scheduling. Would you like to talk about them? Are there some interesting contributions by those teams you want to highlight?

KN: Let me pick up three subprojects: Kueue, KWOK and descheduler.

Kueue: Recently, many people have been trying to manage batch workloads with Kubernetes, and in 2022, Kubernetes community founded WG-Batch for better support for such batch workloads in Kubernetes. Kueue is a project that takes a crucial role for it. It’s a job queueing controller, deciding when a job should wait, when a job should be admitted to start, and when a job should be preempted. Kueue aims to be installed on a vanilla Kubernetes cluster while cooperating with existing matured controllers (scheduler, cluster-autoscaler, kube-controller-manager, etc).
KWOK: KWOK is a component in which you can create a cluster of thousands of Nodes in seconds. It’s mostly useful for simulation/testing as a lightweight cluster, and actually another SIG sub project kube-scheduler-simulator uses KWOK background.
descheduler: Descheduler is a component recreating pods that are running on undesired Nodes. In Kubernetes, scheduling constraints (PodAffinity, NodeAffinity, PodTopologySpread, etc) are honored only at Pod schedule, but it’s not guaranteed that the contrtaints are kept being satisfied afterwards. Descheduler evicts Pods violating their scheduling constraints (or other undesired conditions) so that they’re recreated and rescheduled.
Descheduling Framework: One very interesting on-going project, similar to Scheduling Framework in the scheduler, aiming to make descheduling logic extensible and allow maintainers to focus on building a core engine of descheduler.

AP: Thank you for letting us know! And I have to ask, what are some of your favorite things about this SIG?

KN: What I really like about this SIG is how actively engaged everyone is. We come from various companies and industries, bringing diverse perspectives to the table. Instead of these differences causing division, they actually generate a wealth of opinions. Each view is respected, and this makes our discussions both rich and productive.

I really appreciate this collaborative atmosphere, and I believe it has been key to continuously improving our components over the years.

Contributing to SIG Scheduling

AP: Kubernetes is a community-driven project. Any recommendations for new contributors or beginners looking to get involved and contribute to SIG scheduling? Where should they start?

KN: Let me start with a general recommendation for contributing to any SIG: a common approach is to look for good-first-issue. However, you’ll soon realize that many people worldwide are trying to contribute to the Kubernetes repository.

I suggest starting by examining the implementation of a component that interests you. If you have any questions about it, ask in the corresponding Slack channel (e.g., #sig-scheduling for the scheduler, #sig-node for kubelet, etc). Once you have a rough understanding of the implementation, look at issues within the SIG (e.g., sig-scheduling), where you’ll find more unassigned issues compared to good-first-issue ones. You may also want to filter issues with the kind/cleanup label, which often indicates lower-priority tasks and can be starting points.

Specifically for SIG Scheduling, you should first understand the Scheduling Framework, which is the fundamental architecture of kube-scheduler. Most of the implementation is found in pkg/scheduler. I suggest starting with ScheduleOne function and then exploring deeper from there.

Additionally, apart from the main kubernetes/kubernetes repository, consider looking into sub-projects. These typically have fewer maintainers and offer more opportunities to make a significant impact. Despite being called “sub” projects, many have a large number of users and a considerable impact on the community.

And last but not least, remember contributing to the community isn’t just about code. While I talked a lot about the implementation contribution, there are many ways to contribute, and each one is valuable. One comment to an issue, one feedback to an existing feature, one review comment in PR, one clarification on the documentation; every small contribution helps drive the Kubernetes ecosystem forward.

AP: Those are some pretty useful tips! And if I may ask, how do you assist new contributors in getting started, and what skills are contributors likely to learn by participating in SIG Scheduling?

KN: Our maintainers are available to answer your questions in the #sig-scheduling Slack channel. By participating, you’ll gain a deeper understanding of Kubernetes scheduling and have the opportunity to collaborate and network with maintainers from diverse backgrounds. You’ll learn not just how to write code, but also how to maintain a large project, design and discuss new features, address bugs, and much more.

Future Directions

AP: What are some Kubernetes-specific challenges in terms of scheduling? Are there any particular pain points?

KN: Scheduling in Kubernetes can be quite challenging because of the diverse needs of different organizations with different business requirements. Supporting all possible use cases in kube-scheduler is impossible. Therefore, extensibility is a key focus for us. A few years ago, we rearchitected kube-scheduler with Scheduling Framework, which offers flexible extensibility for users to implement various scheduling needs through plugins. This allows maintainers to focus on the core scheduling features and the framework runtime.

Another major issue is maintaining sufficient scheduling throughput. Typically, a Kubernetes cluster has only one kube-scheduler, so its throughput directly affects the overall scheduling scalability and, consequently, the cluster’s scalability. Although we have an internal performance test (scheduler_perf), unfortunately, we sometimes overlook performance degradation in less common scenarios. It’s difficult as even small changes, which look irrelevant to performance, can lead to degradation.

AP: What are some upcoming goals or initiatives for SIG Scheduling? How do you envision the SIG evolving in the future?

KN: Our primary goal is always to build and maintain extensible and stable scheduling runtime, and I bet this goal will remain unchanged forever.

As already mentioned, extensibility is key to solving the challenge of the diverse needs of scheduling. Rather than trying to support every different use case directly in kube-scheduler, we will continue to focus on enhancing extensibility so that it can accommodate various use cases. kube-scheduler-wasm-extension that I mentioned is also part of this initiative.

Regarding stability, introducing new optimizations like QueueHint is one of our strategies. Additionally, maintaining throughput is also a crucial goal towards the future. We’re planning to enhance our throughput monitoring (ref), so that we can notice degradation as much as possible on our own before releasing. But, realistically, we can’t cover every possible scenario. We highly appreciate any attention the community can give to scheduling throughput and encourage feedback and alerts regarding performance issues!

Closing Remarks

AP: Finally, what message would you like to convey to those who are interested in learning more about SIG Scheduling?

KN: Scheduling is one of the most complicated areas in Kubernetes, and you may find it difficult at first. But, as I shared earlier, you can find many opportunities for contributions, and many maintainers are willing to help you understand things. We know your unique perspective and skills are what makes our open source so powerful 😊

Feel free to reach out to us in Slack (#sig-scheduling) or meetings. I hope this article interests everyone and we can see new contributors!

AP: Thank you so much for taking the time to do this! I’m confident that many will find this information invaluable for understanding more about SIG Scheduling and for contributing to the SIG.

Blog: Spotlight on SIG API Machinery

Wed, 07 Aug 2024 00:00:00 +0000

We recently talked with Federico Bongiovanni (Google) and David Eads (Red Hat), Chairs of SIG API Machinery, to know a bit more about this Kubernetes Special Interest Group.

Introductions

Frederico (FSM): Hello, and thank your for your time. To start with, could you tell us about yourselves and how you got involved in Kubernetes?

David: I started working on OpenShift (the Red Hat distribution of Kubernetes) in the fall of 2014 and got involved pretty quickly in API Machinery. My first PRs were fixing kube-apiserver error messages and from there I branched out to kubectl (kubeconfigs are my fault!), auth (RBAC and *Review APIs are ports from OpenShift), apps (workqueues and sharedinformers for example). Don’t tell the others, but API Machinery is still my favorite :)

Federico: I was not as early in Kubernetes as David, but now it’s been more than six years. At my previous company we were starting to use Kubernetes for our own products, and when I came across the opportunity to work directly with Kubernetes I left everything and boarded the ship (no pun intended). I joined Google and Kubernetes in early 2018, and have been involved since.

SIG Machinery’s scope

FSM: It only takes a quick look at the SIG API Machinery charter to see that it has quite a significant scope, nothing less than the Kubernetes control plane. Could you describe this scope in your own words?

David: We own the kube-apiserver and how to efficiently use it. On the backend, that includes its contract with backend storage and how it allows API schema evolution over time. On the frontend, that includes schema best practices, serialization, client patterns, and controller patterns on top of all of it.

Federico: Kubernetes has a lot of different components, but the control plane has a really critical mission: it’s your communication layer with the cluster and also owns all the extensibility mechanisms that make Kubernetes so powerful. We can’t make mistakes like a regression, or an incompatible change, because the blast radius is huge.

FSM: Given this breadth, how do you manage the different aspects of it?

Federico: We try to organize the large amount of work into smaller areas. The working groups and subprojects are part of it. Different people on the SIG have their own areas of expertise, and if everything fails, we are really lucky to have people like David, Joe, and Stefan who really are “all terrain”, in a way that keeps impressing me even after all these years. But on the other hand this is the reason why we need more people to help us carry the quality and excellence of Kubernetes from release to release.

An evolving collaboration model

FSM: Was the existing model always like this, or did it evolve with time - and if so, what would you consider the main changes and the reason behind them?

David: API Machinery has evolved over time both growing and contracting in scope. When trying to satisfy client access patterns it’s very easy to add scope both in terms of features and applying them.

A good example of growing scope is the way that we identified a need to reduce memory utilization by clients writing controllers and developed shared informers. In developing shared informers and the controller patterns use them (workqueues, error handling, and listers), we greatly reduced memory utilization and eliminated many expensive lists. The downside: we grew a new set of capability to support and effectively took ownership of that area from sig-apps.

For an example of more shared ownership: building out cooperative resource management (the goal of server-side apply), kubectl expanded to take ownership of leveraging the server-side apply capability. The transition isn’t yet complete, but SIG CLI manages that usage and owns it.

FSM: And for the boundary between approaches, do you have any guidelines?

David: I think much depends on the impact. If the impact is local in immediate effect, we advise other SIGs and let them move at their own pace. If the impact is global in immediate effect without a natural incentive, we’ve found a need to press for adoption directly.

FSM: Still on that note, SIG Architecture has an API Governance subproject, is it mostly independent from SIG API Machinery or are there important connection points?

David: The projects have similar sounding names and carry some impacts on each other, but have different missions and scopes. API Machinery owns the how and API Governance owns the what. API conventions, the API approval process, and the final say on individual k8s.io APIs belong to API Governance. API Machinery owns the REST semantics and non-API specific behaviors.

Federico: I really like how David put it: “API Machinery owns the how and API Governance owns the what”: we don’t own the actual APIs, but the actual APIs live through us.

The challenges of Kubernetes popularity

FSM: With the growth in Kubernetes adoption we have certainly seen increased demands from the Control Plane: how is this felt and how does it influence the work of the SIG?

David: It’s had a massive influence on API Machinery. Over the years we have often responded to and many times enabled the evolutionary stages of Kubernetes. As the central orchestration hub of nearly all capability on Kubernetes clusters, we both lead and follow the community. In broad strokes I see a few evolution stages for API Machinery over the years, with constantly high activity.

Finding purpose: pre-1.0 up until v1.3 (up to our first 1000+ nodes/namespaces) or so. This time was characterized by rapid change. We went through five different versions of our schemas and rose to meet the need. We optimized for quick, in-tree API evolution (sometimes to the detriment of longer term goals), and defined patterns for the first time.
Scaling to meet the need: v1.3-1.9 (up to shared informers in controllers) or so. When we started trying to meet customer needs as we gained adoption, we found severe scale limitations in terms of CPU and memory. This was where we broadened API machinery to include access patterns, but were still heavily focused on in-tree types. We built the watch cache, protobuf serialization, and shared caches.
Fostering the ecosystem: v1.8-1.21 (up to CRD v1) or so. This was when we designed and wrote CRDs (the considered replacement for third-party-resources), the immediate needs we knew were coming (admission webhooks), and evolution to best practices we knew we needed (API schemas). This enabled an explosion of early adopters willing to work very carefully within the constraints to enable their use-cases for servicing pods. The adoption was very fast, sometimes outpacing our capability, and creating new problems.
Simplifying deployments: v1.22+. In the relatively recent past, we’ve been responding to pressures or running kube clusters at scale with large numbers of sometimes-conflicting ecosystem projects using our extensions mechanisms. Lots of effort is now going into making platform extensions easier to write and safer to manage by people who don’t hold PhDs in kubernetes. This started with things like server-side-apply and continues today with features like webhook match conditions and validating admission policies.

Work in API Machinery has a broad impact across the project and the ecosystem. It’s an exciting area to work for those able to make a significant time investment on a long time horizon.

The road ahead

FSM: With those different evolutionary stages in mind, what would you pinpoint as the top priorities for the SIG at this time?

David: Reliability, efficiency, and capability in roughly that order.

With the increased usage of our kube-apiserver and extensions mechanisms, we find that our first set of extensions mechanisms, while fairly complete in terms of capability, carry significant risks in terms of potential mis-use with large blast radius. To mitigate these risks, we’re investing in features that reduce the blast radius for accidents (webhook match conditions) and which provide alternative mechanisms with lower risk profiles for most actions (validating admission policy).

At the same time, the increased usage has made us more aware of scaling limitations that we can improve both server and client-side. Efforts here include more efficient serialization (CBOR), reduced etcd load (consistent reads from cache), and reduced peak memory usage (streaming lists).

And finally, the increased usage has highlighted some long existing gaps that we’re closing. Things like field selectors for CRDs which the Batch Working Group is eager to leverage and will eventually form the basis for a new way to prevent trampoline pod attacks from exploited nodes.

Joining the fun

FSM: For anyone wanting to start contributing, what’s your suggestions?

Federico: SIG API Machinery is not an exception to the Kubernetes motto: Chop Wood and Carry Water. There are multiple weekly meetings that are open to everybody, and there is always more work to be done than people to do it.

I acknowledge that API Machinery is not easy, and the ramp up will be steep. The bar is high, because of the reasons we’ve been discussing: we carry a huge responsibility. But of course with passion and perseverance many people has ramped up through the years, and we hope more will come.

In terms of concrete opportunities, there is the SIG meeting every two weeks. Everyone is welcome to attend and listen, see what the group talks about, see what’s going on in this release, etc.

Also two times a week, Tuesday and Thursday, we have the public Bug Triage, where we go through everything new from the last meeting. We’ve been keeping this practice for more than 7 years now. It’s a great opportunity to volunteer to review code, fix bugs, improve documentation, etc. Tuesday’s it’s at 1 PM (PST) and Thursday is on an EMEA friendly time (9:30 AM PST). We are always looking to improve, and we hope to be able to provide more concrete opportunities to join and participate in the future.

FSM: Excellent, thank you! Any final comments you would like to share with our readers?

Federico: As I mentioned, the first steps might be hard, but the reward is also larger. Working on API Machinery is working on an area of huge impact (millions of users?), and your contributions will have a direct outcome in the way that Kubernetes works and the way that it’s used. For me that’s enough reward and motivation!

Blog: Spotlight on SIG Node

Thu, 20 Jun 2024 00:00:00 +0000

In the world of container orchestration, Kubernetes reigns supreme, powering some of the most complex and dynamic applications across the globe. Behind the scenes, a network of Special Interest Groups (SIGs) drives Kubernetes’ innovation and stability.

Today, I have the privilege of speaking with Matthias Bertschy, Gunju Kim, and Sergey Kanzhelev, members of SIG Node, who will shed some light on their roles, challenges, and the exciting developments within SIG Node.

Answers given collectively by all interviewees will be marked by their initials.

Introductions

Arpit: Thank you for joining us today. Could you please introduce yourselves and provide a brief overview of your roles within SIG Node?

Matthias: My name is Matthias Bertschy, I am French and live next to Lake Geneva, near the French Alps. I have been a Kubernetes contributor since 2017, a reviewer for SIG Node and a maintainer of Prow. I work as a Senior Kubernetes Developer for a security startup named ARMO, which donated Kubescape to the CNCF.

Gunju: My name is Gunju Kim. I am a software engineer at NAVER, where I focus on developing a cloud platform for search services. I have been contributing to the Kubernetes project in my free time since 2021.

Sergey: My name is Sergey Kanzhelev. I have worked on Kubernetes and Google Kubernetes Engine for 3 years and have worked on open-source projects for many years now. I am a chair of SIG Node.

Understanding SIG Node

Arpit: Thank you! Could you provide our readers with an overview of SIG Node’s responsibilities within the Kubernetes ecosystem?

M/G/S: SIG Node is one of the first if not the very first SIG in Kubernetes. The SIG is responsible for all iterations between Kubernetes and node resources, as well as node maintenance itself. This is quite a large scope, and the SIG owns a large part of the Kubernetes codebase. Because of this wide ownership, SIG Node is always in contact with other SIGs such as SIG Network, SIG Storage, and SIG Security and almost any new features and developments in Kubernetes involves SIG Node in some way.

Arpit: How does SIG Node contribute to Kubernetes’ performance and stability?

M/G/S: Kubernetes works on nodes of many different sizes and shapes, from small physical VMs with cheap hardware to large AI/ML-optimized GPU-enabled nodes. Nodes may stay online for months or maybe be short-lived and be preempted at any moment as they are running on excess compute of a cloud provider.

kubelet — the Kubernetes agent on a node — must work in all these environments reliably. As for the performance of kubelet operations, this is becoming increasingly important today. On one hand, as Kubernetes is being used on extra small nodes more and more often in telecom and retail environments, it needs to scale into the smallest footprint possible. On the other hand, with AI/ML workloads where every node is extremely expensive, every second of delayed operations can visibly change the price of computation.

Challenges and Opportunities

Arpit: What upcoming challenges and opportunities is SIG Node keeping an eye on?

M/G/S: As Kubernetes enters the second decade of its life, we see a huge demand to support new workload types. And SIG Node will play a big role in this. The Sidecar KEP, which we will be talking about later, is one of the examples of increased emphasis on supporting new workload types.

The key challenge we will have in the next few years is how to keep innovations while maintaining high quality and backward compatibility of existing scenarios. SIG Node will continue to play a central role in Kubernetes.

Arpit: And are there any ongoing research or development areas within SIG Node that excite you?

M/G/S: Supporting new workload types is a fascinating area for us. Our recent exploration of sidecar containers is a testament to this. Sidecars offer a versatile solution for enhancing application functionality without altering the core codebase.

Arpit: What are some of the challenges you’ve faced while maintaining SIG Node, and how have you overcome them?

M/G/S: The biggest challenge of SIG Node is its size and the many feature requests it receives. We are encouraging more people to join as reviewers and are always open to improving processes and addressing feedback. For every release, we run the feedback session at the SIG Node meeting and identify problematic areas and action items.

Arpit: Are there specific technologies or advancements that SIG Node is closely monitoring or integrating into Kubernetes?

M/G/S: Developments in components that the SIG depends on, like container runtimes (e.g. containerd and CRI-O, and OS features are something we contribute to and monitor closely. For example, there is an upcoming cgroup v1 deprecation and removal that Kubernetes and SIG Node will need to guide Kubernetes users through. Containerd is also releasing version 2.0, which removes deprecated features, which will affect Kubernetes users.

Arpit: Could you share a memorable experience or achievement from your time as a SIG Node maintainer that you’re particularly proud of?

Matthias: I think the best moment was when my first KEP (introducing the startupProbe) finally graduated to GA (General Availability). I also enjoy seeing my contributions being used daily by contributors, such as the comment containing the GitHub tree hash used to retain LGTM despite squash commits.

Sidecar containers

Arpit: Can you provide more context on the concept of sidecar containers and their evolution in the context of Kubernetes?

M/G/S: The concept of sidecar containers dates back to 2015 when Kubernetes introduced the idea of composite containers. These additional containers, running alongside the main application container within the same pod, were seen as a way to extend and enhance application functionality without modifying the core codebase. Early adopters of sidecars employed custom scripts and configurations to manage them, but this approach presented challenges in terms of consistency and scalability.

Arpit: Can you share specific use cases or examples where sidecar containers are particularly beneficial?

M/G/S: Sidecar containers are a versatile tool that can be used to enhance the functionality of applications in a variety of ways:

Logging and monitoring: Sidecar containers can be used to collect logs and metrics from the primary application container and send them to a centralized logging and monitoring system.
Traffic filtering and routing: Sidecar containers can be used to filter and route traffic to and from the primary application container.
Encryption and decryption: Sidecar containers can be used to encrypt and decrypt data as it flows between the primary application container and external services.
Data synchronization: Sidecar containers can be used to synchronize data between the primary application container and external databases or services.
Fault injection: Sidecar containers can be used to inject faults into the primary application container in order to test its resilience to failures.

Arpit: The proposal mentions that some companies are using a fork of Kubernetes with sidecar functionality added. Can you provide insights into the level of adoption and community interest in this feature?

M/G/S: While we lack concrete metrics to measure adoption rates, the KEP has garnered significant interest from the community, particularly among service mesh vendors like Istio, who actively participated in its alpha testing phase. The KEP’s visibility through numerous blog posts, interviews, talks, and workshops further demonstrates its widespread appeal. The KEP addresses the growing demand for additional capabilities alongside main containers in Kubernetes pods, such as network proxies, logging systems, and security measures. The community acknowledges the importance of providing easy migration paths for existing workloads to facilitate widespread adoption of the feature.

Arpit: Are there any notable examples or success stories from companies using sidecar containers in production?

M/G/S: It is still too early to expect widespread adoption in production environments. The 1.29 release has only been available in Google Kubernetes Engine (GKE) since January 11, 2024, and there still needs to be comprehensive documentation on how to enable and use them effectively via universal injector. Istio, a popular service mesh platform, also lacks proper documentation for enabling native sidecars, making it difficult for developers to get started with this new feature. However, as native sidecar support matures and documentation improves, we can expect to see wider adoption of this technology in production environments.

Arpit: The proposal suggests introducing a restartPolicy field for init containers to indicate sidecar functionality. Can you explain how this solution addresses the outlined challenges?

M/G/S: The proposal to introduce a restartPolicy field for init containers addresses the outlined challenges by utilizing existing infrastructure and simplifying sidecar management. This approach avoids adding new fields to the pod specification, keeping it manageable and avoiding more clutter. By leveraging the existing init container mechanism, sidecars can be run alongside regular init containers during pod startup, ensuring a consistent ordering of initialization. Additionally, setting the restart policy of sidecar init containers to Always explicitly states that they continue running even after the main application container terminates, enabling persistent services like logging and monitoring until the end of the workload.

Arpit: How will the introduction of the restartPolicy field for init containers affect backward compatibility with existing Kubernetes configurations?

M/G/S: The introduction of the restartPolicy field for init containers will maintain backward compatibility with existing Kubernetes configurations. Existing init containers will continue to function as they have before, and the new restartPolicy field will only apply to init containers explicitly marked as sidecars. This approach ensures that existing applications and deployments will not be disrupted by the new feature, and provides a more streamlined way to define and manage sidecars.

Contributing to SIG Node

Arpit: What is the best place for the new members and especially beginners to contribute?

M/G/S: New members and beginners can contribute to the Sidecar KEP (Kubernetes Enhancement Proposal) by:

Raising awareness: Create content that highlights the benefits and use cases of sidecars. This can educate others about the feature and encourage its adoption.
Providing feedback: Share your experiences with sidecars, both positive and negative. This feedback can be used to improve the feature and make it more widely usable.
Sharing your use cases: If you are using sidecars in production, share your experiences with others. This can help to demonstrate the real-world value of the feature and encourage others to adopt it.
Improving the documentation: Help to clarify and expand the documentation for the feature. This can make it easier for others to understand and use sidecars.

In addition to the Sidecar KEP, there are many other areas where SIG Node needs more contributors:

Test coverage: SIG Node is always looking for ways to improve the test coverage of Kubernetes components.
CI maintenance: SIG Node maintains a suite of e2e tests ensuring Kubernetes components function as intended across a variety of scenarios.

Conclusion

In conclusion, SIG Node stands as a cornerstone in Kubernetes’ journey, ensuring its reliability and adaptability in the ever-changing landscape of cloud-native computing. With dedicated members like Matthias, Gunju, and Sergey leading the charge, SIG Node remains at the forefront of innovation, driving Kubernetes towards new horizons.

Blog: Introducing Hydrophone

Thu, 23 May 2024 00:00:00 +0000

In the ever-changing landscape of Kubernetes, ensuring that clusters operate as intended is essential. This is where conformance testing becomes crucial, verifying that a Kubernetes cluster meets the required standards set by the community. Today, we’re thrilled to introduce Hydrophone, a lightweight runner designed to streamline Kubernetes tests using the official conformance images released by the Kubernetes release team.

Simplified Kubernetes testing with Hydrophone

Hydrophone’s design philosophy centers around ease of use. By starting the conformance image as a pod within the conformance namespace, Hydrophone waits for the tests to conclude, then prints and exports the results. This approach offers a hassle-free method for running either individual tests or the entire Conformance Test Suite.

Key features of Hydrophone

Ease of Use: Designed with simplicity in mind, Hydrophone provides an easy-to-use tool for conducting Kubernetes conformance tests.
Official Conformance Images: It leverages the official conformance images from the Kubernetes Release Team, ensuring that you’re using the most up-to-date and reliable resources for testing.
Flexible Test Execution: Whether you need to run a single test, the entire Conformance Test Suite, or anything in between.

Streamlining Kubernetes conformance with Hydrophone

In the Kubernetes world, where providers like EKS, Rancher, and k3s offer diverse environments, ensuring consistent experiences is vital. This consistency is anchored in conformance testing, which validates whether these environments adhere to Kubernetes community standards. Historically, this validation has either been cumbersome or requires third-party tools. Hydrophone offers a simple, single binary tool that streamlines running these essential conformance tests. It’s designed to be user-friendly, allowing for straightforward validation of Kubernetes clusters against community benchmarks, ensuring providers can offer a certified, consistent service.

Hydrophone doesn’t aim to replace the myriad of Kubernetes testing frameworks out there but rather to complement them. It focuses on facilitating conformance tests efficiently, without developing new tests or heavy integration with other tools.

Getting started with Hydrophone

Installing Hydrophone is straightforward. You need a Go development environment; once you have that:

go install sigs.k8s.io/hydrophone@latest

Running hydrophone by default will:

Create a pod, and supporting resources in the conformance namespace on your cluster.
Execute the entire conformance test suite for the cluster version you’re running.
Output the test results and export e2e.log and junit_01.xml needed for conformance validation.

There are supporting flags to specify which tests to run, which to skip, the cluster you’re targeting and much more!

Community and contributions

The Hydrophone project is part of SIG Testing and open to the community for bugs, feature requests, and other contributions. You can engage with the project maintainers via Kubernetes Slack channels #hydrophone, #sig-testing, and #k8s-conformance, or by filing an issue against the repository. We’re also active in the Kubernetes SIG-Testing and SIG-Release Mailing Lists. We encourage pull requests and discussions to make Hydrophone even better.

Join us in simplifying Kubernetes testing

In SIG Testing, we believe Hydrophone will be a valuable tool for anyone looking to validate the conformance of their Kubernetes clusters easily. Whether you’re developing new features, or testing your application, Hydrophone offers an efficient testing experience.

Blog: Spotlight on SIG Architecture: Code Organization

Thu, 11 Apr 2024 00:00:00 +0000

This is the third interview of a SIG Architecture Spotlight series that will cover the different subprojects. We will cover SIG Architecture: Code Organization.

In this SIG Architecture spotlight I talked with Madhav Jivrajani (VMware), a member of the Code Organization subproject.

Introducing the Code Organization subproject

Frederico (FSM): Hello Madhav, thank you for your availability. Could you start by telling us a bit about yourself, your role and how you got involved in Kubernetes?

Madhav Jivrajani (MJ): Hello! My name is Madhav Jivrajani, I serve as a technical lead for SIG Contributor Experience and a GitHub Admin for the Kubernetes project. Apart from that I also contribute to SIG API Machinery and SIG Etcd, but more recently, I’ve been helping out with the work that is needed to help Kubernetes stay on supported versions of Go, and it is through this that I am involved with the Code Organization subproject of SIG Architecture.

FSM: A project the size of Kubernetes must have unique challenges in terms of code organization – is this a fair assumption? If so, what would you pick as some of the main challenges that are specific to Kubernetes?

MJ: That’s a fair assumption! The first interesting challenge comes from the sheer size of the Kubernetes codebase. We have ≅2.2 million lines of Go code (which is steadily decreasing thanks to dims and other folks in this sub-project!), and a little over 240 dependencies that we rely on either directly or indirectly, which is why having a sub-project dedicated to helping out with dependency management is crucial: we need to know what dependencies we’re pulling in, what versions these dependencies are at, and tooling to help make sure we are managing these dependencies across different parts of the codebase in a consistent manner.

Another interesting challenge with Kubernetes is that we publish a lot of Go modules as part of the Kubernetes release cycles, one example of this is client-go.However, we as a project would also like the benefits of having everything in one repository to get the advantages of using a monorepo, like atomic commits… so, because of this, code organization works with other SIGs (like SIG Release) to automate the process of publishing code from the monorepo to downstream individual repositories which are much easier to consume, and this way you won’t have to import the entire Kubernetes codebase!

Code organization and Kubernetes

FSM: For someone just starting contributing to Kubernetes code-wise, what are the main things they should consider in terms of code organization? How would you sum up the key concepts?

MJ: I think one of the key things to keep in mind at least as you’re starting off is the concept of staging directories. In the kubernetes/kubernetes repository, you will come across a directory called staging/. The sub-folders in this directory serve as a bunch of pseudo-repositories. For example, the kubernetes/client-go repository that publishes releases for client-go is actually a staging repo.

FSM: So the concept of staging directories fundamentally impact contributions?

MJ: Precisely, because if you’d like to contribute to any of the staging repos, you will need to send in a PR to its corresponding staging directory in kubernetes/kubernetes. Once the code merges there, we have a bot called the publishing-bot that will sync the merged commits to the required staging repositories (like kubernetes/client-go). This way we get the benefits of a monorepo but we also can modularly publish code for downstream consumption. PS: The publishing-bot needs more folks to help out!

For more information on staging repositories, please see the contributor documentation.

FSM: Speaking of contributions, the very high number of contributors, both individuals and companies, must also be a challenge: how does the subproject operate in terms of making sure that standards are being followed?

MJ: When it comes to dependency management in the project, there is a dedicated team that helps review and approve dependency changes. These are folks who have helped lay the foundation of much of the tooling that Kubernetes uses today for dependency management. This tooling helps ensure there is a consistent way that contributors can make changes to dependencies. The project has also worked on additional tooling to signal statistics of dependencies that is being added or removed: depstat

Apart from dependency management, another crucial task that the project does is management of the staging repositories. The tooling for achieving this (publishing-bot) is completely transparent to contributors and helps ensure that the staging repos get a consistent view of contributions that are submitted to kubernetes/kubernetes.

Code Organization also works towards making sure that Kubernetes stays on supported versions of Go. The linked KEP provides more context on why we need to do this. We collaborate with SIG Release to ensure that we are testing Kubernetes as rigorously and as early as we can on Go releases and working on changes that break our CI as a part of this. An example of how we track this process can be found here.

Release cycle and current priorities

FSM: Is there anything that changes during the release cycle?

MJ During the release cycle, specifically before code freeze, there are often changes that go in that add/update/delete dependencies, fix code that needs fixing as part of our effort to stay on supported versions of Go.

Furthermore, some of these changes are also candidates for backporting to our supported release branches.

FSM: Is there any major project or theme the subproject is working on right now that you would like to highlight?

MJ: I think one very interesting and immensely useful change that has been recently added (and I take the opportunity to specifically highlight the work of Tim Hockin on this) is the introduction of Go workspaces to the Kubernetes repo. A lot of our current tooling for dependency management and code publishing, as well as the experience of editing code in the Kubernetes repo, can be significantly improved by this change.

Wrapping up

FSM: How would someone interested in the topic start helping the subproject?

MJ: The first step, as is the first step with any project in Kubernetes, is to join our slack: slack.k8s.io, and after that join the #k8s-code-organization channel. There is also a code-organization office hours that takes place that you can choose to attend. Timezones are hard, so feel free to also look at the recordings or meeting notes and follow up on slack!

FSM: Excellent, thank you! Any final comments you would like to share?

MJ: The Code Organization subproject always needs help! Especially areas like the publishing bot, so don’t hesitate to get involved in the #k8s-code-organization Slack channel.

Blog: Using Go workspaces in Kubernetes

Tue, 19 Mar 2024 08:30:00 -0800

The Go programming language has played a huge role in the success of Kubernetes. As Kubernetes has grown, matured, and pushed the bounds of what “regular” projects do, the Go project team has also grown and evolved the language and tools. In recent releases, Go introduced a feature called “workspaces” which was aimed at making projects like Kubernetes easier to manage.

We’ve just completed a major effort to adopt workspaces in Kubernetes, and the results are great. Our codebase is simpler and less error-prone, and we’re no longer off on our own technology island.

GOPATH and Go modules

Kubernetes is one of the most visible open source projects written in Go. The earliest versions of Kubernetes, dating back to 2014, were built with Go 1.3. Today, 10 years later, Go is up to version 1.22 — and let’s just say that a whole lot has changed.

In 2014, Go development was entirely based on GOPATH. As a Go project, Kubernetes lived by the rules of GOPATH. In the buildup to Kubernetes 1.4 (mid 2016), we introduced a directory tree called staging. This allowed us to pretend to be multiple projects, but still exist within one git repository (which had advantages for development velocity). The magic of GOPATH allowed this to work.

Kubernetes depends on several code-generation tools which have to find, read, and write Go code packages. Unsurprisingly, those tools grew to rely on GOPATH. This all worked pretty well until Go introduced modules in Go 1.11 (mid 2018).

Modules were an answer to many issues around GOPATH. They gave more control to projects on how to track and manage dependencies, and were overall a great step forward. Kubernetes adopted them. However, modules had one major drawback — most Go tools could not work on multiple modules at once. This was a problem for our code-generation tools and scripts.

Thankfully, Go offered a way to temporarily disable modules (GO111MODULE to the rescue). We could get the dependency tracking benefits of modules, but the flexibility of GOPATH for our tools. We even wrote helper tools to create fake GOPATH trees and played tricks with symlinks in our vendor directory (which holds a snapshot of our external dependencies), and we made it all work.

And for the last 5 years it has worked pretty well. That is, it worked well unless you looked too closely at what was happening. Woe be upon you if you had the misfortune to work on one of the code-generation tools, or the build system, or the ever-expanding suite of bespoke shell scripts we use to glue everything together.

The problems

Like any large software project, we Kubernetes developers have all learned to deal with a certain amount of constant low-grade pain. Our custom staging mechanism let us bend the rules of Go; it was a little clunky, but when it worked (which was most of the time) it worked pretty well. When it failed, the errors were inscrutable and un-Googleable — nobody else was doing the silly things we were doing. Usually the fix was to re-run one or more of the update-* shell scripts in our aptly named hack directory.

As time went on we drifted farther and farher from “regular” Go projects. At the same time, Kubernetes got more and more popular. For many people, Kubernetes was their first experience with Go, and it wasn’t always a good experience.

Our eccentricities also impacted people who consumed some of our code, such as our client library and the code-generation tools (which turned out to be useful in the growing ecosystem of custom resources). The tools only worked if you stored your code in a particular GOPATH-compatible directory structure, even though GOPATH had been replaced by modules more than four years prior.

This state persisted because of the confluence of three factors:

Most of the time it only hurt a little (punctuated with short moments of more acute pain).
Kubernetes was still growing in popularity - we all had other, more urgent things to work on.
The fix was not obvious, and whatever we came up with was going to be both hard and tedious.

As a Kubernetes maintainer and long-timer, my fingerprints were all over the build system, the code-generation tools, and the hack scripts. While the pain of our mess may have been low on_average, I was one of the people who felt it regularly.

Enter workspaces

Along the way, the Go language team saw what we (and others) were doing and didn’t love it. They designed a new way of stitching multiple modules together into a new workspace concept. Once enrolled in a workspace, Go tools had enough information to work in any directory structure and across modules, without GOPATH or symlinks or other dirty tricks.

When I first saw this proposal I knew that this was the way out. This was how to break the logjam. If workspaces was the technical solution, then I would put in the work to make it happen.

The work

Adopting workspaces was deceptively easy. I very quickly had the codebase compiling and running tests with workspaces enabled. I set out to purge the repository of anything GOPATH related. That’s when I hit the first real bump - the code-generation tools.

We had about a dozen tools, totalling several thousand lines of code. All of them were built using an internal framework called gengo, which was built on Go’s own parsing libraries. There were two main problems:

Those parsing libraries didn’t understand modules or workspaces.
GOPATH allowed us to pretend that Go package paths and directories on disk were interchangeable in trivial ways. They are not.

Switching to a modules- and workspaces-aware parsing library was the first step. Then I had to make a long series of changes to each of the code-generation tools. Critically, I had to find a way to do it that was possible for some other person to review! I knew that I needed reviewers who could cover the breadth of changes and reviewers who could go into great depth on specific topics like gengo and Go’s module semantics. Looking at the history for the areas I was touching, I asked Joe Betz and Alex Zielenski (SIG API Machinery) to go deep on gengo and code-generation, Jordan Liggitt (SIG Architecture and all-around wizard) to cover Go modules and vendoring and the hack scripts, and Antonio Ojea (wearing his SIG Testing hat) to make sure the whole thing made sense. We agreed that a series of small commits would be easiest to review, even if the codebase might not actually work at each commit.

Sadly, these were not mechanical changes. I had to dig into each tool to figure out where they were processing disk paths versus where they were processing package names, and where those were being conflated. I made extensive use of the delve debugger, which I just can’t say enough good things about.

One unfortunate result of this work was that I had to break compatibility. The gengo library simply did not have enough information to process packages outside of GOPATH. After discussion with gengo and Kubernetes maintainers, we agreed to make gengo/v2. I also used this as an opportunity to clean up some of the gengo APIs and the tools’ CLIs to be more understandable and not conflate packages and directories. For example you can’t just string-join directory names and assume the result is a valid package name.

Once I had the code-generation tools converted, I shifted attention to the dozens of scripts in the hack directory. One by one I had to run them, debug, and fix failures. Some of them needed minor changes and some needed to be rewritten.

Along the way we hit some cases that Go did not support, like workspace vendoring. Kubernetes depends on vendoring to ensure that our dependencies are always available, even if their source code is removed from the internet (it has happened more than once!). After discussing with the Go team, and looking at possible workarounds, they decided the right path was to implement workspace vendoring.

The eventual Pull Request contained over 200 individual commits.

Results

Now that this work has been merged, what does this mean for Kubernetes users? Pretty much nothing. No features were added or changed. This work was not about fixing bugs (and hopefully none were introduced).

This work was mainly for the benefit of the Kubernetes project, to help and simplify the lives of the core maintainers. In fact, it would not be a lie to say that it was rather self-serving - my own life is a little bit better now.

This effort, while unusually large, is just a tiny fraction of the overall maintenance work that needs to be done. Like any large project, we have lots of “technical debt” — tools that made point-in-time assumptions and need revisiting, internal APIs whose organization doesn’t make sense, code which doesn’t follow conventions which didn’t exist at the time, and tests which aren’t as rigorous as they could be, just to throw out a few examples. This work is often called “grungy” or “dirty”, but in reality it’s just an indication that the project has grown and evolved. I love this stuff, but there’s far more than I can ever tackle on my own, which makes it an interesting way for people to get involved. As our unofficial motto goes: “chop wood and carry water”.

Kubernetes used to be a case-study of how not to do large-scale Go development, but now our codebase is simpler (and in some cases faster!) and more consistent. Things that previously seemed like they should work, but didn’t, now behave as expected.

Our project is now a little more “regular”. Not completely so, but we’re getting closer.

Thanks

This effort would not have been possible without tons of support.

First, thanks to the Go team for hearing our pain, taking feedback, and solving the problems for us.

Special mega-thanks goes to Michael Matloob, on the Go team at Google, who designed and implemented workspaces. He guided me every step of the way, and was very generous with his time, answering all my questions, no matter how dumb.

Writing code is just half of the work, so another special thanks to my reviewers: Jordan Liggitt, Joe Betz, Alexander Zielenski, and Antonio Ojea. These folks brought a wealth of expertise and attention to detail, and made this work smarter and safer.

Blog: Spotlight on SIG Cloud Provider

Fri, 01 Mar 2024 00:00:00 +0000

One of the most popular ways developers use Kubernetes-related services is via cloud providers, but have you ever wondered how cloud providers can do that? How does this whole process of integration of Kubernetes to various cloud providers happen? To answer that, let’s put the spotlight on SIG Cloud Provider.

SIG Cloud Provider works to create seamless integrations between Kubernetes and various cloud providers. Their mission? Keeping the Kubernetes ecosystem fair and open for all. By setting clear standards and requirements, they ensure every cloud provider plays nicely with Kubernetes. It is their responsibility to configure cluster components to enable cloud provider integrations.

In this blog of the SIG Spotlight series, Arujjwal Negi interviews Michael McCune (Red Hat), also known as elmiko, co-chair of SIG Cloud Provider, to give us an insight into the workings of this group.

Introduction

Arujjwal: Let’s start by getting to know you. Can you give us a small intro about yourself and how you got into Kubernetes?

Michael: Hi, I’m Michael McCune, most people around the community call me by my handle, elmiko. I’ve been a software developer for a long time now (Windows 3.1 was popular when I started!), and I’ve been involved with open-source software for most of my career. I first got involved with Kubernetes as a developer of machine learning and data science applications; the team I was on at the time was creating tutorials and examples to demonstrate the use of technologies like Apache Spark on Kubernetes. That said, I’ve been interested in distributed systems for many years and when an opportunity arose to join a team working directly on Kubernetes, I jumped at it!

Functioning and working

Arujjwal: Can you give us an insight into what SIG Cloud Provider does and how it functions?

Michael: SIG Cloud Provider was formed to help ensure that Kubernetes provides a neutral integration point for all infrastructure providers. Our largest task to date has been the extraction and migration of in-tree cloud controllers to out-of-tree components. The SIG meets regularly to discuss progress and upcoming tasks and also to answer questions and bugs that arise. Additionally, we act as a coordination point for cloud provider subprojects such as the cloud provider framework, specific cloud controller implementations, and the Konnectivity proxy project.

Arujjwal: After going through the project README, I learned that SIG Cloud Provider works with the integration of Kubernetes with cloud providers. How does this whole process go?

Michael: One of the most common ways to run Kubernetes is by deploying it to a cloud environment (AWS, Azure, GCP, etc). Frequently, the cloud infrastructures have features that enhance the performance of Kubernetes, for example, by providing elastic load balancing for Service objects. To ensure that cloud-specific services can be consistently consumed by Kubernetes, the Kubernetes community has created cloud controllers to address these integration points. Cloud providers can create their own controllers either by using the framework maintained by the SIG or by following the API guides defined in the Kubernetes code and documentation. One thing I would like to point out is that SIG Cloud Provider does not deal with the lifecycle of nodes in a Kubernetes cluster; for those types of topics, SIG Cluster Lifecycle and the Cluster API project are more appropriate venues.

Important subprojects

Arujjwal: There are a lot of subprojects within this SIG. Can you highlight some of the most important ones and what job they do?

Michael: I think the two most important subprojects today are the cloud provider framework and the extraction/migration project. The cloud provider framework is a common library to help infrastructure integrators build a cloud controller for their infrastructure. This project is most frequently the starting point for new people coming to the SIG. The extraction and migration project is the other big subproject and a large part of why the framework exists. A little history might help explain further: for a long time, Kubernetes needed some integration with the underlying infrastructure, not necessarily to add features but to be aware of cloud events like instance termination. The cloud provider integrations were built into the Kubernetes code tree, and thus the term “in-tree” was created (check out this article on the topic for more info). The activity of maintaining provider-specific code in the main Kubernetes source tree was considered undesirable by the community. The community’s decision inspired the creation of the extraction and migration project to remove the “in-tree” cloud controllers in favor of “out-of-tree” components.

Arujjwal: What makes [the cloud provider framework] a good place to start? Does it have consistent good beginner work? What kind?

Michael: I feel that the cloud provider framework is a good place to start as it encodes the community’s preferred practices for cloud controller managers and, as such, will give a newcomer a strong understanding of how and what the managers do. Unfortunately, there is not a consistent stream of beginner work on this component; this is due in part to the mature nature of the framework and that of the individual providers as well. For folks who are interested in getting more involved, having some Go language knowledge is good and also having an understanding of how at least one cloud API (e.g., AWS, Azure, GCP) works is also beneficial. In my personal opinion, being a newcomer to SIG Cloud Provider can be challenging as most of the code around this project deals directly with specific cloud provider interactions. My best advice to people wanting to do more work on cloud providers is to grow your familiarity with one or two cloud APIs, then look for open issues on the controller managers for those clouds, and always communicate with the other contributors as much as possible.

Accomplishments

Arujjwal: Can you share about an accomplishment(s) of the SIG that you are proud of?

Michael: Since I joined the SIG, more than a year ago, we have made great progress in advancing the extraction and migration subproject. We have moved from an alpha status on the defining KEP to a beta status and are inching ever closer to removing the old provider code from the Kubernetes source tree. I’ve been really proud to see the active engagement from our community members and to see the progress we have made towards extraction. I have a feeling that, within the next few releases, we will see the final removal of the in-tree cloud controllers and the completion of the subproject.

Advice for new contributors

Arujjwal: Is there any suggestion or advice for new contributors on how they can start at SIG Cloud Provider?

Michael: This is a tricky question in my opinion. SIG Cloud Provider is focused on the code pieces that integrate between Kubernetes and an underlying infrastructure. It is very common, but not necessary, for members of the SIG to be representing a cloud provider in an official capacity. I recommend that anyone interested in this part of Kubernetes should come to an SIG meeting to see how we operate and also to study the cloud provider framework project. We have some interesting ideas for future work, such as a common testing framework, that will cut across all cloud providers and will be a great opportunity for anyone looking to expand their Kubernetes involvement.

Arujjwal: Are there any specific skills you’re looking for that we should highlight? To give you an example from our own [SIG ContribEx] (https://github.com/kubernetes/community/blob/master/sig-contributor-experience/README.md): if you’re an expert in Hugo, we can always use some help with k8s.dev!

Michael: The SIG is currently working through the final phases of our extraction and migration process, but we are looking toward the future and starting to plan what will come next. One of the big topics that the SIG has discussed is testing. Currently, we do not have a generic common set of tests that can be exercised by each cloud provider to confirm the behaviour of their controller manager. If you are an expert in Ginkgo and the Kubetest framework, we could probably use your help in designing and implementing the new tests.

This is where the conversation ends. I hope this gave you some insights about SIG Cloud Provider’s aim and working. This is just the tip of the iceberg. To know more and get involved with SIG Cloud Provider, try attending their meetings here.

Blog: A look into the Kubernetes Book Club

Thu, 22 Feb 2024 00:00:00 +0000

Learning Kubernetes and the entire ecosystem of technologies around it is not without its challenges. In this interview, we will talk with Carlos Santana (AWS) to learn a bit more about how he created the Kubernetes Book Club, how it works, and how anyone can join in to take advantage of a community-based learning experience.

Frederico Muñoz (FSM): Hello Carlos, thank you so much for your availability. To start with, could you tell us a bit about yourself?

Carlos Santana (CS): Of course. My experience in deploying Kubernetes in production six years ago opened the door for me to join Knative and then contribute to Kubernetes through the Release Team. Working on upstream Kubernetes has been one of the best experiences I’ve had in open-source. Over the past two years, in my role as a Senior Specialist Solutions Architect at AWS, I have been assisting large enterprises build their internal developer platforms (IDP) on top of Kubernetes. Going forward, my open source contributions are directed towards CNOE and CNCF projects like Argo, Crossplane, and Backstage.

Creating the Book Club

FSM: So your path led you to Kubernetes, and at that point what was the motivating factor for starting the Book Club?

CS: The idea for the Kubernetes Book Club sprang from a casual suggestion during a TGIK livestream. For me, it was more than just about reading a book; it was about creating a learning community. This platform has not only been a source of knowledge but also a support system, especially during the challenging times of the pandemic. It’s gratifying to see how this initiative has helped members cope and grow. The first book Production Kubernetes took 36 weeks, when we started on March 5th 2021. Currently don’t take that long to cover a book, one or two chapters per week.

FSM: Could you describe the way the Kubernetes Book Club works? How do you select the books and how do you go through them?

CS: We collectively choose books based on the interests and needs of the group. This practical approach helps members, especially beginners, grasp complex concepts more easily. We have two weekly series, one for the EMEA timezone, and I organize the US one. Each organizer works with their co-host and picks a book on Slack, then sets up a lineup of hosts for a couple of weeks to discuss each chapter.

FSM: If I’m not mistaken, the Kubernetes Book Club is in its 17th book, which is significant: is there any secret recipe for keeping things active?

CS: The secret to keeping the club active and engaging lies in a couple of key factors.

Firstly, consistency has been crucial. We strive to maintain a regular schedule, only cancelling meetups for major events like holidays or KubeCon. This regularity helps members stay engaged and builds a reliable community.

Secondly, making the sessions interesting and interactive has been vital. For instance, I often introduce pop-up quizzes during the meetups, which not only tests members’ understanding but also adds an element of fun. This approach keeps the content relatable and helps members understand how theoretical concepts are applied in real-world scenarios.

Topics covered in the Book Club

FSM: The main topics of the books have been Kubernetes, GitOps, Security, SRE, and Observability: is this a reflection of the cloud native landscape, especially in terms of popularity?

CS: Our journey began with ‘Production Kubernetes’, setting the tone for our focus on practical, production-ready solutions. Since then, we’ve delved into various aspects of the CNCF landscape, aligning our books with a different theme. Each theme, whether it be Security, Observability, or Service Mesh, is chosen based on its relevance and demand within the community. For instance, in our recent themes on Kubernetes Certifications, we brought the book authors into our fold as active hosts, enriching our discussions with their expertise.

FSM: I know that the project had recent changes, namely being integrated into the CNCF as a Cloud Native Community Group. Could you talk a bit about this change?

CS: The CNCF graciously accepted the book club as a Cloud Native Community Group. This is a significant development that has streamlined our operations and expanded our reach. This alignment has been instrumental in enhancing our administrative capabilities, similar to those used by Kubernetes Community Days (KCD) meetups. Now, we have a more robust structure for memberships, event scheduling, mailing lists, hosting web conferences, and recording sessions.

FSM: How has your involvement with the CNCF impacted the growth and engagement of the Kubernetes Book Club over the past six months?

CS: Since becoming part of the CNCF community six months ago, we’ve witnessed significant quantitative changes within the Kubernetes Book Club. Our membership has surged to over 600 members, and we’ve successfully organized and conducted more than 40 events during this period. What’s even more promising is the consistent turnout, with an average of 30 attendees per event. This growth and engagement are clear indicators of the positive influence of our CNCF affiliation on the Kubernetes Book Club’s reach and impact in the community.

Joining the Book Club

FSM: For anyone wanting to join, what should they do?

CS: There are three steps to join:

First, join the Kubernetes Book Club Community
Then RSVP to the events on the community page
Lastly, join the CNCF Slack channel #kubernetes-book-club.

FSM: Excellent, thank you! Any final comments you would like to share?

CS: The Kubernetes Book Club is more than just a group of professionals discussing books; it’s a vibrant community and amazing volunteers that help organize and host Neependra Khare, Eric Smalling, Sevi Karakulak, Chad M. Crowell, and Walid (CNJ) Shaari. Look us up at KubeCon and get your Kubernetes Book Club sticker!

Blog: Spotlight on SIG Release (Release Team Subproject)

Mon, 15 Jan 2024 00:00:00 +0000

The Release Special Interest Group (SIG Release), where Kubernetes sharpens its blade with cutting-edge features and bug fixes every 4 months. Have you ever considered how such a big project like Kubernetes manages its timeline so efficiently to release its new version, or how the internal workings of the Release Team look like? If you’re curious about these questions or want to know more and get involved with the work SIG Release does, read on!

SIG Release plays a crucial role in the development and evolution of Kubernetes. Its primary responsibility is to manage the release process of new versions of Kubernetes. It operates on a regular release cycle, typically every three to four months. During this cycle, the Kubernetes Release Team works closely with other SIGs and contributors to ensure a smooth and well-coordinated release. This includes planning the release schedule, setting deadlines for code freeze and testing phases, as well as creating release artefacts like binaries, documentation, and release notes.

Before you read further, it is important to note that there are two subprojects under SIG Release - Release Engineering and Release Team.

In this blog post, Nitish Kumar interviews Verónica López (PlanetScale), Technical Lead of SIG Release, with the spotlight on the Release Team subproject, how the release process looks like, and ways to get involved.

What is the typical release process for a new version of Kubernetes, from initial planning to the final release? Are there any specific methodologies and tools that you use to ensure a smooth release?

The release process for a new Kubernetes version is a well-structured and community-driven effort. There are no specific methodologies or tools as such that we follow, except a calendar with a series of steps to keep things organised. The complete release process looks like this:

Release Team Onboarding: We start with the formation of a Release Team, which includes volunteers from the Kubernetes community who will be responsible for managing different components of the new release. This is typically done before the previous release is about to wrap up. Once the team is formed, new members are onboarded while the Release Team Lead and the Branch Manager propose a calendar for the usual deliverables. As an example, you can take a look at the v1.29 team formation issue created at the SIG Release repository. For a contributor to be the part of Release Team, they typically go through the Release Shadow program, but that’s not the only way to get involved with SIG Release.
Beginning Phase: In the initial weeks of each release cycle, SIG Release diligently tracks the progress of new features and enhancements outlined in Kubernetes Enhancement Proposals (KEPs). While not all of these features are entirely new, they often commence their journey in the alpha phase, subsequently advancing to the beta stage, and ultimately attaining the status of stability.
Feature Maturation Phase: We usually cut a couple of Alpha releases, containing new features in an experimental state, to gather feedback from the community, followed by a couple of Beta releases, where features are more stable and the focus is on fixing bugs. Feedback from users is critical at this stage, to the point where sometimes we need to cut an additional Beta release to address bugs or other concerns that may arise during this phase. Once this is cleared, we cut a release candidate (RC) before the actual release. Throughout the cycle, efforts are made to update and improve documentation, including release notes and user guides, a process that, in my opinion, deserves its own post.
Stabilisation Phase: A few weeks before the new release, we implement a code freeze, and no new features are allowed after this point: this allows the focus to shift towards testing and stabilisation. In parallel to the main release, we keep cutting monthly patches of old, officially supported versions of Kubernetes, so you could say that the lifecycle of a Kubernetes version extends for several months afterwards. Throughout the complete release cycle, efforts are made to update and improve documentation, including release notes and user guides, a process that, in our opinion, deserves its own post.

How do you handle the balance between stability and introducing new features in each release? What criteria are used to determine which features make it into a release?

It’s a neverending mission, however, we think that the key is in respecting our process and guidelines. Our guidelines are the result of hours of discussions and feedback from dozens of members of the community who bring a wealth of knowledge and experience to the project. If we didn’t have strict guidelines, we would keep having the same discussions over and over again, instead of using our time for more productive topics that needs our attention. All the critical exceptions require consensus from most of the team members, so we can ensure quality.

The process of deciding what makes it into a release starts way before the Release Teams takes over the workflows. Each individual SIG along with the most experienced contributors gets to decide whether they’d like to include a feature or change, so the planning and ultimate approval usually belongs to them. Then, the Release Team makes sure those contributions meet the requirements of documentation, testing, backwards compatibility, among others, before officially allowing them in. A similar process happens with cherry-picks for the monthly patch releases, where we have strict policies about not accepting PRs that would require a full KEP, or fixes that don’t include all the affected branches.
What are some of the most significant challenges you’ve encountered while developing and releasing Kubernetes? How have you overcome these challenges?

Every cycle of release brings its own array of challenges. It might involve tackling last-minute concerns like newly discovered Common Vulnerabilities and Exposures (CVEs), resolving bugs within our internal tools, or addressing unexpected regressions caused by features from previous releases. Another obstacle we often face is that, although our team is substantial, most of us contribute on a volunteer basis. Sometimes it can feel like we’re a bit understaffed, however we always manage to get organised and make it work.
As a new contributor, what should be my ideal path to get involved with SIG Release? In a community where everyone is busy with their own tasks, how can I find the right set of tasks to contribute effectively to it?

Everyone’s way of getting involved within the Open Source community is different. SIG Release is a self-serving team, meaning that we write our own tools to be able to ship releases. We collaborate a lot with other SIGs, such as SIG K8s Infra, but all the tools that we used needs to be tailor-made for our massive technical needs, while reducing costs. This means that we are constantly looking for volunteers who’d like to help with different types of projects, beyond “just” cutting a release.

Our current project requires a mix of skills like Go programming, understanding Kubernetes internals, Linux packaging, supply chain security, technical writing, and general open-source project maintenance. This skill set is always evolving as our project grows.

For an ideal path, this is what we suggest:
- Get yourself familiar with the code, including how features are managed, the release calendar, and the overall structure of the Release Team.
- Join the Kubernetes community communication channels, such as Slack (#sig-release), where we are particularly active.
- Join the SIG Release weekly meetings which are open to all in the community. Participating in these meetings is a great way to learn about ongoing and future projects that you might find relevant for your skillset and interests.
Remember, every experienced contributor was once in your shoes, and the community is often more than willing to guide and support newcomers. Don’t hesitate to ask questions, engage in discussions, and take small steps to contribute.
What is the Release Shadow Program and how is it different from other shadow programs included in various other SIGs?

The Release Shadow Program offers a chance for interested individuals to shadow experienced members of the Release Team throughout a Kubernetes release cycle. This is a unique chance to see all the hard work that a Kubernetes release requires across sub-teams. A lot of people think that all we do is cut a release every three months, but that’s just the top of the iceberg.

Our program typically aligns with a specific Kubernetes release cycle, which has a predictable timeline of approximately three months. While this program doesn’t involve writing new Kubernetes features, it still requires a high sense of responsibility since the Release Team is the last step between a new release and thousands of contributors, so it’s a great opportunity to learn a lot about modern software development cycles at an accelerated pace.
What are the qualifications that you generally look for in a person to volunteer as a release shadow/release lead for the next Kubernetes release?

While all the roles require some degree of technical ability, some require more hands-on experience with Go and familiarity with the Kubernetes API while others require people who are good at communicating technical content in a clear and concise way. It’s important to mention that we value enthusiasm and commitment over technical expertise from day 1. If you have the right attitude and show us that you enjoy working with Kubernetes and or/release engineering, even if it’s only through a personal project that you put together in your spare time, the team will make sure to guide you. Being a self-starter and not being afraid to ask questions can take you a long way in our team.
What will you suggest to someone who has got rejected from being a part of the Release Shadow Program several times?

Keep applying.

With every release cycle we have had an exponential growth in the number of applicants, so it gets harder to be selected, which can be discouraging, but please know that getting rejected doesn’t mean you’re not talented. It’s just practically impossible to accept every applicant, however here’s an alternative that we suggest:

Start attending our weekly Kubernetes SIG Release meetings to introduce yourself and get familiar with the team and the projects we are working on.

The Release Team is one of the way to join SIG Release, but we are always looking for more hands to help. Again, in addition to certain technical ability, the most sought after trait that we look for is people we can trust, and that requires time.
Can you discuss any ongoing initiatives or upcoming features that the release team is particularly excited about for Kubernetes v1.28? How do these advancements align with the long-term vision of Kubernetes?

We are excited about finally publishing Kubernetes packages on community infrastructure. It has been something that we have been wanting to do for a few years now, but it’s a project with many technical implications that must be in place before doing the transition. Once that’s done, we’ll be able to increase our productivity and take control of the entire workflows.

Final thoughts

Well, this conversation ends here but not the learning. I hope this interview has given you some idea about what SIG Release does and how to get started in helping out. It is important to mention again that this article covers the first subproject under SIG Release, the Release Team. In the next Spotlight blog on SIG Release, we will provide a spotlight on the Release Engineering subproject, what it does and how to get involved. Finally, you can go through the SIG Release charter to get a more in-depth understanding of how SIG Release operates.

Blog: Blixt - A load-balancer written in Rust, using eBPF, born from Gateway API

Mon, 08 Jan 2024 00:00:00 +0000

In SIG Network we now have a layer 4 (“L4”) load balancer named Blixt. This project started as a fun experiment using emerging technologies and is intended to become a utility for CI and testing to help facilitate the continued development of Gateway API. Are you interested in developing networking tools in Rust and eBPF? Or perhaps you’re specifically interested in Gateway API? We’ll tell you a bit about the project and how it might benefit you.

History

Blixt originated at Kong as an experiment to test load-balancing ingress traffic for Kubernetes clusters using eBPF for the dataplane. Around the time of Kubecon Detroit (2022) we (the Gateway API maintainers) realized it had significant potential to help us move our TCPRoute and UDPRoute support forward, which had been sort of “stuck in alpha” at the time due to a lack of conformance tests being developed for them. At the same time, various others in the SIG Network community developed an interest in the project due to the rapid growth of eBPFs use on Kubernetes. Given the potential for benefit to the Kubernetes ecosystem and the growing interest, Kong decided it would be helpful to donate the project to Kubernetes SIGs to benefit upstream Kubernetes.

Over several months we rewrote the project in Rust (from C), due to a strong contingency of Rust knowledge (and interest) between us developing the project and an active interest in the burgeoning Aya project (a Rust framework for developing eBPF programs). We did eventually move the control plane (specifically) to Golang however, so that we could take advantage of the Kubebuilder and controller-runtime ecosystems. Additionally, we augmented our custom program loader (in eBPF, you generally write loaders that load your BPF byte code into the kernel) with bpfman: a project adjacent to us in the Rust + eBPF ecosystem, which helps solve several security and ergonomic problems with managing BPF programs on Linux systems.

After the recently completed license review process, which provided a blanket exception for the use of dual licensed eBPF in CNCF code, the project became officially part of Kubernetes and interest has been growing. We have several goals for the project which revolve around the continued development of Gateway API, with a specific focus on helping mature Layer 4 support (e.g. the UDPRoute and TCPRoute API kinds).

Goals

Currently the high level goal of the project is to provide a Gateway API driven load-balancer for non-production use cases. Those non-production use cases include:

Driving conformance tests and adoption for L4 use cases.
Using this implementation as part of the Gateway API CI testing strategy.
Having the Blixt control-plane be a reference implementation.
Exploring issues associated with the generic use of eBPF in Kubernetes.

In support of those goals we have some more specific sub-goals we’re actively working towards:

Support GatewayClass and Gateway, meeting conformance requirements
Support UDPRoute and TCPRoute, meanwhile helping to develop the conformance requirements for these APIs.

We have some significant progress on the above in that we have a basic level of support for creating a GatewayClass and at least one Gateway, and then attaching UDPRoute and TCPRoute resources to that Gateway. Doing that drives a process of the underlying data plane receiving corresponding configuration and traffic then flowing as expected. We emphasize the basic: as the project is still quite early on, and being developed in a highly iterative fashion. That said the fundamentals are there and you can try them out yourself on a local system using our usage documentation and sample configurations. You can see more about the project’s current status on the README.md including the milestones and current progress.

One thing that can’t be overstated about this project is that it has been at the center of a lot of learning, community building and fun. We have maintained a policy with this project that it shall never be intended for production use cases which means development of the project is more of a sandbox and a safe space for people to learn and experiment. If any of this sounds interesting to you, now is a great time to get involved!

Getting involved

If you’re interested in networking, Rust, Linux, eBPF (or all of the above) there’s a lot of opportunity here to learn and have fun. We invite you to jump right in on the repository if that’s your style, or reach out to us in the community: You can reach us on Kubernetes Slack on the #sig-network-gateway-api channel as well as the #ebpf channel. Blixt is a topic of discussion at the Gateway API community meetings, and the monthly SIG Network Code Jam as well.

Blog: Kubernetes supports running kube-proxy in an unprivileged container

Fri, 05 Jan 2024 00:00:00 +0000

This post describes how the --init-only flag to kube-proxy can be used to run the main kube-proxy container in a stricter securityContext, by performing the configuration that requires privileged mode in a separate init container. Since Windows doesn’t have the equivalent of capabilities, this only works on Linux.

The kube-proxy Pod still only meets the privileged Pod Security Standard, but there is still an improvement because the running container doesn’t need to run privileged.

Please note that kube-proxy can be installed in different ways. The examples below assume that kube-proxy is run from a pod, but similar changes could be made in clusters where it is run as a system service.

Background

It is undesirable to run a server container like kube-proxy in privileged mode. Security aware users wants to use capabilities instead.

If kube-proxy is installed as a POD, the initialization requires “privileged” mode, mostly for setting sysctl’s. However, kube-proxy only tries to set the sysctl’s if they don’t already have the right values. In theory, then, if a privileged init container set the sysctls to the right values, then kube-proxy could run unprivileged.

The problem is to know what to setup. Until now the only option has been to read the source to see what changes kube-proxy would have made, but with --init-only you can have kube-proxy itself do the setup exactly as on a normal start, and then exit.

Initializing kube-proxy in an init container

The example manifests below are not complete, but narrowed down to what is essential to illustrate the function.

Usually, cluster operators run kube-proxy in a privileged security context.

apiVersion: apps/v1
kind: DaemonSet
metadata:
 labels:
 k8s-app: kube-proxy
spec:
 template:
 spec:
 containers:
 - name: kube-proxy
 command:
 - /usr/local/bin/kube-proxy
 - --config=/var/lib/kube-proxy/config.conf
 - --hostname-override=$(NODE_NAME)
 securityContext:
 privileged: true
 # (lots of stuff omitted here...)

But now it is possible to use:

apiVersion: apps/v1
kind: DaemonSet
metadata:
 labels:
 k8s-app: kube-proxy
spec:
 template:
 spec:
 initContainers:
 - name: kube-proxy-init
 command:
 - /usr/local/bin/kube-proxy
 - --config=/var/lib/kube-proxy/config.conf
 - --hostname-override=$(NODE_NAME)
 - --init-only
 securityContext:
 privileged: true
 # (lots of stuff omitted here...)
 containers:
 - name: kube-proxy
 command:
 - /usr/local/bin/kube-proxy
 - --config=/var/lib/kube-proxy/config.conf
 - --hostname-override=$(NODE_NAME)
 securityContext:
 capabilities:
 add: ["NET_ADMIN"]
 # (lots of stuff omitted here...)

Summary

The --init-only flag can be used to perform privileged initialization in an init container and run the main container with NET_ADMIN capabilities only. Installers like kubeadm will likely be altered to use this feature in the future.

Blog: Contextual logging in Kubernetes 1.29: Better troubleshooting and enhanced logging

Wed, 20 Dec 2023 09:30:00 -0800

On behalf of the Structured Logging Working Group and SIG Instrumentation, we are pleased to announce that the contextual logging feature introduced in Kubernetes v1.24 has now been successfully migrated to two components (kube-scheduler and kube-controller-manager) as well as some directories. This feature aims to provide more useful logs for better troubleshooting of Kubernetes and to empower developers to enhance Kubernetes.

What is contextual logging?

Contextual logging is based on the go-logr API. The key idea is that libraries are passed a logger instance by their caller and use that for logging instead of accessing a global logger. The binary decides the logging implementation, not the libraries. The go-logr API is designed around structured logging and supports attaching additional information to a logger.

This enables additional use cases:

The caller can attach additional information to a logger:
- WithName adds a “logger” key with the names concatenated by a dot as value
- WithValues adds key/value pairs
When passing this extended logger into a function, and the function uses it instead of the global logger, the additional information is then included in all log entries, without having to modify the code that generates the log entries. This is useful in highly parallel applications where it can become hard to identify all log entries for a certain operation, because the output from different operations gets interleaved.
When running unit tests, log output can be associated with the current test. Then, when a test fails, only the log output of the failed test gets shown by go test. That output can also be more verbose by default because it will not get shown for successful tests. Tests can be run in parallel without interleaving their output.

One of the design decisions for contextual logging was to allow attaching a logger as value to a context.Context. Since the logger encapsulates all aspects of the intended logging for the call, it is part of the context, and not just using it. A practical advantage is that many APIs already have a ctx parameter or can add one. This provides additional advantages, like being able to get rid of context.TODO() calls inside the functions.

How to use it

The contextual logging feature is alpha starting from Kubernetes v1.24, so it requires the ContextualLogging feature gate to be enabled. If you want to test the feature while it is alpha, you need to enable this feature gate on the kube-controller-manager and the kube-scheduler.

For the kube-scheduler, there is one thing to note, in addition to enabling the ContextualLogging feature gate, instrumentation also depends on log verbosity. To avoid slowing down the scheduler with the logging instrumentation for contextual logging added for 1.29, it is important to choose carefully when to add additional information:

At -v3 or lower, only WithValues("pod") is used once per scheduling cycle. This has the intended effect that all log messages for the cycle include the pod information. Once contextual logging is GA, “pod” key/value pairs can be removed from all log calls.
At -v4 or higher, richer log entries get produced where WithValues is also used for the node (when applicable) and WithName is used for the current operation and plugin.

Here is an example that demonstrates the effect:

I1113 08:43:37.029524 87144 default_binder.go:53] “Attempting to bind pod to node” logger=“Bind.DefaultBinder” pod=“kube-system/coredns-69cbfb9798-ms4pq” node=“127.0.0.1”

The immediate benefit is that the operation and plugin name are visible in logger. pod and node are already logged as parameters in individual log calls in kube-scheduler code. Once contextual logging is supported by more packages outside of kube-scheduler, they will also be visible there (for example, client-go). Once it is GA, log calls can be simplified to avoid repeating those values.

In kube-controller-manager, WithName is used to add the user-visible controller name to log output, for example:

I1113 08:43:29.284360 87141 graph_builder.go:285] “garbage controller monitor not synced: no monitors” logger=“garbage-collector-controller”

The logger=”garbage-collector-controller” was added by the kube-controller-manager core when instantiating that controller and appears in all of its log entries - at least as long as the code that it calls supports contextual logging. Further work is needed to convert shared packages like client-go.

Performance impact

Supporting contextual logging in a package, i.e. accepting a logger from a caller, is cheap. No performance impact was observed for the kube-scheduler. As noted above, adding WithName and WithValues needs to be done more carefully.

In Kubernetes 1.29, enabling contextual logging at production verbosity (-v3 or lower) caused no measurable slowdown for the kube-scheduler and is not expected for the kube-controller-manager either. At debug levels, a 28% slowdown for some test cases is still reasonable given that the resulting logs make debugging easier. For details, see the discussion around promoting the feature to beta.

Impact on downstream users

Log output is not part of the Kubernetes API and changes regularly in each release, whether it is because developers work on the code or because of the ongoing conversion to structured and contextual logging.

If downstream users have dependencies on specific logs, they need to be aware of how this change affects them.

Get involved

If you’re interested in getting involved, we always welcome new contributors to join us. Contextual logging provides a fantastic opportunity for you to contribute to Kubernetes development and make a meaningful impact. By joining Structured Logging WG, you can actively participate in the development of Kubernetes and make your first contribution. It’s a great way to learn and engage with the community while gaining valuable experience.

We encourage you to explore the repository and familiarize yourself with the ongoing discussions and projects. It’s a collaborative environment where you can exchange ideas, ask questions, and work together with other contributors.

If you have any questions or need guidance, don’t hesitate to reach out to us and you can do so on our public Slack channel. If you’re not already part of that Slack workspace, you can visit https://slack.k8s.io/ for an invitation.

We would like to express our gratitude to all the contributors who provided excellent reviews, shared valuable insights, and assisted in the implementation of this feature (in alphabetical order):

Aldo Culquicondor (alculquicondor)
Andy Goldstein (ncdc)
Feruzjon Muyassarov (fmuyassarov)
Freddie (freddie400)
JUN YANG (yangjunmyfm192085)
Kante Yin (kerthcet)
Kiki (carlory)
Lucas Severo Alve (knelasevero)
Maciej Szulik (soltysh)
Mengjiao Liu (mengjiao-liu)
Naman Lakhwani (Namanl2001)
Oksana Baranova (oxxenix)
Patrick Ohly (pohly)
songxiao-wang87 (songxiao-wang87)
Tim Allclai (tallclair)
ZhangYu (Octopusjust)
Ziqi Zhao (fatsheep9146)
Zac (249043822)

Blog: Spotlight on SIG Testing

Fri, 24 Nov 2023 00:00:00 +0000

Welcome to another edition of the SIG spotlight blog series, where we highlight the incredible work being done by various Special Interest Groups (SIGs) within the Kubernetes project. In this edition, we turn our attention to SIG Testing, a group interested in effective testing of Kubernetes and automating away project toil. SIG Testing focus on creating and running tools and infrastructure that make it easier for the community to write and run tests, and to contribute, analyze and act upon test results.

To gain some insights into SIG Testing, Sandipan Panda spoke with Michelle Shepardson, a senior software engineer at Google and a chair of SIG Testing, and Patrick Ohly, a software engineer and architect at Intel and a SIG Testing Tech Lead.

Meet the contributors

Sandipan: Could you tell us a bit about yourself, your role, and how you got involved in the Kubernetes project and SIG Testing?

Michelle: Hi! I’m Michelle, a senior software engineer at Google. I first got involved in Kubernetes through working on tooling for SIG Testing, like the external instance of TestGrid. I’m part of oncall for TestGrid and Prow, and am now a chair for the SIG.

Patrick: Hello! I work as a software engineer and architect in a team at Intel which focuses on open source Cloud Native projects. When I ramped up on Kubernetes to develop a storage driver, my very first question was “how do I test it in a cluster and how do I log information?” That interest led to various enhancement proposals until I had (re)written enough code that also took over official roles as SIG Testing Tech Lead (for the E2E framework) and structured logging WG lead.

Testing practices and tools

Sandipan: Testing is a field in which multiple approaches and tools exist; how did you arrive at the existing practices?

Patrick: I can’t speak about the early days because I wasn’t around yet 😆, but looking back at some of the commit history it’s pretty obvious that developers just took what was available and started using it. For E2E testing, that was Ginkgo+Gomega. Some hacks were necessary, for example around cleanup after a test run and for categorising tests. Eventually this led to Ginkgo v2 and revised best practices for E2E testing. Regarding unit testing opinions are pretty diverse: some maintainers prefer to use just the Go standard library with hand-written checks. Others use helper packages like stretchr/testify. That diversity is okay because unit tests are self-contained - contributors just have to be flexible when working on many different areas. Integration testing falls somewhere in the middle. It’s based on Go unit tests, but needs complex helper packages to bring up an apiserver and other components, then runs tests that are more like E2E tests.

Subprojects owned by SIG Testing

Sandipan: SIG Testing is pretty diverse. Can you give a brief overview of the various subprojects owned by SIG Testing?

Michelle: Broadly, we have subprojects related to testing frameworks, and infrastructure, though they definitely overlap. So for the former, there’s e2e-framework (used externally), test/e2e/framework (used for Kubernetes itself) and kubetest2 for end-to-end testing, as well as boskos (resource rental for e2e tests), KIND (Kubernetes-in-Docker, for local testing and development), and the cloud provider for KIND. For the latter, there’s Prow (K8s-based CI/CD and chatops), and a litany of other tools and utilities for triage, analysis, coverage, Prow/TestGrid config generation, and more in the test-infra repo.

If you are willing to learn more and get involved with any of the SIG Testing subprojects, check out the SIG Testing README.

Key challenges and accomplishments

Sandipan: What are some of the key challenges you face?

Michelle: Kubernetes is a gigantic project in every aspect, from contributors to code to users and more. Testing and infrastructure have to meet that scale, keeping up with every change from every repo under Kubernetes while facilitating developing, improving, and releasing the project as much as possible, though of course, we’re not the only SIG involved in that. I think another other challenge is staffing subprojects. SIG Testing has a number of subprojects that have existed for years, but many of the original maintainers for them have moved on to other areas or no longer have the time to maintain them. We need to grow long-term expertise and owners in those subprojects.

Patrick: As Michelle said, the sheer size can be a challenge. It’s not just the infrastructure, also our processes must scale with the number of contributors. It’s good to document best practices, but not good enough: we have many new contributors, which is good, but having reviewers explain best practices doesn’t scale - assuming that the reviewers even know about them! It also doesn’t help that existing code cannot get updated immediately because there is so much of it, in particular for E2E testing. The initiative to apply stricter linting to new or modified code while accepting that existing code doesn’t pass those same linter checks helps a bit.

Sandipan: Any SIG accomplishments that you are proud of and would like to highlight?

Patrick: I am biased because I have been driving this, but I think that the E2E framework and linting are now in a much better shape than they used to be. We may soon be able to run integration tests with race detection enabled, which is important because we currently only have that for unit tests and those tend to be less complex.

Sandipan: Testing is always important, but is there anything specific to your work in terms of the Kubernetes release process?

Patrick: test flakes… if we have too many of those, development velocity goes down because PRs cannot be merged without clean test runs and those become less likely. Developers also lose trust in testing and just “retest” until they have a clean run, without checking whether failures might indeed be related to a regression in their current change.

The people and the scope

Sandipan: What are some of your favourite things about this SIG?

Michelle: The people, of course 🙂. Aside from that, I like the broad scope SIG Testing has. I feel like even small changes can make a big difference for fellow contributors, and even if my interests change over time, I’ll never run out of projects to work on.

Patrick: I can work on things that make my life and the life of my fellow developers better, like the tooling that we have to use every day while working on some new feature elsewhere.

Sandipan: Are there any funny / cool / TIL anecdotes that you could tell us?

Patrick: I started working on E2E framework enhancements five years ago, then was less active there for a while. When I came back and wanted to test some new enhancement, I asked about how to write unit tests for the new code and was pointed to some existing tests which looked vaguely familiar, as if I had seen them before. I looked at the commit history and found that I had written them! I’ll let you decide whether that says something about my failing long-term memory or simply is normal… Anyway, folks, remember to write good commit messages and comments; someone will need them at some point - it might even be yourself!

Looking ahead

Sandipan: What areas and/or subprojects does your SIG need help with?

Michelle: Some subprojects aren’t staffed at the moment and could use folks willing to learn more about them. boskos and kubetest2 especially stand out to me, since both are important for testing but lack dedicated owners.

Sandipan: Are there any useful skills that new contributors to SIG Testing can bring to the table? What are some things that people can do to help this SIG if they come from a background that isn’t directly linked to programming?

Michelle: I think user empathy, writing clear feedback, and recognizing patterns are really useful. Someone who uses the test framework or tooling and can outline pain points with clear examples, or who can recognize a wider issue in the project and pull data to inform solutions for it.

Sandipan: What’s next for SIG Testing?

Patrick: Stricter linting will soon become mandatory for new code. There are several E2E framework sub-packages that could be modernised, if someone wants to take on that work. I also see an opportunity to unify some of our helper code for E2E and integration testing, but that needs more thought and discussion.

Michelle: I’m looking forward to making some usability improvements for some of our tools and infra, and to supporting more long-term contributions and growth of contributors into long-term roles within the SIG. If you’re interested, hit us up!

Looking ahead, SIG Testing has exciting plans in store. You can get in touch with the folks at SIG Testing in their Slack channel or attend one of their regular bi-weekly meetings on Tuesdays. If you are interested in making it easier for the community to run tests and contribute test results, to ensure Kubernetes is stable across a variety of cluster configurations and cloud providers, join the SIG Testing community today!

Blog: Kubernetes Contributor Summit: Behind-the-scenes

Fri, 03 Nov 2023 00:00:00 +0000

Every year, just before the official start of KubeCon+CloudNativeCon, there’s a special event that has a very special place in the hearts of those organizing and participating in it: the Kubernetes Contributor Summit. To find out why, and to provide a behind-the-scenes perspective, we interview Noah Abrahams, whom amongst other roles was the co-lead for the Kubernetes Contributor Summit in 2023.

Frederico Muñoz (FSM): Hello Noah, and welcome. Could you start by introducing yourself and telling us how you got involved in Kubernetes?

Noah Abrahams (NA): I’ve been in this space for quite a while. I got started in IT in the mid 90’s, and I’ve been working in the “Cloud” space for about 15 years. It was, frankly, through a combination of sheer luck (being in the right place at the right time) and having good mentors to pull me into those places (thanks, Tim!), that I ended up at a startup called Apprenda in 2016. While I was there, they pivoted into Kubernetes, and it was the best thing that could have happened to my career. It was around v1.2 and someone asked me if I could give a presentation on Kubernetes concepts at “my local meetup” in Las Vegas. The meetup didn’t exist yet, so I created it, and got involved in the wider community. One thing led to another, and soon I was involved in ContribEx, joined the release team, was doing booth duty for the CNCF, became an ambassador, and here we are today.

The Contributor Summit

FM: Before leading the organisation of the KCSEU 2023, how many other Contributor Summits were you a part of?

NA: I was involved in four or five before taking the lead. If I’m recalling correctly, I attended the summit in Copenhagen, then sometime in 2018 I joined the wrong meeting, because the summit staff meeting was listed on the ContribEx calendar. Instead of dropping out of the call, I listened a bit, then volunteered to take on some work that didn’t look like it had anybody yet dedicated to it. I ended up running Ops in Seattle and helping run the New Contributor Workshop in Shanghai, that year. Since then, I’ve been involved in all but two, since I missed both Barcelona and Valencia.

FM: Have you noticed any major changes in terms of how the conference is organized throughout the years? Namely in terms of number of participants, venues, speakers, themes…

NA: The summit changes over the years with the ebb and flow of the desires of the contributors that attend. While we can typically expect about the same number of attendees, depending on the region that the event is held in, we adapt the style and content greatly based on the feedback that we receive at the end of each event. Some years, contributors ask for more free-style or unconference type sessions, and we plan on having more of those, but some years, people ask for more planned sessions or workshops, so that’s what we facilitate. We also have to continually adapt to the venue that we have, the number of rooms we’re allotted, how we’re going to share the space with other events and so forth. That all goes into the planning ahead of time, from how many talk tracks we’ll have, to what types of tables and how many microphones we want in a room.

There has been one very significant change over the years, though, and that is that we no longer run the New Contributor Workshop. While the content was valuable, running the session during the summit never led to any people who weren’t already contributing to the project becoming dedicated contributors to the project, so we removed it from the schedule. We’ll deliver that content another way, while we’ll keep the summit focused on existing contributors.

What makes it special

FM: Going back to the introduction I made, I’ve heard several participants saying that KubeCon is great, but that the Contributor Summit is for them the main event. In your opinion, why do you think that makes it so?

NA: I think part of it ties into what I mentioned a moment ago, the flexibility in our content types. For many contributors, I think the summit is basically “How Kubecon used to be”, back when it was primarily a gathering of the contributors to talk about the health of the project and the work that needed to be done. So, in that context, if the contributors want to discuss, say, a new Working Group, then they have dedicated space to do so in the summit. They also have the space to sit down and hack on a tough problem, discuss architectural philosophy, bring potential problems to more people’s attention, refine our methods, and so forth. Plus, the unconference aspect allows for some malleability on the day-of, for whatever is most important right then and there. Whatever folks want to get out of this environment is what we’ll provide, and having a space and time specifically to address your particular needs is always going to be well received.

Let’s not forget the social aspect, too. Despite the fact that we’re a global community and work together remotely and asynchronously, it’s still easier to work together when you have a personal connection, and can put a face to a Github handle. Zoom meetings are a good start, but even a single instance of in-person time makes a big difference in how people work together. So, getting folks together a couple times a year makes the project run more smoothly.

Organizing the Summit

FM: In terms of the organization team itself, could you share with us a general overview of the staffing process? Who are the people that make it happen? How many different teams are involved?

NA: There’s a bit of the “usual suspects” involved in making this happen, many of whom you’ll find in the ContribEx meetings, but really it comes down to whoever is going to step up and do the work. We start with a general call out for volunteers from the org. There’s a Github issue where we’ll track the staffing and that will get shouted out to all the usual comms channels: slack, k-dev, etc.

From there, there’s a handful of different teams, overseeing content/program committee, registration, communications, day-of operations, the awards the SIGs present to their members, the after-summit social event, and so on. The leads for each team/role are generally picked from folks who have stepped up and worked the event before, either as a shadow, or a previous lead, so we know we can rely on them, which is a recurring theme. The leads pick their shadows from whoever pipes up on the issue, and the teams move forward, operating according to their role books, which we try to update at the end of each summit, with what we’ve learned over the past few months. It’s expected that a shadow will be in line to lead that role at some point in a future summit, so we always have a good bench of folks available to make this event happen. A couple of the roles also have some non-shadow volunteers where people can step in to help a bit, like as an on-site room monitor, and get a feel for how things are put together without having to give a serious up-front commitment, but most of the folks working the event are dedicated to both making the summit successful, and coming back to do so in the future. Of course, the roster can change over time, or even suddenly, as people gain or lose travel budget, get new jobs, only attend Europe or North America or Asia, etc. It’s a constant dance, relying 100% on the people who want to make this project successful.

Last, but not least, is the Summit lead. They have to keep the entire process moving forward, be willing to step in to keep bike-shedding from derailing our deadlines, make sure the right people are talking to one another, lead all our meetings to make sure everyone gets a voice, etc. In some cases, the lead has to even be willing to take over an entirely separate role, in case someone gets sick or has any other extenuating circumstances, to make sure absolutely nothing falls through the cracks. The lead is only allowed to volunteer after they’ve been through this a few times and know what the event entails. Event planning is not for the faint of heart.

FM: The participation of volunteers is essential, but there’s also the topic of CNCF support: how does this dynamic play out in practice?

NA: This event would not happen in its current form without our CNCF liaison. They provide us with space, make sure we are fed and caffeinated and cared for, bring us outside spaces to evaluate, so we have somewhere to hold the social gathering, get us the budget so we have t-shirts and patches and the like, and generally make it possible for us to put this event together. They’re even responsible for the signage and arrows, so the attendees know where to go. They’re the ones sitting at the front desk, keeping an eye on everything and answering people’s questions. At the same time, they’re along to facilitate, and try to avoid influencing our planning.

There’s a ton of work that goes into making the summit happen that is easy to overlook, as an attendee, because people tend to expect things to just work. It is not exaggerating to say this event would not have happened like it has over the years, without the help from our liaisons, like Brienne and Deb. They are an integral part of the team.

A look ahead

FM: Currently, we’re preparing the NA 2023 summit, how is it going? Any changes in format compared with previous ones?

NA: I would say it’s going great, though I’m sort of emeritus lead for this event, mostly picking up the things that I see need to be done and don’t have someone assigned to it. We’re always learning from our past experiences and making small changes to continually be better, from how many people need to be on a particular rotation to how far in advance we open and close the CFP. There’s no major changes right now, just continually providing the content that the contributors want.

FM: For our readers that might be interested in joining in the Kubernetes Contributor Summit, is there anything they should know?

NA: First of all, the summit is an event by and for Org members. If you’re not already an org member, you should be getting involved before trying to attend the summit, as the content is curated specifically towards the contributors and maintainers of the project. That applies to the staff, as well, as all the decisions should be made with the interests and health of kubernetes contributors being the end goal. We get a lot of people who show interest in helping out, but then aren’t ready to make any sort of commitment, and that just makes more work for us. If you’re not already a proven and committed member of this community, it’s difficult for us to place you in a position that requires reliability. We have made some rare exceptions when we need someone local to help us out, but those are few and far between.

If you are, however, already a member, we’d love to have you. The more people that are involved, the better the event becomes. That applies to both dedicated staff, and those in attendance bringing CFPs, unconference topics, and just contributing to the discussions. If you’re part of this community and you’re going to be at KubeCon, I would highly urge you to attend, and if you’re not yet an org member, let’s make that happen!

FM: Indeed! Any final comments you would like to share?

NA: Just that the Contributor Summit is, for me, the ultimate manifestation of the Hallway Track. By being here, you’re part of the conversations that move this project forward. It’s good for you, and it’s good for Kubernetes. I hope to see you all in Chicago!

Blog: Spotlight on SIG Architecture: Production Readiness

Thu, 02 Nov 2023 00:00:00 +0000

This is the second interview of a SIG Architecture Spotlight series that will cover the different subprojects. In this blog, we will cover the SIG Architecture: Production Readiness subproject.

In this SIG Architecture spotlight, we talked with Wojciech Tyczynski (Google), lead of the Production Readiness subproject.

About SIG Architecture and the Production Readiness subproject

Frederico (FSM): Hello Wojciech, could you tell us a bit about yourself, your role and how you got involved in Kubernetes?

Wojciech Tyczynski (WT): I started contributing to Kubernetes in January 2015. At that time, Google (where I was and still am working) decided to start a Kubernetes team in the Warsaw office (in addition to already existing teams in California and Seattle). I was lucky enough to be one of the seeding engineers for that team.

After two months of onboarding and helping with different tasks across the project towards 1.0 launch, I took ownership of the scalability area and I was leading Kubernetes to support clusters with 5000 nodes. I’m still involved in SIG Scalability as its Technical Lead. That was the start of a journey since scalability is such a cross-cutting topic, and I started contributing to many other areas including, over time, to SIG Architecture.

FSM: In SIG Architecture, why specifically the Production Readiness subproject? Was it something you had in mind from the start, or was it an unexpected consequence of your initial involvement in scalability?

WT: After reaching that milestone of Kubernetes supporting 5000-node clusters, one of the goals was to ensure that Kubernetes would not degrade its scalability properties over time. While non-scalable implementation is always fixable, designing non-scalable APIs or contracts is problematic. I was looking for a way to ensure that people are thinking about scalability when they create new features and capabilities without introducing too much overhead.

This is when I joined forces with John Belamaric and David Eads and created a Production Readiness subproject within SIG Architecture. While setting the bar for scalability was only one of a few motivations for it, it ended up fitting quite well. At the same time, I was already involved in the overall reliability of the system internally, so other goals of Production Readiness were also close to my heart.

FSM: To anyone new to how SIG Architecture works, how would you describe the main goals and areas of intervention of the Production Readiness subproject?

WT: The goal of the Production Readiness subproject is to ensure that any feature that is added to Kubernetes can be reliably used in production clusters. This primarily means that those features are observable, scalable, supportable, can always be safely enabled and in case of production issues also disabled.

Production readiness and the Kubernetes project

FSM: Architectural consistency being one of the goals of the SIG, is this made more challenging by the distributed and open nature of Kubernetes? Do you feel this impacts the approach that Production Readiness has to take?

WT: The distributed nature of Kubernetes certainly impacts Production Readiness, because it makes thinking about aspects like enablement/disablement or scalability more challenging. To be more precise, when enabling or disabling features that span multiple components you need to think about version skew between them and design for it. For scalability, changes in one component may actually result in problems for a completely different one, so it requires a good understanding of the whole system, not just individual components. But it’s also what makes this project so interesting.

FSM: Those running Kubernetes in production will have their own perspective on things, how do you capture this feedback?

WT: Fortunately, we aren’t talking about “them” here, we’re talking about “us”: all of us are working for companies that are managing large fleets of Kubernetes clusters and we’re involved in that too, so we suffer from those problems ourselves.

So while we’re trying to get feedback (our annual PRR survey is very important for us), it rarely reveals completely new problems - it rather shows the scale of them. And we try to react to it - changes like “Beta APIs off by default” happen in reaction to the data that we observe.

FSM: On the topic of reaction, that made me think of how the Kubernetes Enhancement Proposal (KEP) template has a Production Readiness Review (PRR) section, which is tied to the graduation process. Was this something born out of identified insufficiencies? How would you describe the results?

WT: As mentioned above, the overall goal of the Production Readiness subproject is to ensure that every newly added feature can be reliably used in production. It’s not possible to enforce that by a central team - we need to make it everyone’s problem.

To achieve it, we wanted to ensure that everyone designing their new feature is thinking about safe enablement, scalability, observability, supportability, etc. from the very beginning. Which means not when the implementation starts, but rather during the design. Given that KEPs are effectively Kubernetes design docs, making it part of the KEP template was the way to achieve the goal.

FSM: So, in a way making sure that feature owners have thought about the implications of their proposal.

WT: Exactly. We already observed that just by forcing feature owners to think through the PRR aspects (via forcing them to fill in the PRR questionnaire) many of the original issues are going away. Sure - as PRR approvers we’re still catching gaps, but even the initial versions of KEPs are better now than they used to be a couple of years ago in what concerns thinking about productionisation aspects, which is exactly what we wanted to achieve - spreading the culture of thinking about reliability in its widest possible meaning.

FSM: We’ve been talking about the PRR process, could you describe it for our readers?

WT: The PRR process is fairly simple - we just want to ensure that you think through the productionisation aspects of your feature early enough. If you do your job, it’s just a matter of answering some questions in the KEP template and getting approval from a PRR approver (in addition to regular SIG approval). If you didn’t think about those aspects earlier, it may require spending more time and potentially revising some decisions, but that’s exactly what we need to make the Kubernetes project reliable.

Helping with Production Readiness

FSM: Production Readiness seems to be one area where a good deal of prior exposure is required in order to be an effective contributor. Are there also ways for someone newer to the project to contribute?

WT: PRR approvers have to have a deep understanding of the whole Kubernetes project to catch potential issues. Kubernetes is such a large project now with so many nuances that people who are new to the project can simply miss the context, no matter how senior they are.

That said, there are many ways that you may implicitly help. Increasing the reliability of particular areas of the project by improving its observability and debuggability, increasing test coverage, and building new kinds of tests (upgrade, downgrade, chaos, etc.) will help us a lot. Note that the PRR subproject is focused on keeping the bar at the design level, but we should also care equally about the implementation. For that, we’re relying on individual SIGs and code approvers, so having people there who are aware of productionisation aspects, and who deeply care about it, will help the project a lot.

FSM: Thank you! Any final comments you would like to share with our readers?

WT: I would like to highlight and thank all contributors for their cooperation. While the PRR adds some additional work for them, we see that people care about it, and what’s even more encouraging is that with every release the quality of the answers improves, and questions “do I really need a metric reflecting if my feature works” or “is downgrade really that important” don’t really appear anymore.

Blog: A Quick Recap of 2023 China Kubernetes Contributor Summit

Fri, 20 Oct 2023 00:00:00 +0000

On September 26, 2023, the first day of KubeCon + CloudNativeCon + Open Source Summit China 2023, nearly 50 contributors gathered in Shanghai for the Kubernetes Contributor Summit.

All participants in the 2023 Kubernetes Contributor Summit

This marked the first in-person offline gathering held in China after three years of the pandemic.

A joyful meetup

The event began with welcome speeches from Kevin Wang from Huawei Cloud, one of the co-chairs of KubeCon, and Puja from Giant Swarm.

Following the opening remarks, the contributors introduced themselves briefly. Most attendees were from China, while some contributors had made the journey from Europe and the United States specifically for the conference. Technical experts from companies such as Microsoft, Intel, Huawei, as well as emerging forces like DaoCloud, were present. Laughter and cheerful voices filled the room, regardless of whether English was spoken with European or American accents or if conversations were carried out in authentic Chinese language. This created an atmosphere of comfort, joy, respect, and anticipation. Past contributions brought everyone closer, and mutual recognition and accomplishments made this offline gathering possible.

Face to face meeting in Shanghai

The attending contributors were no longer just GitHub IDs; they transformed into vivid faces. From sitting together and capturing group photos to attempting to identify “Who is who,” a loosely connected collective emerged. This team structure, although loosely knit and free-spirited, was established to pursue shared dreams.

As the saying goes, “You reap what you sow.” Each effort has been diligently documented within the Kubernetes community contributions. Regardless of the passage of time, the community will not erase those shining traces. Brilliance can be found in your PRs, issues, or comments. It can also be seen in the smiling faces captured in meetup photos or heard through stories passed down among contributors.

Next, there were three technical sharing sessions:

sig-multi-cluster: Hongcai Ren, a maintainer of Karmada, provided an introduction to the responsibilities and roles of this SIG. Their focus is on designing, discussing, implementing, and maintaining APIs, tools, and documentation related to multi-cluster management. Cluster Federation, one of Karmada’s core concepts, is also part of their work.
helmfile: yxxhero from GitLab presented how to deploy Kubernetes manifests declaratively, customize configurations, and leverage the latest features of Helm, including Helmfile.
sig-scheduling: william-wang from Huawei Cloud shared the recent updates and future plans of SIG Scheduling. This SIG is responsible for designing, developing, and testing components related to Pod scheduling.

A technical session about SIG Multicluster

Following the sessions, a video featuring a call for contributors by Sergey Kanzhelev, the SIG-Node Chair, was played. The purpose was to encourage more contributors to join the Kubernetes community, with a special emphasis on the popular SIG-Node.

Lastly, Kevin hosted an Unconference collective discussion session covering topics such as multi-cluster management, scheduling, elasticity, AI, and more. For detailed minutes of the Unconference meeting, please refer to https://docs.qq.com/doc/DY3pLWklzQkhjWHNT.

China’s contributor statistics

The contributor summit took place in Shanghai, with 90% of the attendees being Chinese. Within the Cloud Native Computing Foundation (CNCF) ecosystem, contributions from China have been steadily increasing. Currently:

Chinese contributors account for 9% of the total.
Contributions from China make up 11.7% of the overall volume.
China ranks second globally in terms of contributions.

The data is from KubeCon keynotes by Chris Aniszczyk, CTO of Cloud Native Computing Foundation, on September 26, 2023. This probably understates Chinese contributions. A lot of Chinese contributors use VPNs and may not show up as being from China in the stats accurately.

The Kubernetes Contributor Summit is an inclusive meetup that welcomes all community contributors, including:

New Contributors
Current Contributors
- docs
- code
- community management
Subproject members
Members of Special Interest Group (SIG) / Working Group (WG)
Active Contributors
Casual Contributors

Acknowledgments

We would like to express our gratitude to the organizers of this event:

Kevin Wang, the co-chair of KubeCon and the lead of the kubernetes contributor summit.
Paco Xu, who actively coordinated the venue, meals, invited contributors from both China and international sources, and established WeChat groups to collect agenda topics. They also shared details of the event before and after its occurrence through pre and post announcements.
Mengjiao Liu, who was responsible for organizing, coordinating, and facilitating various matters related to the summit.

We extend our appreciation to all the contributors who attended the China Kubernetes Contributor Summit in Shanghai. Your dedication and commitment to the Kubernetes community are invaluable. Together, we continue to push the boundaries of cloud native technology and shape the future of this ecosystem.

Blog: Spotlight on SIG Architecture: Conformance

Thu, 05 Oct 2023 00:00:00 +0000

This is the first interview of a SIG Architecture Spotlight series that will cover the different subprojects. We start with the SIG Architecture: Conformance subproject

In this SIG Architecture spotlight, we talked with Riaan Kleinhans (ii-Team), Lead for the Conformance sub-project.

About SIG Architecture and the Conformance subproject

Frederico (FSM): Hello Riaan, and welcome! For starters, tell us a bit about yourself, your role and how you got involved in Kubernetes.

Riaan Kleinhans (RK): Hi! My name is Riaan Kleinhans and I live in South Africa. I am the Project manager for the ii-Team in New Zealand. When I joined ii the plan was to move to New Zealand in April 2020 and then Covid happened. Fortunately, being a flexible and dynamic team we were able to make it work remotely and in very different time zones.

The ii.nz team have been tasked with managing the Kubernetes Conformance testing technical debt and writing tests to clear the technical debt. I stepped into the role of project manager to be the link between monitoring, test writing and the community. Through that work I had the privilege of meeting the late Dan Kohn in those first months, his enthusiasm about the work we were doing was a great inspiration.

FSM: Thank you - so, your involvement in SIG Architecture started because of the conformance work?

RK: SIG Architecture is the home for the Kubernetes Conformance subproject. Initially, most of my interactions were directly with SIG Architecture through the Conformance sub-project. However, as we began organizing the work by SIG, we started engaging directly with each individual SIG. These engagements with the SIGs that own the untested APIs have helped us accelerate our work.

FSM: How would you describe the main goals and areas of intervention of the Conformance sub-project?

RM: The Kubernetes Conformance sub-project focuses on guaranteeing compatibility and adherence to the Kubernetes specification by developing and maintaining a comprehensive conformance test suite. Its main goals include assuring compatibility across different Kubernetes implementations, verifying adherence to the API specification, supporting the ecosystem by encouraging conformance certification, and fostering collaboration within the Kubernetes community. By providing standardised tests and promoting consistent behaviour and functionality, the Conformance subproject ensures a reliable and compatible Kubernetes ecosystem for developers and users alike.

More on the Conformance Test Suite

FSM: A part of providing those standardised tests is, I believe, the Conformance Test Suite. Could you explain what it is and its importance?

RK: The Kubernetes Conformance Test Suite checks if Kubernetes distributions meet the project’s specifications, ensuring compatibility across different implementations. It covers various features like APIs, networking, storage, scheduling, and security. Passing the tests confirms proper implementation and promotes a consistent and portable container orchestration platform.

FSM: Right, the tests are important in the way they define the minimum features that any Kubernetes cluster must support. Could you describe the process around determining which features are considered for inclusion? Is there any tension between a more minimal approach, and proposals from the other SIGs?

RK: The requirements for each endpoint that undergoes conformance testing are clearly defined by SIG Architecture. Only API endpoints that are generally available and non-optional features are eligible for conformance. Over the years, there have been several discussions regarding conformance profiles, exploring the possibility of including optional endpoints like RBAC, which are widely used by most end users, in specific profiles. However, this aspect is still a work in progress.

Endpoints that do not meet the conformance criteria are listed in ineligible_endpoints.yaml, which is publicly accessible in the Kubernetes repo. This file can be updated to add or remove endpoints as their status or requirements change. These ineligible endpoints are also visible on APISnoop.

Ensuring transparency and incorporating community input regarding the eligibility or ineligibility of endpoints is of utmost importance to SIG Architecture.

FSM: Writing tests for new features is something generally requires some kind of enforcement. How do you see the evolution of this in Kubernetes? Was there a specific effort to improve the process in a way that required tests would be a first-class citizen, or was that never an issue?

RK: When discussions surrounding the Kubernetes conformance programme began in 2018, only approximately 11% of endpoints were covered by tests. At that time, the CNCF’s governing board requested that if funding were to be provided for the work to cover missing conformance tests, the Kubernetes Community should adopt a policy of not allowing new features to be added unless they include conformance tests for their stable APIs.

SIG Architecture is responsible for stewarding this requirement, and APISnoop has proven to be an invaluable tool in this regard. Through automation, APISnoop generates a pull request every weekend to highlight any discrepancies in Conformance coverage. If any endpoints are promoted to General Availability without a conformance test, it will be promptly identified. This approach helps prevent the accumulation of new technical debt.

Additionally, there are plans in the near future to create a release informing job, which will add an additional layer to prevent any new technical debt.

FSM: I see, tooling and automation play an important role there. What are, in your opinion, the areas that, conformance-wise, still require some work to be done? In other words, what are the current priority areas marked for improvement?

RK: We have reached the “100% Conformance Tested” milestone in release 1.27!

At that point, the community took another look at all the endpoints that were listed as ineligible for conformance. The list was populated through community input over several years. Several endpoints that were previously deemed ineligible for conformance have been identified and relocated to a new dedicated list, which is currently receiving focused attention for conformance test development. Again, that list can also be checked on apisnoop.cncf.io.

To ensure the avoidance of new technical debt in the conformance project, there are upcoming plans to establish a release informing job as an additional preventive measure.

While APISnoop is currently hosted on CNCF infrastructure, the project has been generously donated to the Kubernetes community. Consequently, it will be transferred to community-owned infrastructure before the end of 2023.

FSM: That’s great news! For anyone wanting to help, what are the venues for collaboration that you would highlight? Do all of them require solid knowledge of Kubernetes as a whole, or are there ways someone newer to the project can contribute?

RK: Contributing to conformance testing is akin to the task of “washing the dishes” – it may not be highly visible, but it remains incredibly important. It necessitates a strong understanding of Kubernetes, particularly in the areas where the endpoints need to be tested. This is why working with each SIG that owns the API endpoint being tested is so important.

As part of our commitment to making test writing accessible to everyone, the ii team is currently engaged in the development of a “click and deploy” solution. This solution aims to enable anyone to swiftly create a working environment on real hardware within minutes. We will share updates regarding this development as soon as we are ready.

FSM: That’s very helpful, thank you. Any final comments you would like to share with our readers?

RK: Conformance testing is a collaborative community endeavour that involves extensive cooperation among SIGs. SIG Architecture has spearheaded the initiative and provided guidance. However, the progress of the work relies heavily on the support of all SIGs in reviewing, enhancing, and endorsing the tests.

I would like to extend my sincere appreciation to the ii team for their unwavering commitment to resolving technical debt over the years. In particular, Hippie Hacker’s guidance and stewardship of the vision has been invaluable. Additionally, I want to give special recognition to Stephen Heywood for shouldering the majority of the test writing workload in recent releases, as well as to Zach Mandeville for his contributions to APISnoop.

FSM: Many thanks for your availability and insightful comments, I’ve personally learned quite a bit with it and I’m sure our readers will as well.

Blog: Announcing the 2023 Steering Committee Election Results

Mon, 02 Oct 2023 00:00:00 +0000

The 2023 Steering Committee Election is now complete. The Kubernetes Steering Committee consists of 7 seats, 4 of which were up for election in 2023. Incoming committee members serve a term of 2 years, and all members are elected by the Kubernetes Community.

Thank you to everyone who voted in the election; your participation helps support the community’s continued health and success.

Results

Congratulations to the elected committee members whose two year terms begin immediately (listed in alphabetical order by GitHub handle):

Stephen Augustus (@justaugustus), Cisco
Paco Xu 徐俊杰 (@pacoxu), DaoCloud
Patrick Ohly (@pohly), Intel
Maciej Szulik (@soltysh), Red Hat

They join continuing members:

Benjamin Elder (@bentheelder), Google
Bob Killen (@mrbobbytables), Google
Nabarun Pal (@palnabarun, VMware

Stephen Augustus is a returning Steering Committee Member.

Big Thanks!

Thank you and congratulations on a successful election to this round’s election officers:

Bridget Kromhout (@bridgetkromhout)
Davanum Srinavas (@dims)
Kaslin Fields (@kaslin)

Thanks to the Emeritus Steering Committee Members. Your service is appreciated by the community:

Christoph Blecker (@cblecker)
Carlos Tadeu Panato Jr. (@cpanato)
Tim Pepper (@tpepper)

And thank you to all the candidates who came forward to run for election.

Get Involved with the Steering Committee

This governing body, like all of Kubernetes, is open to all. You can follow along with Steering Committee backlog items and weigh in by filing an issue or creating a PR against their repo. They have an open meeting on the first Monday at 9:30am PT of every month. They can also be contacted at their public mailing list steering@kubernetes.io.

You can see what the Steering Committee meetings are all about by watching past meetings on the YouTube Playlist.

If you want to meet some of the newly elected Steering Committee members, join us for the Steering AMA at the Kubernetes Contributor Summit in Chicago.

This post was written by the Contributor Comms Subproject. If you want to write stories about the Kubernetes community, learn more about us.

Blog: Spotlight on SIG ContribEx

Mon, 14 Aug 2023 00:00:00 +0000

Author: Fyka Ansari

Welcome to the world of Kubernetes and its vibrant contributor community! In this blog post, we’ll be shining a spotlight on the Special Interest Group for Contributor Experience (SIG ContribEx), an essential component of the Kubernetes project.

SIG ContribEx in Kubernetes is responsible for developing and maintaining a healthy and productive community of contributors to the project. This involves identifying and addressing bottlenecks that may hinder the project’s growth and feature velocity, such as pull request latency and the number of open pull requests and issues.

SIG ContribEx works to improve the overall contributor experience by creating and maintaining guidelines, tools, and processes that facilitate collaboration and communication among contributors. They also focus on community building and support, including outreach programs and mentorship initiatives to onboard and retain new contributors.

Ultimately, the role of SIG ContribEx is to foster a welcoming and inclusive environment that encourages contribution and supports the long-term sustainability of the Kubernetes project.

In this blog post, Fyka Ansari interviews Kaslin Fields, a DevRel Engineer at Google, who is a chair of SIG ContribEx, and Madhav Jivrajani, a Software Engineer at VMWare who serves as a SIG ContribEx Tech Lead. This interview covers various aspects of SIG ContribEx, including current initiatives, exciting developments, and how interested individuals can get involved and contribute to the group. It provides valuable insights into the workings of SIG ContribEx and highlights the importance of its role in the Kubernetes ecosystem.

Introductions

Fyka: Let’s start by diving into your background and how you got involved in the Kubernetes ecosystem. Can you tell us more about that journey?

Kaslin: I first got involved in the Kubernetes ecosystem through my mentor, Jonathan Rippy, who introduced me to containers during my early days in tech. Eventually, I transitioned to a team working with containers, which sparked my interest in Kubernetes when it was announced. While researching Kubernetes in that role, I eagerly sought opportunities to engage with the containers/Kubernetes community. It was not until my subsequent job that I found a suitable role to contribute consistently. I joined SIG ContribEx, specifically in the Contributor Comms subproject, to both deepen my knowledge of Kubernetes and support the community better.

Madhav: My journey with Kubernetes began when I was a student, searching for interesting and exciting projects to work on. With my peers, I discovered open source and attended The New Contributor Workshop organized by the Kubernetes community. The workshop not only provided valuable insights into the community structure but also gave me a sense of warmth and welcome, which motivated me to join and remain involved. I realized that collaboration is at the heart of open-source communities, and to get answers and support, I needed to contribute and do my part. I started working on issues in ContribEx, particularly focusing on GitHub automation, despite not fully understanding the task at first. I continued to contribute for various technical and non-technical aspects of the project, finding it to be one of the most professionally rewarding experiences in my life.

Fyka: That’s such an inspiration in itself! I’m sure beginners who are reading this got the ultimate motivation to take their first steps. Embracing the Learning journey, seeking mentorship, and engaging with the Kubernetes community can pave the way for exciting opportunities in the tech industry. Your stories proved the importance of starting small and being proactive, just like Madhav said Don’t be afraid to take on tasks, even if you’re uncertain at first.

Primary goals and scope

Fyka: Given your experience as a member of SIG ContribEx, could you tell us a bit about the group’s primary goals and initiatives? Its current focus areas? What do you see as the scope of SIG ContribEx and the impact it has on the Kubernetes community?

Kaslin: SIG ContribEx’s primary goals are to simplify the contributions of Kubernetes contributors and foster a welcoming community. It collaborates with other Kubernetes SIGs, such as planning the Contributor Summit at KubeCon, ensuring it meets the needs of various groups. The group’s impact is evident in projects like updating org membership policies and managing critical platforms like Zoom, YouTube, and Slack. Its scope encompasses making the contributor experience smoother and supporting the overall Kubernetes community.

Madhav: The Kubernetes project has vertical SIGs and cross-cutting SIGs, ContribEx is a deeply cross-cutting SIG, impacting virtually every area of the Kubernetes community. Adding to Kaslin, sustainability in the Kubernetes project and community is critical now more than ever, it plays a central role in addressing critical issues, such as maintainer succession, by facilitating cohorts for SIGs to train experienced community members to take on leadership roles. Excellent examples include SIG CLI and SIG Apps, leading to the onboarding of new reviewers. Additionally, SIG ContribEx is essential in managing GitHub automation tools, including bots and commands used by contributors for interacting with Prow and other automation (label syncing, group and GitHub team management, etc).

Beginner’s guide!

Fyka: I’ll never forget talking to Kaslin when I joined the community and needed help with contributing. Kaslin, your quick and clear answers were a huge help in getting me started. Can you both give some tips for people new to contributing to Kubernetes? What makes SIG ContribEx a great starting point? Why should beginners and current contributors consider it? And what cool opportunities are there for newbies to jump in?

Kaslin: If you want to contribute to Kubernetes for the first time, it can be overwhelming to know where to start. A good option is to join SIG ContribEx as it offers great opportunities to know and serve the community. Within SIG ContribEx, various subprojects allow you to explore different parts of the Kubernetes project while you learn how contributions work. Once you know a bit more, it’s common for you to move to other SIGs within the project, and we think that’s wonderful. While many newcomers look for “good first issues” to start with, these opportunities can be scarce and get claimed quickly. Instead, the real benefit lies in attending meetings and getting to know the community. As you learn more about the project and the people involved, you’ll be better equipped to offer your help, and the community will be more inclined to seek your assistance when needed. As a co-lead for the Contributor Comms subproject, I can confidently say that it’s an excellent place for beginners to get involved. We have supportive leads and particularly beginner-friendly projects too.

Madhav: To begin, read the SIG README on GitHub, which provides an overview of the projects the SIG manages. While attending meetings is beneficial for all SIGs, it’s especially recommended for SIG ContribEx, as each subproject gets dedicated slots for updates and areas that need help. If you can’t attend in real-time due to time zone differences, you can catch the meeting recordings or Notes later.

Skills you learn!

Fyka: What skills do you look for when bringing in new contributors to SIG ContribEx, from passion to expertise? Additionally, what skills can contributors expect to develop while working with SIG ContribEx?

Kaslin: Skills folks need to have or will acquire vary depending on what area of ContribEx they work upon. Even within a subproject, a range of skills can be useful and/or developed. For example, the tech lead role involves technical tasks and overseeing automation, while the social media lead role requires excellent communication skills. Working with SIG ContribEx allows contributors to acquire various skills based on their chosen subproject. By participating in meetings, listening, learning, and taking on tasks related to their interests, they can develop and hone these skills. Some subprojects may require more specialized skills, like program management for the mentoring project, but all contributors can benefit from offering their talents to help teach others and contribute to the community.

Sub-projects under SIG ContribEx

Fyka: SIG ContribEx has several smaller projects. Can you tell me about the aims of these projects and how they’ve impacted the Kubernetes community?

Kaslin: Some SIGs have one or two subprojects and some have none at all, but in SIG ContribEx, we have eleven!

Here’s a list of them and their respective mission statements

Community: Manages the community repository, documentation, and operations.
Community management: Handles communication platforms and policies for the community.
Contributor-comms: Focuses on promoting the success of Kubernetes contributors through marketing.
Contributors-documentation: Writes and maintains documentation for contributing to Kubernetes.
Devstats: Maintains and updates the Kubernetes statistics website.
Elections: Oversees community elections and maintains related documentation and software.
Events: Organizes contributor-focused events like the Contributor Summit.
Github management: Manages permissions, repositories, and groups on GitHub.
Mentoring: Develop programs to help contributors progress in their contributions.
Sigs-GitHub-actions: Repository for GitHub actions related to all SIGs in Kubernetes.
Slack-infra: Creates and maintains tools and automation for Kubernetes Slack.

Madhav: Also, Devstats is critical from a sustainability standpoint!

(If you are willing to learn more and get involved with any of these sub-projects, check out the SIG ContribEx README).

Accomplishments

Fyka: With that said, any SIG-related accomplishment that you’re proud of?

Kaslin: I’m proud of the accomplishments made by SIG ContribEx and its contributors in supporting the community. Some of the recent achievements include:

Establishment of the elections subproject: Kubernetes is a massive project, and ensuring smooth leadership transitions is crucial. The contributors in this subproject organize fair and consistent elections, which helps keep the project running effectively.
New issue triage proces: With such a large open-source project like Kubernetes, there’s always a lot of work to be done. To ensure things progress safely, we implemented new labels and updated functionality for issue triage using our PROW tool. This reduces bottlenecks in the workflow and allows leaders to accomplish more.
New org membership requirements: Becoming an org member in Kubernetes can be overwhelming for newcomers. We view org membership as a significant milestone for contributors aiming to take on leadership roles. We recently updated the rules to automatically remove privileges from inactive members, making sure that the right people have access to the necessary tools and responsibilities.

Overall, these accomplishments have greatly benefited our fellow contributors and strengthened the Kubernetes community.

Upcoming initiatives

Fyka: Could you give us a sneak peek into what’s next for the group? We’re excited to hear about upcoming projects and initiatives from this dynamic team.

Madhav: We’d love for more groups to sign up for mentoring cohorts! We’re probably going to have to spend some time polishing the process around that.

Final thoughts

Fyka: As we wrap up our conversation, would you like to share some final thoughts for those interested in contributing to SIG ContribEx or getting involved with Kubernetes?

Madhav: Kubernetes is meant to be overwhelming and difficult initially! You’re coming into something that’s taken multiple people, from multiple countries, multiple years to build. Embrace that diversity! Use the high entropy initially to collide around and gain as much knowledge about the project and community as possible before you decide to settle in your niche.

Fyka: Thank You Madhav and Kaslin, it was an absolute pleasure chatting about SIG ContribEx and your experiences as a member. It’s clear that the role of SIG ContribEx in Kubernetes is significant and essential, ensuring scalability, growth and productivity, and I hope this interview inspires more people to get involved and contribute to Kubernetes. I wish SIG ContribEx all the best, and can’t wait to see what exciting things lie ahead!

What next?

We love meeting new contributors and helping them in investigating different Kubernetes project spaces. If you are interested in getting more involved with SIG ContribEx, here are some resources for you to get started:

GitHub
Mailing list
Open Community Issues/PRs
Slack
Slack channel #sig-contribex
SIG Contribex also hosted a KubeCon talk about studying Kubernetes Contributor experiences.

Blog: Spotlight on SIG CLI

Thu, 20 Jul 2023 00:00:00 +0000

In the world of Kubernetes, managing containerized applications at scale requires powerful and efficient tools. The command-line interface (CLI) is an integral part of any developer or operator’s toolkit, offering a convenient and flexible way to interact with a Kubernetes cluster.

SIG CLI plays a crucial role in improving the Kubernetes CLI experience by focusing on the development and enhancement of kubectl, the primary command-line tool for Kubernetes.

In this SIG CLI Spotlight, Arpit Agrawal, SIG ContribEx-Comms team member, talked with Katrina Verey, Tech Lead & Chair of SIG CLI,and Maciej Szulik, SIG CLI Batch Lead, about SIG CLI, current projects, challenges and how anyone can get involved.

So, whether you are a seasoned Kubernetes enthusiast or just getting started, understanding the significance of SIG CLI will undoubtedly enhance your Kubernetes journey.

Introductions

Arpit: Could you tell us a bit about yourself, your role, and how you got involved in SIG CLI?

Maciej: I’m one of the technical leads for SIG-CLI. I was working on Kubernetes in multiple areas since 2014, and in 2018 I got appointed a lead.

Katrina: I’ve been working with Kubernetes as an end-user since 2016, but it was only in late 2019 that I discovered how well SIG CLI aligned with my experience from internal projects. I started regularly attending meetings and made a few small PRs, and by 2021 I was working more deeply with the Kustomize team specifically. Later that year, I was appointed to my current roles as subproject owner for Kustomize and KRM Functions, and as SIG CLI Tech Lead and Chair.

About SIG CLI

Arpit: Thank you! Could you share with us the purpose and goals of SIG CLI?

Maciej: Our charter has the most detailed description, but in few words, we handle all CLI tooling that helps you manage your Kubernetes manifests and interact with your Kubernetes clusters.

Arpit: I see. And how does SIG CLI work to promote best-practices for CLI development and usage in the cloud native ecosystem?

Maciej: Within kubectl, we have several on-going efforts that try to encourage new contributors to align existing commands to new standards. We publish several libraries which hopefully make it easier to write CLIs that interact with Kubernetes APIs, such as cli-runtime and kyaml.

Katrina: We also maintain some interoperability specifications for CLI tooling, such as the KRM Functions Specification (GA) and the new ApplySet Specification (alpha).

Current projects and challenges

Arpit: Going through the README file, it’s clear SIG CLI has a number of subprojects, could you highlight some important ones?

Maciej: The four most active subprojects that are, in my opinion, worthy of your time investment would be:

kubectl: the canonical Kubernetes CLI.
Kustomize: a template-free customization tool for Kubernetes yaml manifest files.
KUI - a GUI interface to Kubernetes, think kubectl on steroids.
krew: a plugin manager for kubectl.

Arpit: Are there any upcoming initiatives or developments that SIG CLI is working on?

Maciej: There are always several initiatives we’re working on at any given point in time. It’s best to join one of our calls to learn about the current ones.

Katrina: For major features, you can check out our open KEPs. For instance, in 1.27 we introduced alphas for a new pruning mode in kubectl apply, and for kubectl create plugins. Exciting ideas that are currently under discussion include an interactive mode for kubectl delete (KEP 3895) and the kuberc user preferences file (KEP 3104).

Arpit: Could you discuss any challenges that SIG CLI faces in its efforts to improve CLIs for cloud-native technologies? What are the future efforts to solve them?

Katrina: The biggest challenge we’re facing with every decision is backwards compatibility and ensuring we don’t break existing users. It frequently happens that fixing what’s on the surface may seem straightforward, but even fixing a bug could constitute a breaking change for some users, which means we need to go through an extended deprecation process to change it, or in some cases we can’t change it at all. Another challenge is the need to balance customization with usability in the flag sets we expose on our tools. For example, we get many proposals for new flags that would certainly be useful to some users, but not a large enough subset to justify the increased complexity having them in the tool entails for everyone. The kuberc proposal may help with some of these problems by giving individual users the ability to set or override default values we can’t change, and even create custom subcommands via aliases

Arpit: With every new version release of Kubernetes, maintaining consistency and integrity is surely challenging: how does the SIG CLI team tackle it?

Maciej: This is mostly similar to the topic mentioned in the previous question: every new change, especially to existing commands goes through a lot of scrutiny to ensure we don’t break existing users. At any point in time we have to keep a reasonable balance between features and not breaking users.

Future plans and contribution

Arpit: How do you see the role of CLI tools in the cloud-native ecosystem evolving in the future?

Maciej: I think that CLI tools were and will always be an important piece of the ecosystem. Whether used by administrators on remote machines that don’t have GUI or in every CI/CD pipeline, they are irreplaceable.

Arpit: Kubernetes is a community-driven project. Any recommendation for anyone looking into getting involved in SIG CLI work? Where should they start? Are there any prerequisites?

Maciej: There are no prerequisites other than a little bit of free time on your hands and willingness to learn something new :-)

Katrina: A working knowledge of Go often helps, but we also have areas in need of non-code contributions, such as the Kustomize docs consolidation project.

Blog: Spotlight on SIG Network

Tue, 09 May 2023 00:00:00 +0000

Networking is one of the core pillars of Kubernetes, and the Special Interest Group for Networking (SIG Network) is responsible for developing and maintaining the networking features of Kubernetes. It covers all aspects to ensure Kubernetes provides a reliable and scalable network infrastructure for containerized applications.

In this SIG Network spotlight, Sujay Dey talked with Shane Utt, Software Engineer at Kong, chair of SIG Network and maintainer of Gateway API, on different aspects of the SIG, what are the exciting things going on and how anyone can get involved and contribute here.

Sujay: Hello, and first of all, thanks for the opportunity of learning more about SIG Network. I would love to hear your story, so could you please tell us a bit about yourself, your role, and how you got involved in Kubernetes, especially in SIG Network?

Shane: Hello! Thank you for reaching out.

My Kubernetes journey started while I was working for a small data centre: we were early adopters of Kubernetes and focused on using Kubernetes to provide SaaS products. That experience led to my next position developing a distribution of Kubernetes with a focus on networking. During this period in my career, I was active in SIG Network (predominantly as a consumer).

When I joined Kong my role in the community changed significantly, as Kong actively encourages upstream participation. I greatly increased my engagement and contributions to the Gateway API project during those years, and eventually became a maintainer.

I care deeply about this community and the future of our technology, so when a chair position for the SIG became available, I volunteered my time immediately. I’ve enjoyed working on Kubernetes over the better part of a decade and I want to continue to do my part to ensure our community and technology continues to flourish.

Sujay: I have to say, that was a truely inspiring journey! Now, let us talk a bit more about SIG Network. Since we know it covers a lot of ground, could you please highlight its scope and current focus areas?

Shane: For those who may be uninitiated: SIG Network is responsible for the components, interfaces, and APIs which expose networking capabilities to Kubernetes users and workloads. The charter is a pretty good indication of our scope, but I can add some additional highlights on some of our current areas of focus (this is a non-exhaustive list of sub-projects):

kube-proxy & KPNG

Those familiar with Kubernetes will know the Service API, which enables exposing a group of pods over a network. The current standard implementation of Service is known as kube-proxy, but what may be unfamiliar to people is that there are a growing number of disparate alternative implementations on the rise in recent years. To try and give provisions to these implementations (and also provide some areas of alignment so that implementations do not become too disparate from each other) upstream Kubernetes efforts are underway to create a more modular public interface for kube-proxy. The intention is for implementations to join in around a common set of libraries and speak a common language. This area of focus is known as the KPNG project, and if this sounds interesting to you, please join us in the KPNG community meetings and #sig-network-kpng on Kubernetes Slack.

Multi-Network

Today one of the primary requirements for Kubernetes networking is to achieve connectivity between pods in a cluster, satisfying a large number of Kubernetes end-users. However, some use cases require isolated networks and special interfaces for performance-oriented needs (e.g. AF_XDP, memif, SR-IOV). There’s a growing need for special networking configurations in Kubernetes in general. The Multi-Network project exists to improve the management of multiple different networks for pods: anyone interested in some of the lower-level details of pod networking (or anyone having relevant use cases) can join us in the Multi-Network community meetings and #sig-network-multi-network on Kubernetes Slack.

Network Policy

The NetworkPolicy API sub-group was formed to address network security beyond the well-known version 1 of the NetworkPolicy resource. We’ve also been working on the AdminNetworkPolicy resource (previously known as ClusterNetworkPolicy) to provide cluster administrator-focused functionality. The network policy sub-project is a great place to join in if you’re particularly interested in security and CNI, please feel free to join our community meetings and the #sig-network-policy-api channel on Kubernetes Slack.

Gateway API

If you’re specially interested in ingress or mesh networking the Gateway API may be a sub-project you would enjoy. In Gateway API , we’re actively developing the successor to the illustrious Ingress API, which includes a Gateway resource which defines the addresses and listeners of the gateway and various routing types (e.g. HTTPRoute, GRPCRoute, TLSRoute, TCPRoute, UDPRoute, etc.) that attach to Gateways. We also have an initiative within this project called GAMMA, geared towards using Gateway API resources in a mesh network context. There are some up-and-coming side projects within Gateway API as well, including ingress2gateway which is a tool for compiling existing Ingress objects to equivalent Gateway API resources, and Blixt, a Layer4 implementation of Gateway API using Rust/eBPF for the data plane, intended as a testing and reference implementation. If this sounds interesting, we would love to have readers join us in our Gateway API community meetings and #sig-network-gateway-api on Kubernetes Slack.

Sujay: Couldn’t agree more! That was a very informative description, thanks for highlighting them so nicely. As you have already mentioned about the SIG channels to get involved, would you like to add anything about where people like beginners can jump in and contribute?

Shane: For help getting started Kubernetes Slack is a great place to talk to community members and includes several #sig-network-<project> channels as well as our main #sig-network channel. Also, check for issues labelled good-first-issue if you prefer to just dive right into the repositories. Let us know how we can help you!

Sujay: What skills are contributors to SIG Network likely to learn?

Shane: To me, it feels limitless. Practically speaking, it’s very much up to the individual what they want to learn. However, if you just intend to learn as much as you possibly can about networking, SIG Network is a great place to join in and grow your knowledge.

If you’ve ever wondered how Kubernetes Service API works or wanted to implement an ingress controller, this is a great place to join in. If you wanted to dig down deep into the inner workings of CNI, or how the network interfaces at the pod level are configured, you can do that here as well.

We have an awesome and diverse community of people from just about every kind of background you can imagine. This is a great place to share ideas and raise proposals, improving your skills in design, as well as alignment and consensus building.

There’s a wealth of opportunities here in SIG Network. There are lots of places to jump in, and the learning opportunities are boundless.

Sujay: Thanks a lot! It was a really great discussion, we got to know so many great things about SIG Network. I’m sure that many others will find this just as useful as I did.

Blog: E2E Testing Best Practices, Reloaded

Wed, 12 Apr 2023 00:00:00 +0000

End-to-end (E2E) testing in Kubernetes is how the project validates functionality with real clusters. Contributors sooner or later encounter it when asked to write E2E tests for new features or to help with debugging test failures. Cluster admins or vendors might run the conformance tests, a subset of all tests in the E2E test suite.

The underlying E2E framework for writing these E2E tests has been around for a long time. Functionality was added to it as needed, leading to code that became hard to maintain and use. The testing commons WG started cleaning it up, but dissolved before completely achieving their goals.

After the migration to Gingko v2 in Kubernetes 1.25, I picked up several of the loose ends and started untangling them. This blog post is a summary of those changes. Some of this content is also found in the Kubernetes contributor document about writing good E2E tests and gets reproduced here to raise awareness that the document has been updated.

Overall architecture

At the moment, the framework is used in-tree for testing against a cluster (test/e2e), testing kubeadm (test/e2e_kubeadm) and kubelet (test/e2e_node). The goal is to make the core test/e2e/framework a package that has no dependencies on internal code and that can be used in different E2E suites without polluting them with features or options that make no sense for them. This is currently only a technical goal. There are no plans anymore to actually move the code into a staging repository.

The framework acts like a normal client of an apiserver and thus doesn’t need much more than client-go. Since the sub-package refacoring, additional sub-packages like test/e2e/framework/pod depend on the framework, not the other way around. Those other sub-packages therefore can still use internal code. The import boss configuration enforces these constraints.

What’s left to clean up is that the framework contains a TestContext with fields that are used only by some tests or some test suites. The configuration for test/e2e_node is the last remaining dependency on internal code. Such settings should get moved into the different test suites and/or tests. The advantage besides avoiding such dependencies will be that they will only show up in the command line of a suite when the option really has an effect.

Debuggability

If your test fails, it should provide as detailed as possible reasons for the failure in its failure message. The failure message is the string that gets passed (directly or indirectly) to ginkgo.Fail[f]. That text is what gets shown in the overview of failed tests for a Prow job and what gets aggregated by https://go.k8s.io/triage.

A good failure message:

identifies the test failure
has enough details to provide some initial understanding of what went wrong

It’s okay for it to contain information that changes during each test run. Aggregation simplifies the failure message with regular expressions before looking for similar failures.

Helper libraries like Gomega or testify can be used to produce informative failure messages. Gomega is a bit easier to use in combination with Ginkgo.

The E2E framework itself only has one helper function for assertions that is still recommended. The others are deprecated. Compared to gomega.Expect(err).NotTo(gomega.HaveOccurred()), framework.ExpectNoError(err) is shorter and produces better failure messages because it logs the full error and then includes only the shorter err.Error() in the failure message.

As with any other assertion, it is recommended to include additional context in cases where the parameters being checked by an assertion helper lack relevant information:

framework.ExpectNoError(err, "tried creating %d foobars, only created %d", foobarsReqd, foobarsCreated)

Use assertions that match the check in the test. Using Go code to evaluate some condition and then checking the result often isn’t informative. For example this check should be avoided:

gomega.Expect(strings.Contains(actualStr, expectedSubStr)).To(gomega.Equal(true))

Comparing a boolean like this against true or false with gomega.Equal or framework.ExpectEqual is not useful because dumping the actual and expected value just distracts from the underlying failure reason. Better pass the actual values to Gomega, which will automatically include them in the failure message. Add an annotation that explains what the assertion is about:

gomega.Expect(actualStr).To(gomega.ContainSubstring("xyz"), "checking log output")

This produces the following failure message:

 [FAILED] checking log output
Expected
<string>: hello world
to contain substring
<string>: xyz

If there is no suitable Gomega assertion, call ginkgo.Failf directly:

import "github.com/onsi/gomega/format"
ok := someCustomCheck(abc)
if !ok {
ginkgo.Failf("check xyz failed for object:\n%s", format.Object(abc))
}

It is good practice to include details like the object that failed some assertion in the failure message because then a) the information is available when analyzing a failure that occurred in the CI and b) it only gets logged when some assertion fails. Always dumping objects via log messages can make the test output very large and may distract from the relevant information.

Dumping structs with format.Object is recommended. Starting with Kubernetes 1.26, format.Object will pretty-print Kubernetes API objects or structs as YAML and omit unset fields, which is more readable than other alternatives like fmt.Sprintf("%+v").

import (
"fmt"
"k8s.io/api/core/v1"
"k8s.io/kubernetes/test/utils/format"
)
var pod v1.Pod
fmt.Printf("Printf: %+v\n\n", pod)
fmt.Printf("format.Object:\n%s", format.Object(pod, 1 /* indent one level */))
=>
Printf: {TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name: GenerateName: Namespace: SelfLink: UID: ResourceVersion: Generation:0 CreationTimestamp:0001-01-01 00:00:00 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[] Finalizers:[] ManagedFields:[]} Spec:{Volumes:[] InitContainers:[] Containers:[] EphemeralContainers:[] RestartPolicy: TerminationGracePeriodSeconds:<nil> ActiveDeadlineSeconds:<nil> DNSPolicy: NodeSelector:map[] ServiceAccountName: DeprecatedServiceAccount: AutomountServiceAccountToken:<nil> NodeName: HostNetwork:false HostPID:false HostIPC:false ShareProcessNamespace:<nil> SecurityContext:nil ImagePullSecrets:[] Hostname: Subdomain: Affinity:nil SchedulerName: Tolerations:[] HostAliases:[] PriorityClassName: Priority:<nil> DNSConfig:nil ReadinessGates:[] RuntimeClassName:<nil> EnableServiceLinks:<nil> PreemptionPolicy:<nil> Overhead:map[] TopologySpreadConstraints:[] SetHostnameAsFQDN:<nil> OS:nil HostUsers:<nil> SchedulingGates:[] ResourceClaims:[]} Status:{Phase: Conditions:[] Message: Reason: NominatedNodeName: HostIP: PodIP: PodIPs:[] StartTime:<nil> InitContainerStatuses:[] ContainerStatuses:[] QOSClass: EphemeralContainerStatuses:[] Resize:}}
format.Object:
<v1.Pod>:
metadata:
creationTimestamp: null
spec:
containers: null
status: {}

Recovering from test failures

All tests should ensure that a cluster is restored to the state that it was in before the test ran. ginkgo.DeferCleanup is recommended for this because it can be called similar to defer directly after setting up something. It is better than defer because Ginkgo will show additional details about which cleanup code is running and (if possible) handle timeouts for that code (see next section). It is better than ginkgo.AfterEach because it is not necessary to define additional variables and because ginkgo.DeferCleanup executes code in the more useful last-in-first-out order, i.e. things that get set up first get removed last.

Objects created in the test namespace do not need to be deleted because deleting the namespace will also delete them. However, if deleting an object may fail, then explicitly cleaning it up is better because then failures or timeouts related to it will be more obvious.

In cases where the test may have removed the object, framework.IgnoreNotFound can be used to ignore the “not found” error:

podClient := f.ClientSet.CoreV1().Pods(f.Namespace.Name)
pod, err := podClient.Create(ctx, testPod, metav1.CreateOptions{})
framework.ExpectNoError(err, "create test pod")
ginkgo.DeferCleanup(framework.IgnoreNotFound(podClient.Delete), pod.Name, metav1.DeleteOptions{})

Interrupting tests

When aborting a manual gingko ./test/e2e invocation with CTRL-C or a signal, the currently running test(s) should stop immediately. This gets achieved by accepting a ctx context.Context as first parameter in the Ginkgo callback function and then passing that context through to all code that might block. When Ginkgo notices that it needs to shut down, it will cancel that context and all code trying to use it will immediately return with a context canceled error. Cleanup callbacks get a new context which will time out eventually to ensure that tests don’t get stuck. For a detailed description, see https://onsi.github.io/ginkgo/#interrupting-aborting-and-timing-out-suites. Most of the E2E tests were update to use the Ginkgo context at the start of the 1.27 development cycle.

There are some gotchas:

Don’t use the ctx passed into ginkgo.It in a ginkgo.DeferCleanup callback because the context will be canceled when the cleanup code runs. This is wrong:
```
 ginkgo.It("something", func(ctx context.Context) {
...
ginkgo.DeferCleanup(func() {
// do something with ctx
})
})
```
Instead, register a function which accepts a new context:
```
 ginkgo.DeferCleanup(func(ctx context.Context) {
// do something with the new ctx
})
```
Anonymous functions can be avoided by passing some existing function and its parameters directly to ginkgo.DeferCleanup. Again, beware to not pass the wrong ctx. This is wrong:
```
 ginkgo.It("something", func(ctx context.Context) {
...
ginkgo.DeferCleanup(myDeleteFunc, ctx, objName)
})
```
Instead, just pass the other parameters and let ginkgo.DeferCleanup add a new context:
```
 ginkgo.DeferCleanup(myDeleteFunc, objName)
```
When starting some background goroutine in a ginkgo.BeforeEach callback, use context.WithCancel(context.Background()). The context passed into the callback will get canceled when the callback returns, which would cause the background goroutine to stop before the test runs. This works:
```
 backgroundCtx, cancel := context.WithCancel(context.Background())
ginkgo.DeferCleanup(cancel)
_, controller = cache.NewInformer( ... )
go controller.Run(backgroundCtx.Done())
```
When adding a timeout to the context for one particular operation, beware of overwriting the ctx variable. This code here applies the timeout to the next call and everything else that follows:
```
 ctx, cancel := context.WithTimeout(ctx, 5 * time.Second)
defer cancel()
someOperation(ctx)
...
anotherOperation(ctx)
```
Better use some other variable name:
```
 timeoutCtx, cancel := context.WithTimeout(ctx, 5 * time.Second)
defer cancel()
someOperation(timeoutCtx)
```
When the intention is to set a timeout for the entire callback, use ginkgo.NodeTimeout:
```
 ginkgo.It("something", ginkgo.NodeTimeout(30 * time.Second), func(ctx context.Context) {
})
```
There is also a ginkgo.SpecTimeout, but that then also includes the time taken for BeforeEach, AfterEach and DeferCleanup callbacks.

Polling and timeouts

When waiting for something to happen, use a reasonable timeout. Without it, a test might keep running until the entire test suite gets killed by the CI. Beware that the CI under load may take a lot longer to complete some operation compared to running the same test locally. On the other hand, a too long timeout also has drawbacks:

When a feature is broken so that the expected state doesn’t get reached, a test waiting for that state first needs to time out before the test fails.
If a state is expected to be reached within a certain time frame, then a timeout that is much higher will cause test runs to be considered successful although the feature was too slow. A dedicated performance test in a well-know environment may be a better solution for testing such performance expectations.

The framework provides some common timeouts through the framework instance. When writing a test, check whether one of those fits before defining a custom timeout in the test.

Good code that waits for something to happen meets the following criteria:

accepts a context for test timeouts
depending on how the test suite was invoked:
- informative during interactive use (i.e. intermediate reports, either periodically or on demand)
- little to no output during a CI run except when it fails
full explanation when it fails: when it observes some state and then encounters errors reading the state, then dumping both the latest observed state and the latest error is useful
extension mechanism for writing custom checks
early abort when condition cannot be reached anymore

gomega.Eventually satisfies all of these criteria and therefore is recommended, but not required. In https://github.com/kubernetes/kubernetes/pull/113298, test/e2e/framework/pods/wait.go and the framework were modified to use gomega. Typically, Eventually is passed a function which gets an object or lists several of them, then Should checks against the expected result. Errors and retries specific to Kubernetes are handled by wrapping client-go functions.

Using normal Gomega assertions in helper packages is problematic for two reasons:

The stacktrace associated with the failure starts with the helper unless extra care is take to pass in a stack offset.
Additional explanations for a potential failure must be prepared beforehand and passed in.

The E2E framework therefore uses a different approach:

framework.Gomega() offers similar functions as the gomega package, except that they return a normal error instead of failing the test.
That error gets wrapped with fmt.Errorf("<explanation>: %w) to add additional information, just as in normal Go code.
Wrapping the error (%w instead of %v) is important because then framework.ExpectNoError directly uses the error message as failure without additional boiler plate text. It also is able to log the stacktrace where the error occurred and not just where it was finally treated as a test failure.

Tips for writing and debugging long-running tests

Use ginkgo.By to record individual steps. Ginkgo will use that information when describing where a test timed out.
Invoke the ginkgo CLI with --poll-progress-after=30s or some other suitable duration to be informed early why a test doesn’t complete and where it is stuck. A SIGINFO or SIGUSR1 signal can be sent to the CLI and/or e2e.test processes to trigger an immediate progress report.
Use gomega.Eventually to wait for some condition. When it times out or gets stuck, the last failed assertion will be included in the report automatically. A good way to invoke it is:
```
 gomega.Eventually(ctx, func(ctx context.Context) (book Book, err error) {
// Retrieve book from API server and return it.
...
}).WithPolling(5 * time.Second).WithTimeout(30 * time.Second).
Should(gomega.HaveField("Author.DOB.Year()", BeNumerically("<", 1900)))
```
Avoid testing for some condition inside the callback and returning a boolean because then failure messages are not informative (see above). See https://github.com/kubernetes/kubernetes/pull/114640 for an example where gomega/gcustom was used to write assertions.

Some of the E2E framework sub-packages have helper functions that wait for certain domain-specific conditions. Currently most of these functions don’t follow best practices (not using gomega.Eventually, error messages not very informative). Work is planned in that area, so beware that these APIs may change at some point.
Use gomega.Consistently to ensure that some condition is true for a while. As with gomega.Eventually, make assertions about the value instead of checking the value with Go code and then asserting that the code returns true.
Both gomega.Consistently and gomega.Eventually can be aborted early via gomega.StopPolling.
Avoid polling with functions that don’t take a context (wait.Poll, wait.PollImmediate, wait.Until, …) and replace with their counterparts that do (wait.PollWithContext, wait.PollImmediateWithContext, wait.UntilWithContext, …) or even better, with gomega.Eventually.

Next steps

Using wait.Poll in E2E tests can be detected with forbidigo since import alias support was merged. In Kubernetes, that can be enabled in a golangci-lint invocation as soon as a configuration extension is merged. Another enhancement would be useful, but not absolutely required.

Because a lot of existing code wouldn’t pass such a check, it probably will only be enabled in the new stricter pull request linting initially. Converting individual sub packages similar to test/e2e/framework/pod to match current best practices would be a good way for new contributors to get involved.

The SIG Testing’s Slack channel is a good place to start. At KubeCon EU 2023, the “Keeping the lights on and the bugs away” talk will cover some of the material of this blog post. Catch me there or meet me at the Intel booth to discuss this further!

Blog: From Zero to Kubernets Subproject Lead

Wed, 29 Mar 2023 00:00:00 +0000

Getting started in any open-source community can be daunting, especially if it’s a big one like Kubernetes. I wrote this post to share my experience and encourage others to join up. All it takes is some curiosity and a willingness to show up!

Here’s how my journey unfolded at a high level:

What am I interested in? Is there a SIG (Special Interest Group) or a WG (Working Group) that is dedicated to that topic, or something similar?
Sign up for their mailing list and start hopping on meetings.
When (never if!) there are opportunities to help out and it aligns with your skills and desired growth areas, raise your hand.
Ask for lots of help and don’t be shy about not knowing everything (or anything!)
Keep plugging along, even if progress isn’t as fast as you would like it to be.

Starting up

First things first. What are you interested in learning more about? There are so many wonderful SIGs and working groups in the Kubernetes community: there’s something for everyone. And continuing to show up and participate will be so much easier if you think what you are doing is interesting. Likewise, continued participation is what keeps the community thriving, so that interest will drive you to have more of an impact.

Also: it’s ok to show up knowing nothing! I remember showing up knowing very little about Kubernetes or how the community itself worked. And while I know more about how the community functions today, I am still learning all the time about it and the project. Fortunately, the community is full of friendly people who want to help you learn. Learning as you go is expected and celebrated. When you raise your hand to do something, even if you know nothing, people will cheer and help you along the way.

This method was my exact story. It was my first or second meeting with SIG Security, and Pushkar Joglekar mentioned that he needed a lead for a subproject he was creating after having done a security assessment of Cluster API. Everyone was so friendly in the meeting that I thought, “Hey, why not try it out?” And since then, I have received so much support and encouragement from my co-leads who are delighted to have me, especially because I am a beginner; new participation is what keeps the community healthy.

Always learning

My participation has also been a great learning experience on several fronts. First, I have been exposed to techniques for how to build community consensus. It’s simple stuff: show up at other SIG or working group meetings, share your ideas or where you are looking for help, find people who are interested and have the knowledge to help, build an action plan together, do it, and share as you execute. But the other thing that I’m learning is that building this consensus and executing it in a transparent, inviting way simply takes time.

I also have to be patient with myself and remember that I am learning as I go. The Kubernetes git repo can be daunting to navigate. Knowing the next best step isn’t always obvious. But this is where my third learning curve, how to engage the community to get what I need, comes into play. It turns out that asking questions in the Kubernetes Slack workspace and bringing my topics to the SIG Security meetings when I need help is an amazing way to get what I need! Again, simple stuff, but until you do it, it’s not always obvious.

Why you - a beginner - are important to the project

In many ways, beginners are the most important part of the community. To put a finer point on it: asking for, receiving, and then giving help is a very relevant part of how the community grows and flourishes. When we take on and then pass on knowledge, we ensure that the community grows enough to keep supporting the needs of the people who rely on the project, whatever it is. You have superpowers as a beginner!

I hope people who read this post have their curiosity peaked about getting involved in the community. It may seem scary. My experience has been such that, about halfway through your first step, you realize there are loads of people here who want to help you learn and are excited for you expressing interest and trying to participate, and the fear melts away. Sure, I’m still uncertain about a few things, but I know the community has my back and will support my growth.

Come on in, that water’s fine!

Blog: Introducing KWOK: Kubernetes WithOut Kubelet

Wed, 01 Mar 2023 00:00:00 +0000

Author: Shiming Zhang (DaoCloud), Wei Huang (Apple), Yibo Zhuang (Apple)

Have you ever wondered how to set up a cluster of thousands of nodes just in seconds, how to simulate real nodes with a low resource footprint, and how to test your Kubernetes controller at scale without spending much on infrastructure?

If you answered “yes” to any of these questions, then you might be interested in KWOK, a toolkit that enables you to create a cluster of thousands of nodes in seconds.

What is KWOK?

KWOK stands for Kubernetes WithOut Kubelet. So far, it provides two tools:

kwok: kwok is the cornerstone of this project, responsible for simulating the lifecycle of fake nodes, pods, and other Kubernetes API resources.
kwokctl: kwokctl is a CLI tool designed to streamline the creation and management of clusters, with nodes simulated by kwok.

Why use KWOK?

KWOK has several advantages:

Speed: You can create and delete clusters and nodes almost instantly, without waiting for boot or provisioning.
Compatibility: KWOK works with any tools or clients that are compliant with Kubernetes APIs, such as kubectl, helm, kui, etc.
Portability: KWOK has no specific hardware or software requirements. You can run it using pre-built images, once Docker or Nerdctl is installed. Alternatively, binaries are also available for all platforms and can be easily installed.
Flexibility: You can configure different node types, labels, taints, capacities, conditions, etc., and you can configure different pod behaviors, status, etc. to test different scenarios and edge cases.
Performance: You can simulate thousands of nodes on your laptop without significant consumption of CPU or memory resources.

What are the use cases?

KWOK can be used for various purposes:

Learning: You can use KWOK to learn about Kubernetes concepts and features without worrying about resource waste or other consequences.
Development: You can use KWOK to develop new features or tools for Kubernetes without accessing to a real cluster or requiring other components.
Testing:
- You can measure how well your application or controller scales with different numbers of nodes and(or) pods.
- You can generate high loads on your cluster by creating many pods or services with different resource requests or limits.
- You can simulate node failures or network partitions by changing node conditions or randomly deleting nodes.
- You can test how your controller interacts with other components or features of Kubernetes by enabling different feature gates or API versions.

What are the limitations?

KWOK is not intended to replace others completely. It has some limitations that you should be aware of:

Functionality: KWOK is not a kubelet and may exhibit different behaviors in areas such as pod lifecycle management, volume mounting, and device plugins. Its primary function is to simulate updates of node and pod status.
Accuracy: It’s important to note that KWOK doesn’t accurately reflect the performance or behavior of real nodes under various workloads or environments. Instead, it approximates some behaviors using simple formulas.
Security: KWOK does not enforce any security policies or mechanisms on simulated nodes. It assumes that all requests from the kube-apiserver are authorized and valid.

Getting started

If you are interested in trying out KWOK, please check its documents for more details.

Using kwokctl to manage simulated clusters

Getting Involved

If you’re interested in participating in future discussions or development related to KWOK, there are several ways to get involved:

Slack: #kwok for general usage discussion, #kwok-dev for development discussion. (visit slack.k8s.io for a workspace invitation)
Open Issues/PRs/Discussions in sigs.k8s.io/kwok

We welcome feedback and contributions from anyone who wants to join us in this exciting project.

Blog: Spotlight on SIG Instrumentation

Fri, 03 Feb 2023 00:00:00 +0000

Observability requires the right data at the right time for the right consumer (human or piece of software) to make the right decision. In the context of Kubernetes, having best practices for cluster observability across all Kubernetes components is crucial.

SIG Instrumentation helps to address this issue by providing best practices and tools that all other SIGs use to instrument Kubernetes components-like the API server, scheduler, kubelet and kube-controller-manager.

In this SIG Instrumentation spotlight, Imran Noor Mohamed, SIG ContribEx-Comms tech lead talked with Elana Hashman, and Han Kang, chairs of SIG Instrumentation, on how the SIG is organized, what are the current challenges and how anyone can get involved and contribute.

About SIG Instrumentation

Imran (INM): Hello, thank you for the opportunity of learning more about SIG Instrumentation. Could you tell us a bit about yourself, your role, and how you got involved in SIG Instrumentation?

Han (HK): I started in SIG Instrumentation in 2018, and became a chair in 2020. I primarily got involved with SIG instrumentation due to a number of upstream issues with metrics which ended up affecting GKE in bad ways. As a result, we ended up launching an initiative to stabilize our metrics and make metrics a proper API.

Elana (EH): I also joined SIG Instrumentation in 2018 and became a chair at the same time as Han. I was working as a site reliability engineer (SRE) on bare metal Kubernetes clusters and was working to build out our observability stack. I encountered some issues with label joins where Kubernetes metrics didn’t match kube-state-metrics (KSM) and started participating in SIG meetings to improve things. I helped test performance improvements to kube-state-metrics and ultimately coauthored a KEP for overhauling metrics in the 1.14 release to improve usability.

Imran (INM): Interesting! Does that mean SIG Instrumentation involves a lot of plumbing?

Han (HK): I wouldn’t say it involves a ton of plumbing, though it does touch basically every code base. We have our own dedicated directories for our metrics, logs, and tracing frameworks which we tend to work out of primarily. We do have to interact with other SIGs in order to propagate our changes which makes us more of a horizontal SIG.

Imran (INM): Speaking about interaction and coordination with other SIG could you describe how the SIGs is organized?

Elana (EH): In SIG Instrumentation, we have two chairs, Han and myself, as well as two tech leads, David Ashpole and Damien Grisonnet. We all work together as the SIG’s leads in order to run meetings, triage issues and PRs, review and approve KEPs, plan for each release, present at KubeCon and community meetings, and write our annual report. Within the SIG we also have a number of important subprojects, each of which is stewarded by its subproject owners. For example, Marek Siarkowicz is a subproject owner of metrics-server.

Because we’re a horizontal SIG, some of our projects have a wide scope and require coordination from a dedicated group of contributors. For example, in order to guide the Kubernetes migration to structured logging, we chartered the Structured Logging Working Group (WG), organized by Marek and Patrick Ohly. The WG doesn’t own any code, but helps with various components such as the kubelet, scheduler, etc. in migrating their code to use structured logs.

Imran (INM): Walking through the charter alone it’s clear that SIG Instrumentation has a lot of sub-projects. Could you highlight some important ones?

Han (HK): We have many different sub-projects and we are in dire need of people who can come and help shepherd them. Our most important projects in-tree (that is, within the kubernetes/kubernetes repo) are metrics, tracing, and, structured logging. Our most important projects out-of-tree are (a) KSM (kube-state-metrics) and (b) metrics-server.

Elana (EH): Echoing this, we would love to bring on more maintainers for kube-state-metrics and metrics-server. Our friends at WG Structured Logging are also looking for contributors. Other subprojects include klog, prometheus-adapter, and a new subproject that we just launched for collecting high-fidelity, scalable utilization metrics called usage-metrics-collector. All are seeking new contributors!

Current status and ongoing challenges

Imran (INM): For release 1.26 we can see that there are a relevant number of metrics, logs, and tracing KEPs in the pipeline. Would you like to point out important things for last release (maybe alpha & stable milestone candidates?)

Han (HK): We can now generate documentation for every single metric in the main Kubernetes code base! We have a pretty fancy static analysis pipeline that enables this functionality. We’ve also added feature metrics so that you can look at your metrics to determine which features are enabled in your cluster at a given time. Lastly, we added a component-sli endpoint, which should make it easy for people to create availability SLOs for control-plane components.

Elana (EH): We’ve also been working on tracing KEPs for both the API server and kubelet, though neither graduated in 1.26. I’m also really excited about the work Han is doing with WG Reliability to extend and improve our metrics stability framework.

Imran (INM): What do you think are the Kubernetes-specific challenges tackled by the SIG Instrumentation? What are the future efforts to solve them?

Han (HK): SIG instrumentation suffered a bit in the past from being a horizontal SIG. We did not have an obvious location to put our code and did not have a good mechanism to audit metrics that people would randomly add. We’ve fixed this over the years and now we have dedicated spots for our code and a reliable mechanism for auditing new metrics. We also now offer stability guarantees for metrics. We hope to have full-blown tracing up and down the kubernetes stack, and metric support via exemplars.

Elana (EH): I think SIG Instrumentation is a really interesting SIG because it poses different kinds of opportunities to get involved than in other SIGs. You don’t have to be a software developer to contribute to our SIG! All of our components and subprojects are focused on better understanding Kubernetes and its performance in production, which allowed me to get involved as one of the few SIG Chairs working as an SRE at that time. I like that we provide opportunities for newcomers to contribute through using, testing, and providing feedback on our subprojects, which is a lower barrier to entry. Because many of these projects are out-of-tree, I think one of our challenges is to figure out what’s in scope for core Kubernetes SIGs instrumentation subprojects, what’s missing, and then fill in the gaps.

Community and contribution

Imran (INM): Kubernetes values community over products. Any recommendation for anyone looking into getting involved in SIG Instrumentation work? Where should they start (new contributor-friendly areas within SIG?)

Han(HK) and Elana (EH): Come to our bi-weekly triage meetings! They aren’t recorded and are a great place to ask questions and learn about our ongoing work. We strive to be a friendly community and one of the easiest SIGs to get started with. You can check out our latest KubeCon NA 2022 SIG Instrumentation Deep Dive to get more insight into our work. We also invite you to join our Slack channel #sig-instrumentation and feel free to reach out to any of our SIG leads or subproject owners directly.

Thank you so much for your time and insights into the workings of SIG Instrumentation!

Blog: Prow and Tide for Kubernetes Contributors

Mon, 12 Dec 2022 00:00:00 +0000

Authors: Chris Short, Frederico Muñoz

In my work in the Kubernetes world, I look up a label or Prow command often. The systems behind the scenes (Prow and Tide) are here to help Kubernetes Contributors get stuff done.

Labeling which SIG, WG, or subproject is as important as the issue or PR having someone assigned. To quote the docs, “Tide is a Prow component for managing a pool of GitHub PRs that match a given set of criteria. It will automatically retest PRs that meet the criteria (’tide comes in’) and automatically merge them when they have up-to-date passing test results (’tide goes out’).”

What actually prompted this article is the awesomely amazing folks on the Contributor Comms team saying, “I need to squash my commits and push that.” Which immediately made me remember the wonder of the Tide label: tide/merge-method-squash.

Why is this helpful

Contributing to Kubernetes will, most of the time, involve some kind of git-based action, specifically on the Kubernetes GitHub. This can be an obstacle to those less exposed to git and/or GitHub, and is especially noticeable when we’re dealing with non-code contributions (documentation, blog posts, etc.).

When a contributor submits something, it will generally be through a pull request. When it comes to how the change will go from request to approval, there are a number of considerations that must be made, such as:

How should we request reviews?
How do we assign the request to a specific SIG?
How do we approve things while making it public and easily traceable?
How to merge a contribution without carrying all the commit messages that were created during the review?

These are some of the main tasks in which Tide will help, allowing us to use the GitHub interface for these tasks (and more), making the actions more visible to the community (since they are visible as plain comments in the GitHub discussion), and allowing us to manage contributions without necessarily having to clone git repositories or having to manually issue git commands.

Back to squashing

One of the most common examples, and getting back to my initial one, is squashing commits: if someone makes a change in a PR, there will likely be reviews and changes, and each one of them will add a new commit message. If left like this, the PR will add to the main branch all the commit messages created during the review process, which will make the history of the main branch less readable: instead of a single informative description about a specific unit of work, it will contain multiple commit messages that, out of the original context of the PR, will not be very helpful.

To avoid this, we can squash the commit messages to keep just one of them (usually, the first one): the changes will still be visible for anyone reading the PR, but they will appear as a single commit to the main branch.

This can be done through git, and a cursory read of how it’s done will not be obvious to someone relatively new to git! Is there a way to avoid cloning the repository (or PR), issuing git commands locally, and pushing the changes?

There is, with Tide. Squashing your commits is a label away and the tooling will do the rest. To do this on your PR you’ll need to comment with the following:

/label tide/merge-method-squash

This will:

Trigger Tide to squash the messages prior to merging.
As a secondary effect, make your action clearly visible in the discussion section of the PR.

This use of Tide is one of the most useful when submitting changes that undergo changes during the PR discussion, since it automates something to do The Right Thing (TM).

There’s often nothing better than an example, so let’s take a look at this proposed change to the Kubernetes website, on the topic of the dockershim removal FAQ. The initial commit is followed by several others, a result of the conversation and proposed reviews. The result of all those changes is merged as a single commit, with the commit message retaining the title of the very first commit done, and the commit description being the aggregate of all the commits done in the PR. This was achieved by using /label tide/merge-method-squash in a comment, and did away with the need to manually rebase and/or squash using git: everything was possible through the GitHub interface.

Assignment, review, approval.

Another area in which Prow and Tide are very useful is in dealing with assignments, reviews, and approvals.

Starting with assignment, the need to assign a PR to someone is very common. There are ways to do it through the GitHub interface, but using Prow commands, as mentioned before, makes the actions more visible and explicitly trigger the automation mechanism. Using /assign in a comment will assign the PR to yourself, or reassign it to someone else.

Asking for reviews is another very common task: with Prow it’s just a /cc @foo @bar @baz away, and this can be directly added in the initial PR description, or in any subsequent comment.

Approving a PR is one area in which making the process easily visible is very important, and, unsurprisingly, we can use /lgtm (looks good to me) to publicly state our agreement, and at the same time trigger the automated processes that will, hopefully, result in the merging of the contribution. Using lgtm adds (or, if using /lgtm cancel, removes) the lgtm label, while using /approve will approve the PR for merging (and can only be used by those with the necessary authorization).

To summarize, /assign makes it public that an assignment as been made (I use it often when I need to assign an issue to myself), and by who; /lgtm makes it clear from the comments that a review was made and, automatically, adds the lgtm label which is required for approval, and /approve approves the PR for merging.

An example of many of these is this update to the Kubernetes Community site: we can see how additional reviewers were added with /cc, and following the discussion and changes, both the /lgtm and /approve commands are used to trigger the merging.

More information on the review and approval cycle can be found in the documentation, which also explains in more detail when should certain commands be used, and by who.

More about Prow and Tide

The previous examples are some of the most commonly used, but Prow and Tide provide a lot more. These two pages document all the functionality available to Kubernetes contributors (either through labels or Prow):

Labels are most commonly applied by using an associated command (e.g. /lgtm, instead of /label lgtm): only when no such command exists are labels applied directly with the label command; some of my more often used commands and labels:

/assign (using it without adding a name assigns yourself)
/honk
/(woof|bark|this-is-{fine|not-fine|unbearable})
/remove-lifecycle stale (when issues aren’t touched for a period of time they’re marked stale)
/shrug
/area contributor-comms (You can use this to flag down the contributor communications team for reviews, comments on any issue, feedback, etc.)
/label size/X (Sizes are assigned automatically based on the number of lines changed in the PR)
/hold (This one is used for many things; if your PR is a work in progress, needs to be held to a certain date, etc.)
/lgtm (Adds or removes the ’lgtm’ label which is typically used to gate merging)
/approve (Approves a pull request; must be done by someone in the repo’s OWNERS file)

More advanced usage

What if you need a label that isn’t available on a certain GitHub repository? I’m glad you asked! This PR demonstrates how to add labels to a repo: https://github.com/kubernetes/test-infra/pull/24315. You’ll need to update the labels.yaml file (the configuration) and the labels.md file (documentation).

This is why the label_sync tool, along with the logic Prow and Tide, simplify GitHub-based processes: they allow the automation of common actions without necessarily having to leave the web-based GitHub interface. label_sync ensures that labels are applied uniformly across repositories.

I’ve done this once in five years of contributing. But, it’s good to write it down as it’s something that isn’t as trivial as you think because of the importance of the label_sync tooling.

These are a handful of the commands and labels I enjoy. I’m sure there are many others that are helpful to folks. With that in mind, see if there’s something you can benefit from in these resources. They are there to make working on Kubernetes a better experience. If you think there’s some functionality missing, I’d invite you to drop a Slack message in SIG ContribEx or SIG Testing to discuss.

Huge shoutout: To the folks that keep these systems humming along for the Kubernetes community. Couldn’t do it without y’all.

Blog: Implementing the Auto-refreshing Official Kubernetes CVE Feed

Mon, 12 Sep 2022 00:00:00 +0000

Author: Pushkar Joglekar (VMware)

Accompanying the release of Kubernetes v1.25, we announced availability of an official CVE feed as an alpha feature. This blog will cover how we implemented this feature.

Implementation Details

An auto-refreshing CVE feed allows users and implementers to programmatically fetch the list of CVEs announced by the Kubernetes SRC (Security Response Committee).

To ensure freshness and minimal maintainer overhead, the feed updates automatically by fetching the CVE related information from the CVE announcement GitHub Issues. Creating these issues is already part of the existing Security Response Committee (SRC) workflow.

Pre-requisites

Until December 2021, it was not possible to filter for issues or PRs that are tied to CVEs announced by Kubernetes SRC. We added a new label, official-cve-feed to address that, and SIG-Security labelled relevant issues with it. The in-scope issues are closed issues for which there is a CVE ID(s) and is officially announced as a Kubernetes security vulnerability by SRC. You can now filter on all of these issues and find them here .

For future security vulnerabilities, we added the label to the SRC playbook so that all the future in-scope issues will automatically have this label.

Building on existing tooling

For the next step, we created a prow job in order to periodically query the GitHub REST API and pull the relevant issues. The job runs every two hours and pushes the CVE related information fetched from GitHub into a Google Cloud Bucket.

For every website build (at least twice a day), Netlify data templates make a call to this Google Cloud Bucket to pull the CVE information and then parses into fields that are JSON Feed v1.1 compliant. The JSON file is available for programmatic consumption by automated security tools. For humans, the JSON also gets transformed into a Markdown table for easy viewing.

Design Considerations

Building trust and ensuring that the feed is not stale were our main priorities when designing this feature for success and widespread adoption.

Integrity and Access Control Protections

Changes to any of the four artifacts used to build this feed could lead to feed tampering, broken JSON, and inconsistent or stale data.

Let’s look at how access is controlled for them one by one:

GitHub Issues for Publicly Announced CVEs

Adding the official-cve-feed label is restricted to limited number of trusted community members. Access to add this label is defined in this configuration file . Any updates to this configuration file require the changes to go through the existing code review and approval process

Prow Configuration

The Prow job is defined in a kubernetes/infra configuration file The shell script to push and pull the data in Google Cloud Bucket is defined in a kubernetes/sig-security file under sig-security-tooling sub-project. Both of these files go through the same code review and approval process mentioned earlier.

Google Cloud Bucket

Write access to Google Cloud bucket is configured to be restricted to a set of trusted community members managed via an invite-only Google Groups Membership under the kubernetes.io domain.

Website Data templates

Website data templates that fetch and parse the stored JSON blob are managed under kubernetes/website and have to follow the same code review and approval process as mentioned earlier.

Freshness Guarantees

The feed is updated everytime new CVE data is available by periodically verifying if generated data is not the same as the stored data in the feed.

The prow job runs every two hours and compares the sha256 checksum of the existing contents of the bucket with checksum of the latest JSON file generated through GitHub issues. If the there is new data available, the hashes do not match (typically because of a newly announced CVE) and the updated JSON file is pushed onto the bucket replacing the old file and old hash checksum.

If the hashes match, the write to bucket operation is skipped to reduce redundant updates to the cloud bucket. This also sets us up for more frequent runs of the prow job if needed in the future.

What’s Next?

If you would like to get involved in future iterations of this feed or other security relevant work, please consider joining Kubernetes SIG Security by joining our bi-weekly meetings or hanging out with us on our Slack Channel.

A special shout out and massive thanks to Neha Lohia (@nehalohia27) and Tim Bannister (@sftim) for their stellar collaboration for many months from “ideation to implementation” of this feature.

Blog: Enhancements Opt-in Process Change for v1.26

Fri, 09 Sep 2022 00:00:00 +0000

Author: Grace Nguyen

Context and Motivations

Since the inception of the Kubernetes release team, we have used a spreadsheet to keep track of enhancements for the release. The project has scaled massively in the past few years, with almost a hundred enhancements collected for the 1.24 release. This process has become error-prone and time consuming. A lot of manual work is required from the release team and the SIG leads to populate KEPs data in the sheet. We have received continuous feedback from our contributors to streamline the process.

Starting with the 1.26 release, we are replacing the enhancements tracking spreadsheet with an automated GitHub project board.

How does the Github Project Board work?

The board is populated with a script gathering all KEP issues in the kubernetes/enhancements repo that have the label lead-opted-in. The enhancements’ stage and SIG information will also be automatically pulled from the KEP issue.

After the KEP is populated on the Github Project Board, the Enhancements team will manually update the KEP with the label tracked/yes, tracked/no and on occasions, tracked/out-of-tree. The tracked label signifies qualification for the closest approaching milestone. For example, at the beginning of the release, tracked/yes means that the KEP has satisfied all Enhancements Freeze requirements and similarly for Code Freeze, tracked/yes means that all code related to the KEP has been merged. The tracked label is reserved for the Enhancements team use only.

What does this mean for the community?

If you are not a SIG lead, nothing will change beside the view of the enhancements collections and the change of platform. KEP authors will continue working with their respective SIG leads to opt in to the release.

For SIG leads, opting in is simple. The KEP issue will be the single source of truth so ensure that all metadata is up to date. Simply comment /label lead-opted-in on the enhancement tracking issue to opt it into the current release. That’s all you need to do to opt in! Since the script runs periodically, kindly come back to check that the KEP is on the board, labeled with tracked/yes, and that there is an Enhancements team member assigned to it.

We are excited to bring this highly requested feature into our release process and appreciate your patience. Email us at release-enhancements-team@kubernetes.io or find us on Slack at #release-enhancements if you have any feedback, questions or concern.

Blog: Spotlight on SIG Storage

Mon, 22 Aug 2022 00:00:00 +0000

Since the very beginning of Kubernetes, the topic of persistent data and how to address the requirement of stateful applications has been an important topic. Support for stateless deployments was natural, present from the start, and garnered attention, becoming very well-known. Work on better support for stateful applications was also present from early on, with each release increasing the scope of what could be run on Kubernetes.

Message queues, databases, clustered filesystems: these are some examples of the solutions that have different storage requirements and that are, today, increasingly deployed in Kubernetes. Dealing with ephemeral and persistent storage, local or remote, file or block, from many different vendors, while considering how to provide the needed resiliency and data consistency that users expect, all of this is under SIG Storage’s umbrella.

In this SIG Storage spotlight, Frederico Muñoz (Cloud & Architecture Lead at SAS) talked with Xing Yang, Tech Lead at VMware and co-chair of SIG Storage, on how the SIG is organized, what are the current challenges and how anyone can get involved and contribute.

About SIG Storage

Frederico (FSM): Hello, thank you for the opportunity of learning more about SIG Storage. Could you tell us a bit about yourself, your role, and how you got involved in SIG Storage.

Xing Yang (XY): I am a Tech Lead at VMware, working on Cloud Native Storage. I am also a Co-Chair of SIG Storage. I started to get involved in K8s SIG Storage at the end of 2017, starting with contributing to the VolumeSnapshot project. At that time, the VolumeSnapshot project was still in an experimental, pre-alpha stage. It needed contributors. So I volunteered to help. Then I worked with other community members to bring VolumeSnapshot to Alpha in K8s 1.12 release in 2018, Beta in K8s 1.17 in 2019, and eventually GA in 1.20 in 2020.

FSM: Reading the SIG Storage charter alone it’s clear that SIG Storage covers a lot of ground, could you describe how the SIG is organised?

XY: In SIG Storage, there are two Co-Chairs and two Tech Leads. Saad Ali from Google and myself are Co-Chairs. Michelle Au from Google and Jan Šafránek from Red Hat are Tech Leads.

We have bi-weekly meetings where we go through features we are working on for each particular release, getting the statuses, making sure each feature has dev owners and reviewers working on it, and reminding people about the release deadlines, etc. More information on the SIG is on the community page. People can also add PRs that need attention, design proposals that need discussion, and other topics to the meeting agenda doc. We will go over them after project tracking is done.

We also have other regular meetings, i.e., CSI Implementation meeting, Object Bucket API design meeting, and one-off meetings for specific topics if needed. There is also a K8s Data Protection Working Group that is sponsored by SIG Storage and SIG Apps. SIG Storage owns or co-owns features that are being discussed at the Data Protection WG.

Storage and Kubernetes

FSM: Storage is such a foundational component in so many things, not least in Kubernetes: what do you think are the Kubernetes-specific challenges in terms of storage management?

XY: In Kubernetes, there are multiple components involved for a volume operation. For example, creating a Pod to use a PVC has multiple components involved. There are the Attach Detach Controller and the external-attacher working on attaching the PVC to the pod. There’s the kubelet that works on mounting the PVC to the pod. Of course the CSI driver is involved as well. There could be race conditions sometimes when coordinating between multiple components.

Another challenge is regarding core versus CustomResourceDefinitions (CRD), not really storage specific. CRD is a great way to extend Kubernetes capabilities while not adding too much code to the Kubernetes core itself. However, this also means there are many external components that are needed when running a Kubernetes cluster.

From the SIG Storage side, one most notable example is VolumeSnapshot. Volume snapshot APIs are defined as CRDs. API definitions and controllers are out-of-tree. There is a common snapshot controller that should be deployed on the control plane, similar to how kube-controller-manager is deployed. Although VolumeSnapshot is a CRD, it is a core feature of SIG Storage. It is recommended for the K8s cluster distros to deploy VolumeSnapshot CRDs, the snapshot controller, and the snapshot validation webhook, however, most of the time we don’t see distros deploy them. So this becomes a problem for the storage vendors: now it becomes their responsibility to deploy these non-driver specific common components. This could cause conflicts if a customer wants to use more than one storage system and deploy more than one CSI driver.

FSM: Not only the complexity of a single storage system, you have to consider how they will be used together in Kubernetes?

XY: Yes, there are many different storage systems that can provide storage to containers in Kubernetes. They don’t work the same way. It is challenging to find a solution that works for everyone.

FSM: Storage in Kubernetes also involves interacting with external solutions, perhaps more so than other parts of Kubernetes. Is this interaction with vendors and external providers challenging? Has it evolved with time in any way?

XY: Yes, it is definitely challenging. Initially Kubernetes storage had in-tree volume plugin interfaces. Multiple storage vendors implemented in-tree interfaces and have volume plugins in the Kubernetes core code base. This caused lots of problems. If there is a bug in a volume plugin, it affects the entire Kubernetes code base. All volume plugins must be released together with Kubernetes. There was no flexibility if storage vendors need to fix a bug in their plugin or want to align with their own product release.

FSM: That’s where CSI enters the game?

XY: Exactly, then there comes Container Storage Interface (CSI). This is an industry standard trying to design common storage interfaces so that a storage vendor can write one plugin and have it work across a range of container orchestration (CO) systems. Now Kubernetes is the main CO, but back when CSI just started, there were Docker, Mesos, Cloud Foundry, in addition to Kubernetes. CSI drivers are out-of-tree so bug fixes and releases can happen at their own pace.

CSI is definitely a big improvement compared to in-tree volume plugins. Kubernetes implementation of CSI has been GA since the 1.13 release. It has come a long way. SIG Storage has been working on moving in-tree volume plugins to out-of-tree CSI drivers for several releases now.

FSM: Moving drivers away from the Kubernetes main tree and into CSI was an important improvement.

XY: CSI interface is an improvement over the in-tree volume plugin interface, however, there are still challenges. There are lots of storage systems. Currently there are more than 100 CSI drivers listed in CSI driver docs. These storage systems are also very diverse. So it is difficult to design a common API that works for all. We introduced capabilities at CSI driver level, but we also have challenges when volumes provisioned by the same driver have different behaviors. The other day we just had a meeting discussing Per Volume CSI Driver Capabilities. We have a problem differentiating some CSI driver capabilities when the same driver supports both block and file volumes. We are going to have follow up meetings to discuss this problem.

Ongoing challenges

FSM: Specifically for the 1.25 release we can see that there are a relevant number of storage-related KEPs in the pipeline, would you say that this release is particularly important for the SIG?

XY: I wouldn’t say one release is more important than other releases. In any given release, we are working on a few very important things.

FSM: Indeed, but are there any 1.25 specific specificities and highlights you would like to point out though?

XY: Yes. For the 1.25 release, I want to highlight the following:

CSI Migration is an on-going effort that SIG Storage has been working on for a few releases now. The goal is to move in-tree volume plugins to out-of-tree CSI drivers and eventually remove the in-tree volume plugins. There are 7 KEPs that we are targeting in 1.25 are related to CSI migration. There is one core KEP for the general CSI Migration feature. That is targeting GA in 1.25. CSI Migration for GCE PD and AWS EBS are targeting GA. CSI Migration for vSphere is targeting to have the feature gate on by default while staying in 1.25 that are in Beta. Ceph RBD and PortWorx are targeting Beta, with feature gate off by default. Ceph FS is targeting Alpha.
The second one I want to highlight is COSI, the Container Object Storage Interface. This is a sub-project under SIG Storage. COSI proposes object storage Kubernetes APIs to support orchestration of object store operations for Kubernetes workloads. It also introduces gRPC interfaces for object storage providers to write drivers to provision buckets. The COSI team has been working on this project for more than two years now. The COSI feature is targeting Alpha in 1.25. The KEP just got merged. The COSI team is working on updating the implementation based on the updated KEP.
Another feature I want to mention is CSI Ephemeral Volume support. This feature allows CSI volumes to be specified directly in the pod specification for ephemeral use cases. They can be used to inject arbitrary states, such as configuration, secrets, identity, variables or similar information, directly inside pods using a mounted volume. This was initially introduced in 1.15 as an alpha feature, and it is now targeting GA in 1.25.

FSM: If you had to single something out, what would be the most pressing areas the SIG is working on?

XY: CSI migration is definitely one area that the SIG has put in lots of effort and it has been on-going for multiple releases now. It involves work from multiple cloud providers and storage vendors as well.

Community involvement

FSM: Kubernetes is a community-driven project. Any recommendation for anyone looking into getting involved in SIG Storage work? Where should they start?

XY: Take a look at the SIG Storage community page, it has lots of information on how to get started. There are SIG annual reports that tell you what we did each year. Take a look at the Contributing guide. It has links to presentations that can help you get familiar with Kubernetes storage concepts.

Join our bi-weekly meetings on Thursdays. Learn how the SIG operates and what we are working on for each release. Find a project that you are interested in and help out. As I mentioned earlier, I got started in SIG Storage by contributing to the Volume Snapshot project.

FSM: Any closing thoughts you would like to add?

XY: SIG Storage always welcomes new contributors. We need contributors to help with building new features, fixing bugs, doing code reviews, writing tests, monitoring test grid health, and improving documentation, etc.

FSM: Thank you so much for your time and insights into the workings of SIG Storage!

Blog: Meet Our Contributors - APAC (China region)

Mon, 15 Aug 2022 00:00:00 +0000

Authors & Interviewers: Avinesh Tripathi, Debabrata Panigrahi, Jayesh Srivastava, Priyanka Saggu, Purneswar Prasad, Vedant Kakde

Hello, everyone 👋

Welcome back to the third edition of the “Meet Our Contributors” blog post series for APAC.

This post features four outstanding contributors from China, who have played diverse leadership and community roles in the upstream Kubernetes project.

So, without further ado, let’s get straight to the article.

Andy Zhang

Andy Zhang currently works for Microsoft China at the Shanghai site. His main focus is on Kubernetes storage drivers. Andy started contributing to Kubernetes about 5 years ago.

He states that as he is working in Azure Kubernetes Service team and spends most of his time contributing to the Kubernetes community project. Now he is the main contributor of quite a lot Kubernetes subprojects such as Kubernetes cloud provider code.

His open source contributions are mainly self-motivated. In the last two years he has mentored a few students contributing to Kubernetes through the LFX Mentorship program, some of whom got jobs due to their expertise and contributions on Kubernetes projects.

Andy is an active member of the China Kubernetes community. He adds that the Kubernetes community has a good guide about how to become members, code reviewers, approvers and finally when he found out that some open source projects are in the very early stage, he actively contributed to those projects and became the project maintainer.

Shiming Zhang

Shiming Zhang is a Software Engineer working on Kubernetes for DaoCloud in Shanghai, China.

He has mostly been involved with SIG Node as a reviewer. His major contributions have mainly been bug fixes and feature improvements in an ongoing KEP, all revolving around SIG Node.

Some of his major PRs are fixing watchForLockfileContention memory leak, fixing startupProbe behaviour, adding Field status.hostIPs for Pod.

Paco Xu

Paco Xu works at DaoCloud, a Shanghai-based cloud-native firm. He works with the infra and the open source team, focusing on enterprise cloud native platforms based on Kubernetes.

He started with Kubernetes in early 2017 and his first contribution was in March 2018. He started with a bug that he found, but his solution was not that graceful, hence wasn’t accepted. He then started with some good first issues, which helped him to a great extent. In addition to this, from 2016 to 2017, he made some minor contributions to Docker.

Currently, Paco is a reviewer for kubeadm (a SIG Cluster Lifecycle product), and for SIG Node.

Paco says that you should contribute to open source projects you use. For him, an open source project is like a book to learn, getting inspired through discussions with the project maintainers.

In my opinion, the best way for me is learning how owners work on the project.

Jintao Zhang

Jintao Zhang is presently employed at API7, where he focuses on ingress and service mesh.

In 2017, he encountered an issue which led to a community discussion and his contributions to Kubernetes started. Before contributing to Kubernetes, Jintao was a long-time contributor to Docker-related open source projects.

Currently Jintao is a reviewer for the ingress-nginx project.

He suggests keeping track of job opportunities at open source companies so that you can find one that allows you to contribute full time. For new contributors Jintao says that if anyone wants to make a significant contribution to an open source project, then they should choose the project based on their interests and should generously invest time.

If you have any recommendations/suggestions for who we should interview next, please let us know in the #sig-contribex channel channel on the Kubernetes Slack. Your suggestions would be much appreciated. We’re thrilled to have additional folks assisting us in reaching out to even more wonderful individuals of the community.

We’ll see you all in the next one. Everyone, till then, have a happy contributing! 👋

Blog: Enhancing Kubernetes one KEP at a Time

Thu, 11 Aug 2022 00:00:00 +0000

Author: Ryler Hockenbury (Mastercard)

Did you know that Kubernetes v1.24 has 46 enhancements? That’s a lot of new functionality packed into a 4-month release cycle. The Kubernetes release team coordinates the logistics of the release, from remediating test flakes to publishing updated docs. It’s a ton of work, but they always deliver.

The release team comprises around 30 people across six subteams - Bug Triage, CI Signal, Enhancements, Release Notes, Communications, and Docs. Each of these subteams manages a component of the release. This post will focus on the role of the enhancements subteam and how you can get involved.

What’s the enhancements subteam?

Great question. We’ll get to that in a second but first, let’s talk about how features are managed in Kubernetes.

Each new feature requires a Kubernetes Enhancement Proposal - KEP for short. KEPs are small structured design documents that provide a way to propose and coordinate new features. The KEP author describes the motivation, design (and alternatives), risks, and tests - then community members provide feedback to build consensus.

KEPs are submitted and updated through a pull request (PR) workflow on the k/enhancements repo. Features start in alpha and move through a graduation process to beta and stable as they mature. For example, here’s a cool KEP about privileged container support on Windows Server. It was introduced as alpha in Kubernetes v1.22 and graduated to beta in v1.23.

Now getting back to the question - the enhancements subteam coordinates the lifecycle tracking of the KEPs for each release. Each KEP is required to meet a set of requirements to be cleared for inclusion in a release. The enhancements subteam verifies each requirement for each KEP and tracks the status.

At the start of a release, Kubernetes Special Interest Groups (SIGs) submit their enhancements to opt into a release. A typical release might have from 60 to 90 enhancements at the beginning. During the release, many enhancements will drop out. Some do not quite meet the KEP requirements, and others do not complete their implementation in code. About 60%-70% of the opted-in KEPs will make it into the final release.

What does the enhancements subteam do?

Another great question, keep them coming! The enhancements team is involved in two crucial milestones during each release: enhancements freeze and code freeze.

Enhancements Freeze

Enhancements freeze is the deadline for a KEP to be complete in order for the enhancement to be included in a release. It’s a quality gate to enforce alignment around maintaining and updating KEPs. The most notable requirements are a (1) production readiness review (PRR) and a (2) KEP file with a complete test plan and graduation criteria.

The enhancements subteam communicates to each KEP author through comments on the KEP issue on Github. As a first step, they’ll verify the status and check if it meets the requirements. The KEP gets marked as tracked after satisfying the requirements; otherwise, it’s considered at risk. If a KEP is still at risk when enhancement freeze is in effect, the KEP is removed from the release.

This part of the cycle is typically the busiest for the enhancements subteam because of the large number of KEPs to groom, and each KEP might need to be visited multiple times to verify whether it meets requirements.

Code Freeze

Code freeze is the implementation deadline for all enhancements. The code must be implemented, reviewed, and merged by this point if a code change or update is needed for the enhancement. The latter third of the release is focused on stabilizing the codebase - fixing flaky tests, resolving various regressions, and preparing docs - and all the code needs to be in place before those steps can happen.

The enhancements subteam verifies that all PRs for an enhancement are merged into the Kubernetes codebase (k/k). During this period, the subteam reaches out to KEP authors to understand what PRs are part of the KEP, verifies that those PRs get merged, and then updates the status of the KEP. The enhancement is removed from the release if the code isn’t all merged before the code freeze deadline.

How can I get involved with the release team?

I’m glad you asked. The most direct way is to apply to be a release team shadow. The shadow role is a hands-on apprenticeship intended to prepare individuals for leadership positions on the release team. Many shadow roles are non-technical and do not require prior contributions to the Kubernetes codebase.

With 3 Kubernetes releases every year and roughly 25 shadows per release, the release team is always in need of individuals wanting to contribute. Before each release cycle, the release team opens the application for the shadow program. When the application goes live, it’s posted in the Kubernetes Dev Mailing List. You can subscribe to notifications from that list (or check it regularly!) to watch when the application opens. The announcement will typically go out in mid-April, mid-July, and mid-December - or roughly a month before the start of each release.

How can I find out more?

Check out the role handbooks if you’re curious about the specifics of all the Kubernetes release subteams. The handbooks capture the logistics of each subteam, including a week-by-week breakdown of the subteam activities. It’s an excellent reference for getting to know each team better.

You can also check out the release-related Kubernetes slack channels - particularly #release, #sig-release, and #sig-arch. These channels have discussions and updates surrounding many aspects of the release.

Blog: Spotlight on SIG Docs

Tue, 02 Aug 2022 00:00:00 +0000

Author: Purneswar Prasad

Introduction

The official documentation is the go-to source for any open source project. For Kubernetes, it’s an ever-evolving Special Interest Group (SIG) with people constantly putting in their efforts to make details about the project easier to consume for new contributors and users. SIG Docs publishes the official documentation on kubernetes.io which includes, but is not limited to, documentation of the core APIs, core architectural details, and CLI tools shipped with the Kubernetes release.

To learn more about the work of SIG Docs and its future ahead in shaping the community, I have summarised my conversation with the co-chairs, Divya Mohan (DM), Rey Lejano (RL) and Natali Vlatko (NV), who ran through the SIG’s goals and how fellow contributors can help.

A summary of the conversation

Could you tell us a little bit about what SIG Docs does?

SIG Docs is the special interest group for documentation for the Kubernetes project on kubernetes.io, generating reference guides for the Kubernetes API, kubeadm and kubectl as well as maintaining the official website’s infrastructure and analytics. The remit of their work also extends to docs releases, translation of docs, improvement and adding new features to existing documentation, pushing and reviewing content for the official Kubernetes blog and engaging with the Release Team for each cycle to get docs and blogs reviewed.

There are 2 subprojects under Docs: blogs and localization. How has the community benefited from it and are there some interesting contributions by those teams you want to highlight?

Blogs: This subproject highlights new or graduated Kubernetes enhancements, community reports, SIG updates or any relevant news to the Kubernetes community such as thought leadership, tutorials and project updates, such as the Dockershim removal and removal of PodSecurityPolicy, which is upcoming in the 1.25 release. Tim Bannister, one of the SIG Docs tech leads, does awesome work and is a major force when pushing contributions through to the docs and blogs.

Localization: With this subproject, the Kubernetes community has been able to achieve greater inclusivity and diversity among both users and contributors. This has also helped the project gain more contributors, especially students, since a couple of years ago. One of the major highlights and up-and-coming localizations are Hindi and Bengali. The efforts for Hindi localization are currently being spearheaded by students in India.

In addition to that, there are two other subprojects: reference-docs and the website, which is built with Hugo and is an important ownership area.

Recently there has been a lot of buzz around the Kubernetes ecosystem as well as the industry regarding the removal of dockershim in the latest 1.24 release. How has SIG Docs helped the project to ensure a smooth change among the end-users?

Documenting the removal of Dockershim was a mammoth task, requiring the revamping of existing documentation and communicating to the various stakeholders regarding the deprecation efforts. It needed a community effort, so ahead of the 1.24 release, SIG Docs partnered with Docs and Comms verticals, the Release Lead from the Release Team, and also the CNCF to help put the word out. Weekly meetings and a GitHub project board were set up to track progress, review issues and approve PRs and keep the Kubernetes website updated. This has also helped new contributors know about the depreciation, so that if any good-first-issue pops up, they could chip in. A dedicated Slack channel was used to communicate meeting updates, invite feedback or to solicit help on outstanding issues and PRs. The weekly meeting also continued for a month after the 1.24 release to review related issues and fix them. A huge shoutout to Celeste Horgan, who kept the ball rolling on this conversation throughout the deprecation process.

Why should new and existing contributors consider joining this SIG?

Kubernetes is a vast project and can be intimidating at first for a lot of folks to find a place to start. Any open source project is defined by its quality of documentation and SIG Docs aims to be a welcoming, helpful place for new contributors to get onboard. One gets the perks of working with the project docs as well as learning by reading it. They can also bring their own, new perspective to create and improve the documentation. In the long run if they stick to SIG Docs, they can rise up the ladder to be maintainers. This will help make a big project like Kubernetes easier to parse and navigate.

How do you help new contributors get started? Are there any prerequisites to join?

There are no such prerequisites to get started with contributing to Docs. But there is certainly a fantastic Contribution to Docs guide which is always kept as updated and relevant as possible and new contributors are urged to read it and keep it handy. Also, there are a lot of useful pins and bookmarks in the community Slack channel #sig-docs. GitHub issues with the good-first-issue labels in the kubernetes/website repo is a great place to create your first PR. Now, SIG Docs has a monthly New Contributor Meet and Greet on the first Tuesday of the month with the first occupant of the New Contributor Ambassador role, Arsh Sharma. This has helped in making a more accessible point of contact within the SIG for new contributors.

DM & RL : The formalization of the localization subproject in the last few months has been a big win for SIG Docs, given all the great work put in by contributors from different countries. Earlier the localization efforts didn’t have any streamlined process and focus was given to provide a structure by drafting a KEP over the past couple of months for localization to be formalized as a subproject, which is planned to be pushed through by the end of third quarter.

DM : Another area where there has been a lot of success is the New Contributor Ambassador role, which has helped in making a more accessible point of contact for the onboarding of new contributors into the project.

NV : For each release cycle, SIG Docs have to review release docs and feature blogs highlighting release updates within a short window. This is always a big effort for the docs and blogs reviewers.

Is there something exciting coming up for the future of SIG Docs that you want the community to know?

SIG Docs is now looking forward to establishing a roadmap, having a steady pipeline of folks being able to push improvements to the documentation and streamlining community involvement in triaging issues and reviewing PRs being filed. To build one such contributor and reviewership base, a mentorship program is being set up to help current contributors become reviewers. This definitely is a space to watch out for more!

Wrap Up

SIG Docs hosted a deep dive talk during on KubeCon + CloudNativeCon North America 2021, covering their awesome SIG. They are very welcoming and have been the starting ground into Kubernetes for a lot of new folks who want to contribute to the project. Join the SIG’s meetings to find out about the most recent research results, their plans for the forthcoming year, and how to get involved in the upstream Docs team as a contributor!

Blog: Contextual Logging in Kubernetes 1.24

Wed, 25 May 2022 00:00:00 +0000

Authors: Patrick Ohly (Intel)

The Structured Logging Working Group has added new capabilities to the logging infrastructure in Kubernetes 1.24. This blog post explains how developers can take advantage of those to make log output more useful and how they can get involved with improving Kubernetes.

Structured logging

The goal of structured logging is to replace C-style formatting and the resulting opaque log strings with log entries that have a well-defined syntax for storing message and parameters separately, for example as a JSON struct.

When using the traditional klog text output format for structured log calls, strings were originally printed with \n escape sequences, except when embedded inside a struct. For structs, log entries could still span multiple lines, with no clean way to split the log stream into individual entries:

I1112 14:06:35.783529 328441 structured_logging.go:51] "using InfoS" longData={Name:long Data:Multiple
lines
with quite a bit
of text. internal:0}
I1112 14:06:35.783549 328441 structured_logging.go:52] "using InfoS with\nthe message across multiple lines" int=1 stringData="long: Multiple\nlines\nwith quite a bit\nof text." str="another value"

Now, the < and > markers along with indentation are used to ensure that splitting at a klog header at the start of a line is reliable and the resulting output is human-readable:

I1126 10:31:50.378204 121736 structured_logging.go:59] "using InfoS" longData=<
{Name:long Data:Multiple
lines
with quite a bit
of text. internal:0}
>
I1126 10:31:50.378228 121736 structured_logging.go:60] "using InfoS with\nthe message across multiple lines" int=1 stringData=<
long: Multiple
lines
with quite a bit
of text.
> str="another value"

Note that the log message itself is printed with quoting. It is meant to be a fixed string that identifies a log entry, so newlines should be avoided there.

Before Kubernetes 1.24, some log calls in kube-scheduler still used klog.Info for multi-line strings to avoid the unreadable output. Now all log calls have been updated to support structured logging.

Contextual logging

Contextual logging is based on the go-logr API. The key idea is that libraries are passed a logger instance by their caller and use that for logging instead of accessing a global logger. The binary decides about the logging implementation, not the libraries. The go-logr API is designed around structured logging and supports attaching additional information to a logger.

This enables additional use cases:

The caller can attach additional information to a logger:
- WithName adds a prefix
- WithValues adds key/value pairs
When passing this extended logger into a function and a function uses it instead of the global logger, the additional information is then included in all log entries, without having to modify the code that generates the log entries. This is useful in highly parallel applications where it can become hard to identify all log entries for a certain operation because the output from different operations gets interleaved.
When running unit tests, log output can be associated with the current test. Then when a test fails, only the log output of the failed test gets shown by go test. That output can also be more verbose by default because it will not get shown for successful tests. Tests can be run in parallel without interleaving their output.

One of the design decisions for contextual logging was to allow attaching a logger as value to a context.Context. Since the logger encapsulates all aspects of the intended logging for the call, it is part of the context and not just using it. A practical advantage is that many APIs already have a ctx parameter or adding one has additional advantages, like being able to get rid of context.TODO() calls inside the functions.

Another decision was to not break compatibility with klog v2:

Libraries that use the traditional klog logging calls in a binary that has set up contextual logging will work and log through the logging backend chosen by the binary. However, such log output will not include the additional information and will not work well in unit tests, so libraries should be modified to support contextual logging. The migration guide for structured logging has been extended to also cover contextual logging.
When a library supports contextual logging and retrieves a logger from its context, it will still work in a binary that does not initialize contextual logging because it will get a logger that logs through klog.

In Kubernetes 1.24, contextual logging is a new alpha feature with ContextualLogging as feature gate. When disabled (the default), the new klog API calls for contextual logging (see below) become no-ops to avoid performance or functional regressions.

No Kubernetes component has been converted yet. An example program in the Kubernetes repository demonstrates how to enable contextual logging in a binary and how the output depends on the binary’s parameters:

$ cd $GOPATH/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/example/cmd/
$ go run . --help
...
 --feature-gates mapStringBool A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
 AllAlpha=true|false (ALPHA - default=false)
 AllBeta=true|false (BETA - default=false)
 ContextualLogging=true|false (ALPHA - default=false)
$ go run . --feature-gates ContextualLogging=true
...
I0404 18:00:02.916429 451895 logger.go:94] "example/myname: runtime" foo="bar" duration="1m0s"
I0404 18:00:02.916447 451895 logger.go:95] "example: another runtime" foo="bar" duration="1m0s"

The example prefix and foo="bar" were added by the caller of the function which logs the runtime message and duration="1m0s" value.

The sample code for klog includes an example for a unit test with per-test output.

klog enhancements

Contextual logging API

The following calls manage the lookup of a logger:

FromContext: from a context parameter, with fallback to the global logger
Background: the global fallback, with no intention to support contextual logging
TODO: the global fallback, but only as a temporary solution until the function gets extended to accept a logger through its parameters
SetLoggerWithOptions: changes the fallback logger; when called with ContextualLogger(true), the logger is ready to be called directly, in which case logging will be done without going through klog

To support the feature gate mechanism in Kubernetes, klog has wrapper calls for the corresponding go-logr calls and a global boolean controlling their behavior:

Usage of those functions in Kubernetes code is enforced with a linter check. The klog default for contextual logging is to enable the functionality because it is considered stable in klog. It is only in Kubernetes binaries where that default gets overridden and (in some binaries) controlled via the --feature-gate parameter.

ktesting logger

The new ktesting package implements logging through testing.T using klog’s text output format. It has a single API call for instrumenting a test case and support for command line flags.

klogr

klog/klogr continues to be supported and it’s default behavior is unchanged: it formats structured log entries using its own, custom format and prints the result via klog.

However, this usage is discouraged because that format is neither machine-readable (in contrast to real JSON output as produced by zapr, the go-logr implementation used by Kubernetes) nor human-friendly (in contrast to the klog text format).

Instead, a klogr instance should be created with WithFormat(FormatKlog) which chooses the klog text format. A simpler construction method with the same result is the new klog.NewKlogr. That is the logger that klog returns as fallback when nothing else is configured.

Reusable output test

A lot of go-logr implementations have very similar unit tests where they check the result of certain log calls. If a developer didn’t know about certain caveats like for example a String function that panics when called, then it is likely that both the handling of such caveats and the unit test are missing.

klog.test is a reusable set of test cases that can be applied to a go-logr implementation.

Output flushing

klog used to start a goroutine unconditionally during init which flushed buffered data at a hard-coded interval. Now that goroutine is only started on demand (i.e. when writing to files with buffering) and can be controlled with StopFlushDaemon and StartFlushDaemon.

When a go-logr implementation buffers data, flushing that data can be integrated into klog.Flush by registering the logger with the FlushLogger option.

Various other changes

For a description of all other enhancements see in the release notes.

logcheck

Originally designed as a linter for structured log calls, the logcheck tool has been enhanced to support also contextual logging and traditional klog log calls. These enhanced checks already found bugs in Kubernetes, like calling klog.Info instead of klog.Infof with a format string and parameters.

It can be included as a plugin in a golangci-lint invocation, which is how Kubernetes uses it now, or get invoked stand-alone.

We are in the process of moving the tool into a new repository because it isn’t really related to klog and its releases should be tracked and tagged properly.

Next steps

The Structured Logging WG is always looking for new contributors. The migration away from C-style logging is now going to target structured, contextual logging in one step to reduce the overall code churn and number of PRs. Changing log calls is good first contribution to Kubernetes and an opportunity to get to know code in various different areas.

Blog: February 2022 Community Meeting Highlights

Thu, 05 May 2022 00:00:00 +0000

Author: Nigel Brown (VMware)

We just had our first contributor community meeting this year, and it was awesome to be back with you in that format. These meetings will be happening on Zoom once per month, on the third Thursday of the month - that should be available in your calendar if you’re subscribed to the k-dev mailing list. Community meetings are an opportunity for you to meet synchronously with other members of the Kubernetes community to talk about issues of general appeal.

This meeting kicked off with an update on the 1.24 release with Xander Grzywinski, who is one of the shadows for the release team leads. This release is scheduled for April 19, 2022 with a code freeze scheduled for March 30th. At the time of the meeting there were 66 individual enhancements included, as well as bug fixes. You can join the conversation on Slack in #sig-release.

Update: Kubernetes 1.24 was delayed and released on May 3, 2022.

From there, the discussion moved to the dockershim removal and the docs updates we need to make around that, with the discussion led by Kat Cosgrove. The main takeaway was that if you have a platform, talk to folks about this change. To most interacting with Kubernetes, it is probably not as impactful as it sounds. We have a helpful FAQ if you need. You can even try out an alpha release from the 1.24 release page.

We moved on to a spirited discussion of a Kubernetes Enhancement Proposal (KEP) about raising the bar for reliability brought by Wojciech Tyczynski. It was emphasized that effort on this proposal should be a collaborative effort with SIG Testing who are managing dashboards on test flakiness among other metrics. You can find the proposed KEP on GitHub.

Finally, Paris mentioned the k-dev migration of the developer mailing list. If you manage Google Docs assets, you may need to share them with the new developer list. New community members may not be able to join from assets shared with the old lists.

You can find the full meeting notes posted online (thanks Josh Berkus!) as well as the recording on YouTube. If you have topics you would like to discuss or you’re interested in being the host of a future community meeting, please reach out to Laura Santamaria (@nimbinatus) on Slack.

Blog: K8s CI Bot Helper Job: automating "make update"

Tue, 15 Mar 2022 00:00:00 +0000

Authors: Subhasmita Swain, Davanum Srinivas

If you are contributing to the Kubernetes project and are developing on a Windows PC, it is conceivable that you will encounter certain issues that will cause your pull request to get held up by test failures. This article describes a workaround for a similar issue I encountered when attempting to have my modifications approved and merged into the master branch.

Why is this needed?

While contributing to kubernetes/kubernetes for some minor documentation changes, the pushed changes needed to be updated with other verified contents of the entire documentation. So, in order for the change to take effect, a single command must be performed to ensure that all tests on the CI pipeline pass. The single command make update runs all presubmission verification tests. For some reason on the “Windows Subsystem for Linux” environment the tests, specifically the update-openapi-spec.sh script, failed (in my case, take a look at the conversation here), eventually failing the pull-kubernetes-verify tests.

You might encounter the following on your PR

The tests failing the particular issue:

Consecutively,

Additionally one can check the failed test via the link provided under details in the above image.

Potential Workaround

Run the failing .sh scripts individually known from the CI job output, to generate the expected files to fix up the failures. The .sh scripts can be found residing under the hack/ directory at the root of the kubernetes/kubernetes code base.

kubernetes/kubernetes → hack/*.sh

In this particular case, these files were to be run:

After you run these, you might see either of the below issues:

Using Codespaces:

Using Visual Studio Code:

Possible solutions to counter the above errors

Remove the Makefile and Makefile.generated_files files:

 rm Makefile Makefile.generated_files

Create the symlinks:

 ln -s build/root/Makefile Makefile

 ln -s build/root/Makefile.generated_files Makefile.generated_files

PS: takes adequate time to generate the scripts on windows.

The current situation

Many contributors do not have access to powerful environments in which to run make update or make verify. They can utilise vscode/wsl/codespaces and other tools to recommend modifications, but they might get tripped up by make verify scripts because in many cases, we end up with files that needs to be re-generated. It’s a tall order for them to scan the build log from make verify to determine which specific scripts in hack/ directory they need to run.

Solutions

According to @dims, a long term solution for this problem would be to add a new Prow bot command that generates an additional commit on their PR.

In the meantime, the short term workaround is to add a CI job that folks can trigger when they need it, that runs make update and provides a zip archive that they can be told to download. The archive includes all the files that changed as a result of running make update.

Implementing the Short Term Workaround

The problem was that for some of the verify scripts to work well, the scripts needed both Linux and plenty of local resources (CPU, memory). Once we realized this issue, we decided to add a new CI job. This CI job is named pull-kubernetes-update.

You can trigger this CI job by commenting /test pull-kubernetes-update as a Prow bot command in any Kubernetes PR. This CI job runs make update and then generates a zip archive named updated-files.zip in the artifacts directory for that job.

You can then download the archive which has the changes made when make update was run; once the download is finished, you can update your PR with the newly updated code.

Using the short-term solution

Here’s what you can do as a contributor to get automated help updating your pull request with generated files.

Make sure to rebase your working branch against the latest base branch (usually this is master), so that your PR includes the most recent upstream commits. Remember to push or force-push your changes.

On your PR, write a new comment that contains only the text:

 /test pull-kubernetes-update

You will see the automated check lists and details.

Once the checks are complete, click the Details link on pull-kubernetes-update job to go to the artifacts directory and download the updated-files.zip file.

Now, update the PR by adding the extracted files you downloaded.

Conclusion

For the time being, several fantastic people are working on a bot command that will produce an additional commit with the generated files for the failed tests. In the long term, it will make things simpler. If anything becomes confusing at any point, we urge any and all inquiries to be directed on slack, regardless of experience level or complexity! We hope this shortens your debugging time and alleviates some of your concerns!

Blog: Meet Our Contributors - APAC (Aus-NZ region)

Mon, 14 Mar 2022 00:00:00 +0000

Authors & Interviewers: Anubhav Vardhan, Atharva Shinde, Avinesh Tripathi, Brad McCoy, Debabrata Panigrahi, Jayesh Srivastava, Kunal Verma, Pranshu Srivastava, Priyanka Saggu, Purneswar Prasad, Vedant Kakde

Good day, everyone 👋

Welcome back to the second episode of the “Meet Our Contributors” blog post series for APAC.

This post will feature four outstanding contributors from the Australia and New Zealand regions, who have played diverse leadership and community roles in the Upstream Kubernetes project.

So, without further ado, let’s get straight to the article.

Caleb Woodbine

Caleb Woodbine is currently a member of the ii.nz organisation.

He began contributing to the Kubernetes project in 2018 as a member of the Kubernetes Conformance working group. His experience was positive, and he benefited from early guidance from Hippie Hacker, a fellow contributor from New Zealand.

He has made major contributions to Kubernetes project since then through SIG k8s-infra and k8s-conformance working group.

Caleb is also a co-organizer of the CloudNative NZ community events, which aim to expand the reach of Kubernetes project throughout New Zealand in order to encourage technical education and improved employment opportunities.

There need to be more outreach in APAC and the educators and universities must pick up Kubernetes, as they are very slow and about 8+ years out of date. NZ tends to rather pay overseas than educate locals on the latest cloud tech Locally.

Dylan Graham

Dylan Graham is a cloud engineer from Adeliade, Australia. He has been contributing to the upstream Kubernetes project since 2018.

He stated that being a part of such a large-scale project was initially overwhelming, but that the community’s friendliness and openness assisted him in getting through it.

He began by contributing to the project documentation and is now mostly focused on the community support for the APAC region.

He believes that consistent attendance at community/project meetings, taking on project tasks, and seeking community guidance as needed can help new aspiring developers become effective contributors.

The feeling of being a part of a large community is really special. I’ve met some amazing people, even some before the pandemic in real life :)

Hippie Hacker

Hippie has worked for the CNCF.io as a Strategic Initiatives contractor from New Zealand for almost 5+ years. He is an active contributor to k8s-infra, API conformance testing, Cloud provider conformance submissions, and apisnoop.cncf.io domains of the upstream Kubernetes & CNCF projects.

He recounts their early involvement with the Kubernetes project, which began roughly 5 years ago when their firm, ii.nz, demonstrated network booting from a Raspberry Pi using PXE and running Gitlab in-cluster to install Kubernetes on servers.

He describes their own contributing experience as someone who, at first, tried to do all of the hard lifting on their own, but eventually saw the benefit of group contributions which reduced burnout and task division which allowed folks to keep moving forward on their own momentum.

He recommends that new contributors use pair programming.

The cross pollination of approaches and two pairs of eyes on the same work can often yield a much more amplified effect than a PR comment / approval alone can afford.

Nick Young

Nick Young works at VMware as a technical lead for Contour, a CNCF ingress controller. He was active with the upstream Kubernetes project from the beginning, and eventually became the chair of the LTS working group, where he advocated user concerns. He is currently the SIG Network Gateway API subproject’s maintainer.

His contribution path was notable in that he began working on major areas of the Kubernetes project early on, skewing his trajectory.

He asserts the best thing a new contributor can do is to “start contributing”. Naturally, if it is relevant to their employment, that is excellent; however, investing non-work time in contributing can pay off in the long run in terms of work. He believes that new contributors, particularly those who are currently Kubernetes users, should be encouraged to participate in higher-level project discussions.

Just being active and contributing will get you a long way. Once you’ve been active for a while, you’ll find that you’re able to answer questions, which will mean you’re asked questions, and before you know it you are an expert.

If you have any recommendations/suggestions for who we should interview next, please let us know in #sig-contribex. Your suggestions would be much appreciated. We’re thrilled to have additional folks assisting us in reaching out to even more wonderful individuals of the community.

We’ll see you all in the next one. Everyone, till then, have a happy contributing! 👋

Kubernetes Contributors – Contributor Blog

Blog: Spotlight on SIG Architecture: API Governance

Introduction

Goals and scope of API Governance

The impact of Custom Resource Definitions

API Governance in context

Getting involved

Blog: Announcing the Checkpoint/Restore Working Group

Motivation and use cases

Related events

Connect with us

Blog: Ingress NGINX Retirement: What You Need to Know

About Ingress NGINX

History and Challenges

Current State and Next Steps

Blog: Announcing the 2025 Steering Committee Election Results

Results

Big thanks!

Get involved with the Steering Committee

Blog: Spotlight on Policy Working Group

Introduction

About Working Groups

Policy WG

Challenges

Blog: Spotlight on the Kubernetes Steering Committee

Introductions

About the Steering Committee

The road to the Steering Committee

Conclusion

Blog: Post-Quantum Cryptography in Kubernetes

What is Post-Quantum Cryptography

Key exchange vs. digital signatures: different needs, different timelines

State of PQC key exchange mechanisms (KEMs) today

Post-quantum KEMs in Kubernetes: an unexpected arrival

The Go version mismatch pitfall

Limitations: packet size

State of Post-Quantum Signatures

Conclusion

Blog: Changes to Kubernetes Slack

Blog: Spotlight on SIG Apps

Introductions

About SIG Apps

Best practices and challenges

Contributing to SIG Apps

Looking ahead

Blog: Spotlight on SIG etcd

Introducing SIG etcd

Becoming a Kubernetes Special Interest Group (SIG)

The road ahead

Getting involved

Blog: Spotlight on SIG Architecture: Enhancements

The Enhancements subproject

The KEP and its impact

Current areas of focus

Getting involved

Blog: Spotlight on Kubernetes Upstream Training in Japan

What is Kubernetes upstream training in Japan?

Interview with participants

Keita Mochizuki (NTT DATA Group Corporation)

Yoshiki Fujikane (CyberAgent, Inc.)

Future of Kubernetes upstream training

Blog: Announcing the 2024 Steering Committee Election Results

Results

Big thanks!

Get involved with the Steering Committee

Blog: Spotlight on CNCF Deaf and Hard-of-hearing Working Group (DHHWG)

Introduction

Motivation and early milestones

Overcoming challenges and addressing misconceptions

Impact and the role of allies

Expanding DEI efforts and future vision

Call to action

Wrapping up

Blog: Spotlight on SIG Scheduling

Introductions

About SIG Scheduling

Contributing to SIG Scheduling

Future Directions

Closing Remarks

Blog: Spotlight on SIG API Machinery