+1 for one of the coolest and reliable kuberneres distribution

I don't usually +1 any PRs via comment, but definitely +1 here. Happy to see k3s be a part of CNCF in this way (would bring some light to 2020).

This sets a very weird precedent IMO, because k3s is essentially a distribution, where several of the patches could be pushed into upstream k/k.

Where would this end, and what is the policy?

@kubernetes/steering-committee
@cncf/toc

+1 i have nothing against k3s but it is a distro, or a feature, of k8s.

To expand - imho it is ok for the foundation to support distribution projects, provided that

1. There are many, each fit for purpose
2. Users are not confused
3. Vendor specific commitments are not included
4. Each distribution is kept up to date

I worry that 1-2 are in conflict, as are 3-4.

This sets a very weird precedent IMO, because k3s is essentially a distribution, where several of the patches could be pushed into upstream k/k.

Where would this end, and what is the policy?

@kubernetes/steering-committee
@cncf/toc

Not that I have any important status in the project or the CNCF... but my two cents would be that k3s, while it can seem to be just a distribution at first, it is more than that (at least in its current form). It's a project with entirely different goals than k8s at the moment, and it's hyperfocus ends up creating mini projects that may or may not end up being contributed back to upstream (depending on the overarching goals and decisions for upstream, which again doesn't necessarily match the goals of this project.) Wether that's for or against the sandbox status goals... I can't say, but I think labelling it as purely a distribution isn't the right call.

@timothysc Regarding patches, there's no change we do to k/k code that we wouldn't want to push upstream. Overtime we have drastically reduced the amount of changes we do to core k8s. We are slowing working through the right approaches to getting everything upstream. Most of this work happens indirectly though. Any large change we've done to k8s has usually been just doing what upstream was going to do already, so external cloud providers, move to external storage drivers, and similar efforts. Most of the work we do to align with k8s upstream is not very visible but I will say it has always been a clear goal to get to the point of not modifying k8s at all. Getting k3s adopted into CNCF will only better help us achieve that goal.

@monadic The term distribution is not a really well defined in the k8s world. We call k3s a distribution because that's a term people kind of get but I can't really say it's the best term. k3s represents a significant amount of work to enable Kubernetes to be used in a simple and lightweight fashion. k3s enables a use case not easily achieve with pure k8s and that has clearly been reflected by it's adoption. k3s is really not too far off from KubeEdge which is already a CNCF project.

I realise that the whole PR contains a lot of info, but adding an intro / summary as a PR description would be helpful if you're open to adding it?

There is a lot of history here and I think it makes sense to have more folks from the k8s community share their impressions.

Some thoughts:

• Right now k3s is essentially a "fork" of kubernetes. There are bunch of changes that would be great to see upstream but this hasn't happened.
• k3s has a lot in common with what we started doing with hyperkube. It would have been great to see this ideas played out there as part of that effort instead of being done from the outside.
• The TOC has, in the past, decided not to include projects that were mainly "integration" projects. This is clearly a big part of what k3s is. Including projects like flannel and traefik brings in projects outside of the CNCF in a way that should be looked at closely.
• There doesn't seem to be any documentation around how to contribute. There is also no documented governance process. The CoC was added ~5 hours ago. This doesn't seem like a project that has worked toward vendor-neutral governance at all.

But -- overall -- I echo the concerns of Tim and Alexis. This is either a fork of kubernetes or it is an integration/distribution project.

We are slowing working through the right approaches to getting everything upstream.

I'd love to see this happen before joining the CNCF. If I were on the TOC still this is something I'd be looking at very carefully.

As someone coming from IoT, I find inclusion as a separate project quite appealing. Viewing k3s as a stripped down k8s distribution would be short-sighted. But making it separate would encourage contributions focused on making this work well within the typically resource-constrained and hostile (eg unreliable power, internet) IoT/edge environments.

(I can't speak to broader CNCF issues, and if there are specific tasks that k3s needs to do before joining the CNCF then I'd encourage the authors to perform those tasks.)

@jbeda Thanks for your feedback, it's much appreciated and you bring up some great points probably shared by many working on core Kubernetes.

Right now k3s is essentially a "fork" of kubernetes. There are bunch of changes that would be great to see upstream but this hasn't happened.

The idea that k3s is a fork does come up and I think it's important to clarify this point. k3s is not a fork and I'm open to hear any data points to indicate it is, because it's not the intention. Today k3s modifies the core Kubernetes and caries a patch set we keep in sync with all community maintained version of Kubernetes. The patch set is about 300 lines and deals primarily with how we package k8s, not functionality. This is a huge improvement as the first version of k3s had a diff of literally millions of lines. So I would in no way call what we do a fork as it's fairly straight forward for any distribution of X to carry some patches that can't or haven't yet been upstreamed.

Regarding the idea of not contributing to Kubernetes. Again, the patch set we maintain mostly has to do with how we integrate and package k8s, not features. So there really isn't a single feature to upstream and I'd love to hear if there is something in k3s that one would like to see upstream as I'd gladly do that. I'll be honest in that the majority of the patches we carry do not have enough technical merit to be upstreamed, they can be viewed more as technical debt where we did things wrong and there are better known approaches. This is why this patch set is dwindling overtime without having to upstream much of anything beyond bug fixes.

One of the biggest changes we did to Kubernetes was that k3s supports SQL backends. The first version of this code modified k8s storage backend. Based on feedback we got from upstream the recommended approach was to create a etcd shim. As a result we rewrote what was previously called kvsql into a new component kine. The great thing about this project is that it works with any k8s distribution, not just k3s and has seen a lot of adoption already. Kine is a project I think should be donated to kubernetes, but we just haven't had enough time to do that yet.

I 100% understand the impression that k3s hasn't worked enough with k8s upstream. The biggest reason is that the majority of what we do with k3s has nothing to do with upstream k8s. This also justifies why we it exists as a separate project. k3s builds upon k8s.

k3s has a lot in common with what we started doing with hyperkube. It would have been great to see this ideas played out there as part of that effort instead of being done from the outside.

While the goals of k3s for sure overlap with many projects in the ecosystem it's not really complete to say its basically the same as hyperkube. It doesn't take much to see that k3s is far beyond the scope of hyperkube. At this point hyperkube has basically been dropped from k/k so there really isn't much point belaboring this point.

There doesn't seem to be any documentation around how to contribute. There is also no documented governance process. The CoC was added ~5 hours ago. This doesn't seem like a project that has worked toward vendor-neutral governance at all.

k3s has been way more successful of a project than we could have possibility imagined or expected. You are absolutely right in that we probably up to this point have not been as organized of a project as one would hope. But that is basically the entire reason we want to join CNCF. We have the users and adoption, we need help better running this project.

Brief note regarding vendor-neutral governance: this has typically not been a requirement for projects to be accepted into the CNCF sandbox. It has certainly been a requirement for graduation but it's thus far been expected that projects can use their "sandbox time" to attract contributors from outside the initial org.

@lucperkins Agree with you -- but we did look to see if a project was on the trajectory and trying to bring on other contributors and create a community driven project. This is clearly subjective and something that the TOC will have to consider.

Right now k3s is essentially a "fork" of kubernetes. There are bunch of changes that would be great to see upstream but this hasn't happened.

The idea that k3s is a fork does come up and I think it's important to clarify this point. k3s is not a fork...

Even if it is a fork, I see nothing in the charter that would prevent a fork of k8s from being accepted into the CNCF. I see several bylaws that would suggest just the opposite, that inclusion is warranted - maybe even especially if it was a fork.

AFAIK, we used to have an agreement that CNCF will not accept a distribution :)

Agree with Timothy & Joe, k3s is more like a distribution of k8s; anyway, how about raising to k8s steering committee/sig-architecture/sig-node avoiding any confusion to TOC?

@raravena80 , this PR should catalog to sig-runtime, WDYT?

k3s is a distro of Kubernetes in the same way CoreOS is a distro of Linux - i.e. it adds value for use cases that will either conflict with upstream, dilute existing use cases or upstream maintainers will just not be able to prioritize adequately. If Kubernetes is the platform of platforms that we imagined it would be then projects like k3s should be welcome in the CNCF community. There is real end-user value in k3s judging by its popularity and production deployments.

@k82cn yes, I believe this is within the scope of SIG-Runtime.

k3s is really not too far off from KubeEdge which is already a CNCF project.

A distinction that I would add is that KubeEdge is built on top of Kubernetes and uses standard or any other flavor of Kubernetes whereas k3s is a specific flavor of Kubernetes primarily targeted for the edge (i.e single binary instead of separate components/binaries) with some additional components (containerd, Flannel, CoreDNS, CNI, Traefik, etc)

k3s is a opinionated distro optimized for a set of user stories, and their documentation states that outright.

My point earlier is on policy and governance. There are a lot of opinionated k8s distros; gke, eks, aks, openshift, k3s, to name only a few. The question is really, "What is the CNCF policy on housing distros?" What effects would this have on the ecosystem? It's a slippery slope... and requires a well thought out policy to spell out the rules and clauses.

For example, we saw in the early days of kubernetes that opinionated deployment tools raced to become part of the k8s org and it was misused as leverage in an effort to try and land grab on market-share. I'm not saying that is the intent is here, but it could be easily misconstrued, which is why policy is so important.

cc @brendandburns @bgrant0607 @smarterclayton

@timothysc These are all good points and I look forward to hearing the TOC's approach to distributions. My comment previously that a distribution is a not a well defined term in the k8s space is highlighted by your examples of GKE, EKS, AKS, OpenShift and k3s. GKE, EKS, AKS are managed offerings. OpenShift is a fully integrated end to end container platform and product. While all these can be loosely labeled as a distributions they are clearly in different categories. It will be interesting how the TOC chooses to classify the space.

At the end of the day k3s' rate of adoption identifies there is something unique about it and therefore will probably be difficult to easily characterize it. I have a hard time seeing k3s as a CNCF project would somehow be a net negative for CNCF or the Kubernetes ecosystem but I do understand this is something new and therefore deserves consideration.

k3s self-describes as a distribution, and I don’t think distributions in general would make good CNCF projects, because we don’t want to compete against our own community. But is k3s just a distribution? Is it a really distribution in the same sense as commercial or vendor-specific offerings like GKE or OpenShift or Rancher?

It also solves the genuine end-user problem of running cloud-native applications in constrained environments like edge. So I don’t think it would be serving our community well to dismiss it simply because of the word “distribution” without further consideration.

I’d very much like to hear the view of the Kubernetes steering committee about whether and how something like k3s could / should co-exist with Kubernetes, and how they would envisage the CNCF supporting the adoption of K8s in Edge / IoT environments.

(This comment is my own, it’s not based on a TOC discussion, but I wanted to reply publicly since I’ve been messaged about it in private - not, I might add, by anyone directly connected with the project.)

I really appreciate everyone's input on this thread. As to @cjellick question, I just want to clarify that according to https://github.com/cncf/toc/blob/master/process/project_proposals.adoc it seems we have hit "Step 2 - TOC Triage." From my understand that's mostly a sit and wait from the PR creators perspective. We are just asking if there is anything we need to do move the process forward. Any comment from TOC members as to what we should be expecting or are required to do would be helpful. Thanks.

FWIW -- I know that this is a spoof account but this tweet shows that there is legitimate confusion over the name k3s.

https://twitter.com/morecobol/status/1280877846567751680

In what way @jbeda? What is the confusion on the thread or the linked blog post by Civo who have built a commercial service upon it?

The fact that someone has to ask the question implies, to me, that they are confused and see them as overlapping. The closeness of naming implies a connection that isn't there. See also: https://twitter.com/dustinmoris/status/1280886236723503104

The fact that someone has to ask the question implies, to me, that they are confused and see them as overlapping. The closeness of naming implies a connection that isn't there. See also: https://twitter.com/dustinmoris/status/1280886236723503104

I'm not sure how I see one random person's confusion as being connected or representative. We can always find some subjective evidence to support whatever opinion we have... There's a lot of confusion, in some circles, about what is Kubernetes to Chef, or to Ansible, the list goes on. We don't consider those to be the same, because they fundamentally... aren't. While I, personally, think it's good to highlight there is potential confusion, I just think that maybe we should make sure it's a larger sample size.

I'm sorry but no matter the many merits of K3s its name is unambiguously confusing. By all means sample the real world but other than for a handful of super in the know people, k8s and k3s are identical.

I'm sorry but no matter the many merits of K3s its name is unambiguously confusing. By all means sample the real world but other than for a handful of super in the know people, k8s and k3s are identica

Not sure we have enough evidence to make such a statement. Also, it doesn't really have anything to do with including it or not. If k3s has an ambiguous name, rename it, its not really k8s fault that other projects chose a name that makes differentiation hard.

Yes exactly k3s should have a different name and release path, or be a subproject of k8s.

Question:

1. Does the project need to be renamed prior to inclusion? That seems like a completely separate discussion and one that could be made once the project was included.
2. Does the name of the project have an impact on CNCF inclusion? -- if not, why does it matter for this PR?

In my opinion, highlighting a random person tweeting that they don't know what K3S is does not indicate that there is a naming problem. That person doesn't know what K3S is and was not willing to Google it in a follow-up tweet. it's a very small sample size, and quite frankly seems a little bit ridiculous to pull up a tweet like that to prove a point about the naming of this particular project. Nor do I feel like the naming should have anything to do with its validity for inclusion in the CNCF sandbox.

It seems like in general there is a very strong consensus to include K3S in the CNCF sandbox, and correct me if I'm wrong but wouldn't including the project in CNCF give the community greater power to be able to make some changes to the project and also to push certain changes upstream?

Hi there! I want to help, so I also tweeted: https://twitter.com/__yajo/status/1281130802848333824. Now the world is balanced again. 🤣

Now, jokes apart, forget the name. Rancher had a project and had to name it somehow. The name is already known, meaning it has a nice brand and website, and some child projects based on it with names based on it. Don't you like the name? Then you should have created it first and named it however you liked. But right now, it's just a name. Some like it, some not. If you change it, some will like it, some will not, so that will make no difference. Forget it and move on! 💪 😉

@ibuildthecloud Just for transparency. When we were going through the sandbox proposals, this was the next one before we ran out of time in our meeting. This will be discussed at our meeting next week. There is nothing more we need from you or the k3s team at this time. Thanks for your submission.

Thanks for the info @michelleN

if your levenshtein distance is exactly 1 then you can't expect normal people to easily grasp that your project is a totally separate project that does different things but also the same things and understand what that means on a practical level

but maybe cobol was just having a weird day yall

+1!

I don't think it needs to be accepted as a sandbox project.

1. What is the purpose of acceptance?
2. What is the difference between it and k8s?
3. Friends who participate in voting should not be one who tries not to offend anybody.

Finally, sandbox should not be abused.

Note: I am not speaking for rancher labs, I just want to inject my own interpretation:
@zouyee to answer: 2.:

Kubernetes is more a specification/abstract system (to me personally) than "a piece of software", k3s implements thus specification/system and optimizes it for low-spec / edge computing, allowing a kubernetes system to exist at the low end of the computing spectrum. Thus, it is not really a k8s fork, but rather a project that implements k8s in a different way than conventional slightly-modified cloud-provided kubernetes engines.

And I think I can also slightly answer 1., but I'd leave a definitive answer over to @ibuildthecloud:

I think that by accepting k3s as a CNCF sandbox project, it can rise up the project tiers (incubating, graduated) and reach wider recognition, and a "stamp of approval" from the CNCF that k3s is ready for production use, to have another authority validating its development and existence could help with adoption and further recognition across the (admittedly chaotic and fast-paced) cloud landscape.

Ok. I thought this was a no brainer but I'll add my reasoning instead of the generic "fanboy +1" I entered before.
Quoting the CNCF's mission..

"Cloud native technologies empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.

These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil.

The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects. We democratize state-of-the-art patterns to make these innovations accessible for everyone."

What k3s does is it extends k8s to play well at the "smaller scsale" - just like monta-vista linux introduced linuxfor embedded device in the early 2000 and was the prequel to "linux for mobile"(android)
I've been working with k3s for the past 6 monthes and it brings the cloud-native practices to form-factors I can work with empowering me ein the process (Look ma, I can program autonomous drones to do image recognition on a jetson board using kubeflow on ARM)
This skillset would have been much harder for me if I did not apply my existing k8s skills to the smaller k8s compliant k3s project.
By donating k3s to the CNCF - rancher is also addressing the vendor lockin issue and I predict also redhat will leave all kind. of wierd ideas about "mini or microshift" and will adopt k3s as the standard for "embeded k3s systems.

@zouyee I shall repeat: The relation between k3s and kubernetes is that (at this stage) k3s fully implements kubernetes' system, but in a more efficient and (slightly opinionated) fashion, ready for low-end devices.

The development history of k3s being a source code fork of k8s (which, as you said, has heavily diverged now) should not interject with that above principle. K3s' relation to k8s is to implement it on a low-spec scale.

FYI, we've revamped our README.md to more clearly explain what k3s is and isn't. https://github.com/rancher/k3s/blob/master/README.md

guessing this is related ... https://twitter.com/ibuildthecloud/status/1292497393150078976

An interesting question here is as well if it wouldn't make more sense to put k3s and its building blocks into the kubernetes{,-sigs} ecosystem rather than directly into the CNCF

sig-low-spec?

An interesting question here is as well if it wouldn't make more sense to put k3s and its building blocks into the kubernetes{,-sigs} ecosystem rather than directly into the CNCF

I'd be supportive of this.

An interesting question here is as well if it wouldn't make more sense to put k3s and its building blocks into the kubernetes{,-sigs} ecosystem rather than directly into the CNCF

The TOC had a discussion with the Kubernetes Steering Committee about this but KSC expressed a preference not to go that route at this time (which doesn't mean that it can't move in the future if that were good for both projects). Please see this thread on the TOC mailing list.

+1 Binding:
8/11:
Matt Klein: https://lists.cncf.io/g/cncf-toc/message/5094
Alena Prokharchyk: https://lists.cncf.io/g/cncf-toc/message/5095
Michelle Noorali: https://lists.cncf.io/g/cncf-toc/message/5103
Liz Rice: https://lists.cncf.io/g/cncf-toc/message/5109
Sheng Liang: https://lists.cncf.io/g/cncf-toc/message/5110
Brendan Burns: https://lists.cncf.io/g/cncf-toc/message/5116
Justin Cormack: https://lists.cncf.io/g/cncf-toc/message/5118
Katie Gamanji: https://lists.cncf.io/g/cncf-toc/message/5172

Saad abstains: https://lists.cncf.io/g/cncf-toc/message/5113

An interesting question here is as well if it wouldn't make more sense to put k3s and its building blocks into the kubernetes{,-sigs} ecosystem rather than directly into the CNCF

The TOC had a discussion with the Kubernetes Steering Committee about this but KSC expressed a preference not to go that route at this time (which doesn't mean that it can't move in the future if that were good for both projects). Please see this thread on the TOC mailing list.

Let's see where we are with k3s-io/k3s#2245 at the 1 year mark in about a month.