Kubernetes vs. CloudFoundry [closed]

As both a CloudFoundry (past) and Kubernetes (present) commiter, I'm probably uniquely qualified to answer this one.

PaaS-like

I like to call CloudFoundry an "Application PaaS" and Kubernetes a "Container PaaS", but the distinction is fairly subtle and fluid, given that both projects change over time to compete in the same markets.

The distinction between the two is that CF has a staging layer that takes a (12-factor) user app (e.g. jar or gem) and a Heroku-style buildpack (e.g. Java+Tomcat or Ruby) and produces a droplet (analogous to a Docker image). CF doesn't expose the containerization interface to the user, but Kubernetes does.

Audience

CloudFoundry's primary audience is enterprise application devs who want to deploy 12-factor stateless apps using Heroku-style buildpacks.

Kubernetes' audience is a little broader, including both stateless application and stateful service developers who provide their own containers.

This distinction could change in the future:

• CloudFoundry could start to accept docker images (Lattice accepts Docker images).
• Kubernetes could add an image generation layer (OpenShift does something like this).

Feature Comparison

As both projects mature and compete, their similarities and differences will change. So take the following feature comparison with a grain of salt.

Both CF and K8s share many similar features, like containerization, namespacing, authentication,

Kubernetes competitive advantages:

• Group and scale pods of containers that share a networking stack, rather than just scaling independently
• Bring your own container
• Stateful persistance layer
• Larger, more active OSS community
• More extensible architecture with replacable components and 3rd party plugins
• Free web GUI

CloudFoundry competitive advantages:

• Mature authentication, user grouping, and multi-tenancy support [x]
• Bring your own app
• Included load balancer
• Deployed, scaled, and kept alive by BOSH [x]
• Robust logging and metrics aggregation [x]
• Enterprise web GUI [x]

[x] These features are not part of Diego or included in Lattice.

Deployment

One of CloudFoundry's competitive advantages is that it has a mature deployment engine, BOSH, which enables features like scaling, resurrection and monitoring of core CF components. BOSH also supports many IaaS layers with a pluggable cloud provider abstraction. Unfortunately, BOSH's learning curve and deployment configuration management are nightmarish. (As a BOSH committer, I think I can say this with accuracy.)

Kubernetes' deployment abstraction is still in its infancy. Multiple target environments are available in the core repo, but they're not all working, well tested, or supported by the primary developers. This is mostly a maturity thing. One might expect this to improve over time and increase in abstraction. For example, Kubernetes on DCOS allows deploying Kubernetes to an existing DCOS cluster with a single command.

Historical Context

Diego is a rewrite of CF's Droplet Execution Agent. It was originally developed before Kubernetes was announced and has taken on more feature scope as the competitive landscape has evolved. Its original goal was to generate droplets (user application + CF buildpack) and run them in Warden (renamed Garden when rewritten in Go) containers. Since its inception it's also been repackaged as Lattice, which is somewhat of a CloudFoundry-lite (although that name was taken by an existing project). For that reason, Lattice is somewhat toy-like, in that it has deliberately reduced user audience and scope, explicitly missing features that would make it "enterprise-ready". Features that CF already provides. This is partly because Lattice is used to test the core components, without some of the overhead from the more complex CF, but you can also use Lattice in internal high-trust environments where security and multi-tenancy aren't as much of a concern.

It's also worth mentioning that CloudFoundry and Warden (its container engine) predate Docker as well, by a couple years.

Kubernetes on the other hand, is a relatively new project that was developed by Google based on years of container usage with BORG and Omega. Kubernetes could be thought of as 3rd generation container orchestration at Google, the same way Diego is 3rd generation container orchestration at Pivotal/VMware (v1 written at VMware; v2 at VMware with Pivotal Labs help; v3 at Pivotal after it took over the project).

Cloud Foundry is a great tool assuming you are willing to always work within the constraints of the offering as it is very opinionated/prescribed. The web UI is cool to look at the first day but is rarely used after you begin to work with the client and have configured your CI/CD pipeline. I have found that Cloud Foundry is great until use cases pop up that are not easily supported fully within Cloud Foundry. Delivering these use cases can delay projects as you attempt to solve for those problems, as a result you lose visibility of the infrastructure and support benefits of those components that are then mostly running outside of Cloud Foundry(think multiple databases, kafka, hadoop, cassandra, etc) I suspect over time, momentum surrounding Docker and the inflexibility of Cloud Foundry will drive users to Kubernetes, Mesos or Docker Swarm/Datacenter. It is possible that Cloud Foundry could catch up to these three but it would seem unlikely due to the popularity of these open source projects.

It's hard to answer why a company would build a product that is substantially similar to another product. There are a lot of reasons. Maybe they already started using it and are invested in it. Maybe they (CF) think Kubernetes is done badly or is getting the API/model/details wrong. Maybe they think they can move more quickly if they control the whole product rather than contributing.

Granted, I say this as a Kubernetes developer - one might ask the same questions of Kubernetes vs Mesos, Amazon ECS vs Kubernetes, or Docker Swarm vs Kubernetes.

I hope that over time, we are all trending in the same direction and can collaborate more and spend less time reinventing each other's work.

As for Kubernetes - the focus is on app developers: simple and powerful primitives that let you build and deploy apps at scale very quickly. We're leaning on our experience (well, Google's) with similar technologies to chart our course. Other people will have different experiences or opinions.

A significant difference, in my opinion, is the approach they are taking:

CF builds runtime from 3 components automatically: user provided application binary, buildpack containing middleware needed to run the app and an OS image (a stemcell). The CF user (a developer) has to provide application binary only (e.g. an executable jar file). The CF takes care about the rest, that is packaging and running the app.

Kubernetes expects from a developer Docker images that contain middleware and OS already built in and ready to be run. For that, Kubernetes “deployment manifest” (e.g. a Helm chart) describes not only a single app or service but all [micro] services that comprise your solution at runtime. You submit a single declarative description of your runtime and Kubernetes takes care about actual runtime state matches your provided description.

So the CF approach allows it to address such use cases like “replace an OS with a patched security flaw in your entire cloud without downtime for your services”. But it also focuses on service per service deployment instead of declarative description of a target “ideal” runtime of your system.

After 4 years trends look like this:

Kubernetes clusters are getting really cheap these days and tooling environment for kubernetes is better.

Also most of competitive features listed by other posters are cecently easy to replicate inside kubernetes these days.