How We Built The Tech That Powers Our Serverless Cloud

Today we’re releasing the container orchestrator that powers the Bismuth Cloud platform — our homegrown system that enables us to deploy your applications in seconds.

We wanted to build a system from the ground up to give complete control over where, when, and how projects are deployed. Conceptually, Bismuth is a Function as a Service (FaaS)-like platform. You (or our AI!) write code, and the deployment of that code is automatically managed — there are no servers or even containers to worry about. Of course, your code does actually need to be run somewhere, and this is where the container orchestrator comes in. It abstracts away servers (cloud VMs or bare metal) and makes it possible for our upstream services to create containers with your apps across an entire fleet of machines with a single API call.

Before we go further though, let’s define what a “container” is. For those of you who have used Docker before, you’ll already be familiar. But for others, containers provide a way to bundle up an application, all of its dependencies, and a bit of metadata about how to run it into a single package. A container runtime can then take this package (the “container image”) and run the application in a fully hermetic environment. In our case, a container is transparently created by Bismuth when you deploy which includes Python, some core Python libraries, and your code, and this is what we need to deploy.

Going back, this problem of running containers across a cluster of computers isn’t new — things like Kubernetes or Apache Mesos have solved the problem of “run a container somewhere” for years now, so why create a new bespoke system? Well in short, existing solutions are both too complex and also not flexible enough since they’re geared towards a single company deploying many of their applications on a cluster, not multiple companies deploying arbitrary code onto a shared platform. They all provide some basic ways to isolate workloads, however the fundamental idea they’re built around is not what we need, and they are inflexible in ways that we require (e.g. hard network isolation between containers, ability to have part of a workload deploy into an isolated container and part not, etc.). It wouldn’t be impossible to make either Kubernetes or Mesos work, however it’s just not what they’re intended for. Additionally, scaling these systems comes with their own challenges, and using a managed Kubernetes solution for instance would lock us into a specific provider.

There’s also a handful of open source FaaS platforms (e.g. OpenFaaS, OpenWhisk, Fn, etc.) however many of these suffer from the same fate as Kubernetes (as most deploy on top of Kubernetes!). Like many FaaS systems, open source or otherwise, these platforms also have a custom event processing model that would require your code to be invoked by the platform in a non-portable way. We always want your code to be deployable wherever you might want it, but these platforms would require you to run your own instance of the FaaS platform to be able to run your software — a significant amount of operational overhead.

So with the “why” out of the way, let’s take a more technical look at the system with a 10,000ft overview walking through the lifecycle of a container.

Technical Overview

First, a create container request is sent to the API. The API is the “control plane” side of the system, providing a simple CRUD-like interface for all containers running in the orchestrator. To create a container, the API picks an available backend machine the container will run on, and inserts a new node into ZooKeeper representing the deployment of this container.

Next, the backend (bismuthd) receives a notification from ZooKeeper about this new container. It takes a few steps to create and configure the container on the host including:

• Creating the container filesystem and namespaces via containerd
• Configuring container networking
• Starting the service provider for the container (our abstraction providing a super simple interface to K/V, blob storage, and secrets — see below)
• Starting the main container process
• Polling the defined entrypoint until seeing success or the container reports as stopped (indicating startup failure), and saving this state in ZooKeeper.

Now when a request comes in for the given function, the frontend (bismuthfe) looks up all available backends for the function, picks one, and simply proxies the request through to the backend. The backend then receives the proxied request from the frontend and proxies it again into the container, completing the flow from user request to your code.

There’s one last bit to the orchestrator however, and it’s a large part of why we chose to write our own system: the aforementioned “service provider.” This is a separate process run for each container which abstracts away key/value storage, blob storage, and secrets, integrating them into the rest of the Bismuth platform. We chose to run one instance of the provider per container over a centralized service for a few reasons:

First, reliability. Despite having some complexities of its own, it’s often much easier to manage a lot of little services instead of one big one. We don’t have to deal with a single user overloading parts of the system (noisy neighbors), complex sharding or failover to keep this one service up all the time, or any of the “fun” distributed systems problems that arise if we want to do something more complex at this layer.

Second, this approach also makes authentication (from the container to the service provider) easier. We don’t need to create or use a centralized authentication/authorization service, and don’t have to deal with generating and distributing tokens between the service provider running somewhere and the containers. Instead we can use transient tokens created on host between the service provider and container. And even this is not strictly necessary as container networks are also strongly isolated as mentioned previously, so a given container can only access its specific service provider instance.

And that’s the entire request pipeline. When you update your code or delete a project, we just remove the container and start up another if necessary.

So that’s how we deploy and run your code in a flash with our custom container orchestrator. If you want to dig deeper, the repository is fully open source so you can explore the code or give it a spin locally. Or if you want to see it in action, deploy an app on Bismuth today!