The Entire Software Development Process, Open-Source and Automated via BackStage.
BackStage, Developers, Self-Service Environments, and Platform Engineering
Introduction
The #1 KPI is not how fast a developer codes, but rather how long it takes from the time a developer starts to work on a new feature till it gets to production.
In this blog I’d like to describe a few concepts and present a real life example, where we utilized a chain of open source tools to automate the entire software development process, from code to production.
The focus is on the developers and how we can give them time back, so they can innovate and not “waste” their time on building the tooling around.
In a nutshell, a developer needs only an IDE and BackStage for all his software development requirements. I’ll show what we created in BackStage, which becomes the de-facto standard for internal developer portals (IDP).
Let’s start with a few definitions of Platform Engineering, Self Service environments and Internal Development Portal (IDP).
Platform Engineering
Platform engineering is a new emerging trend within enterprises. We are all familiar with DevOps. DevOps work closely with developers to take their code and make it production ready. This is done via CI/CD pipelines, security scanning, cost calculations and many other tools integrated as part of the pipeline.
Platform Engineers work loosely coupled with developers. They create for the the platform where they can create environments, e.g. a Kubernetes cluster that includes all the necessary components, so it provides an environment developers can deploy and test their microservices. In many cases it includes components like Istio and API gateways.
Platform Engineers also provide an automated pipeline for developers. When the developer wants to deploy his code he needs only to click on a pipeline link, or even this process is automated via Git Actions. Upon Push or Merge the all deployment process is automated.
You can read more on Platform Engineering here.
BackStage as the emerging IDP Portal
BackStage provides a 360 view for the developer.
- A developer can look at his source code or other team members’ code.
- View and edit the documentation
- Find dependencies between components,
- Ownership — who is the owner of that component
- Read and understand the APIs from SWAGGER Hub, published by microservices owners
- Provision an environment from a blueprint catalog, already created for him by Platform Engineers, e.g. EKS cluster + Istio
- Deploy his code to the environment via a pipeline created by Platform Engineers, e.g. a Jenkins pipeline.
- Monitor the status of the deployed services and pods in Kubernetes.
BackStage has a service definition language, very similar to Kubernetes YAML, where one can define services and publish them to BackStage service catalog. These services are for developers consumed by developers.
BackStage also comes with an open plugin system shown below, where one can add his own plugins.
You have many plugins out of the box, Jira, Jenkins, Git, API Docs, Artifactories, Kuberenetes, ArgoCD, and many many others, just name it.
Cloudify automated the provisioning of BackStage and you can select it from our marketplace as shown below.
Cloudify also provides Platform Engineers a way to create self service environments, which are built for and consumed by developers. More details on this in the next session.
Creating and Using Self Service Environments
Developers can consume self service environments created for them by Platform Engineers, while they keep working with their familiar IDE and Jira tools.
Platform Engineers create and maintain self service environments using Cloudify.
Cloudify provides an environment life cycle management for the creation, deleting and Day-2 updating of an environment. Environments can be complex like spawning an AWS VPC, defining the right subnets, creating an EKS cluster + credentials, Istio mesh + ingress gateway. Now we abstract only the necessary parameters in a simple way, so they can be consumed easily by developers.
Developers only work in their IDE and BackStage. At the moment they push new code, the pipeline is activated automatically, creating all the artifacts and pushing them to an Artifactory like JFroge or Nexus. Then it publishes them to Kubenetes and creates pods, services and all necessary resources, e.g Istio Ingress rules.
Below is an example of two self service environments created, dev and prod. Each environment exposes a set of parameters to be consumed by CI/CD pipelines or directly by the developers to deploy their microservices artifacts and test their code.
Below is an example of self service environments consumed by developers from BackStage.
How Cloudify is Integrated with BackStage and future work
Cloudify developed a BackStage plugin for creating self-service environments by developers.
This is done as shown below.
In this example we created two cost optimized environments, one for development and on for production.
The way the plugin works today is as follows — there is a scheduled task in Cloudify manager that scans for new environments created by Platform Engineers and it syncs it with BackStage to create new components and entities.
Future work will extend the BackStage system model to define self-service environments as components and their depended resources.
You can think of the above system, which is composed from multiple components, as an environment.
This provides the developers a way to customize their self-service environments like LEGO Blocks. For example I want to create a self-service environment that includes Kubernetes + Istio + RDS or MySQL. Another development team can create another environment that has EKS and MongoDB. It is very flexible.
This is achieved by utilizing BackStage YAML manifest files like for example a Kubernetes cluster manifest, shown below, for each Lego Block resource.
There could be another BackStage YAML file for example RDS database. There will be many LEGO Blocks representing different resource types.
Then the plugin translates the BackStage manifest into Cloudify internal representation. Cloudify will provision these resources and stitch them together to create the required self-service environment.
This process can be streamlined utilizing BackStage entity providers and processors as explained in “The life of an Entity”.
In the future we are going to integrate the Cloudify MarketPlace catalog into BackStage Catalog, as shown below.
Monitoring Microservices Status with Kubernetes Plugin
After the developer pushes his code changes and container images are built and published to Kubernetes (automatically by the CI/CD pipeline), he can monitor their status. This is done utilizing an open-source Kubernetes plugin, created by the BackStage community.
Below you can see the status of three pods and their CPU and Memory consumptions.
POD additional metrics can be retrieved as well.
Cost Calculations
A special plugin can provide cost calculations. This is part of BackStage Cost Insights.
In our context it is very important to understand the cost of an environment. For example a cost breakdown of each environment that we created. In the example below you see the cost of Dev and Prod ebvironments that we created.
You can get overall cost overview over time as well.
Sometimes it is more important to do cost estimation, aka how much this environment is going to cost me.
We’d like to know this info, and even get it approved, before we create a new environment. For example a Kubernetes cluster with three medium size nodes on AWS. How much it costs me before I provision it. Cloudify can provide this information with Infracost and we are working to utilize cloud providers cost calculators like the AWS one.
Summary
In this blog we went over the development work, to automate the software development efforts from code to production, and turn it into a streamlined production line, while giving the developer to continue working with his existing tooling and automate tasks which are not pure development.
This saves an enormous amount of time for developers and they can focus on the business logic code and innovations.
We saw how Platform Engineers can create self-service environments that are consumed by developers.
Developers continue to work with their familiar toolings and get a 360 view from BackStage, from code documentation, API hub and self-service environemnts to deploy and test their code.
What Developers need is only BackStage and their familiar IDE.
We also saw how we can extend the BackStage model and define customized YAML manifest files that represent self-service environment components, and they are translated into Cloudify internal representation for life-cycle management, provisioning, Day-2 updates and deletion.
And last but not least, doing the above we achieved our #1 KPI — how fast it takes from the time a developer starts to work on a new feature till it gets to production.
