Twelve-Factor App – only something for the cloud?
In 2011, the developers at PaaS operator Heroku developed a set of rules that applications should follow if they want to be optimally prepared for operation in the cloud: the Twelve-Factor App methodology. It is a set of 12 guidelines that summarizes collected best practices and knowledge from years of operating their own platform. The goals are to standardize communication with the operating system and to make the cloud usable as a platform. At the same time, this can also achieve quite a bit of scalability. They published the Twelve-Factor App rules at https://12factor.net.
Rules designed by a cloud provider for applications to work particularly well on cloud platforms? As a developer of classic on-premise solutions, you don’t have to worry about that, do you? You can disagree with that quite specifically: With these rules, experienced developers got together and wrote down their shared experience. We are dealing here with distilled knowledge that takes an agnostic stance toward deployed technologies and hosting solutions. Rather, emphasis was placed on a higher altitude and general structure of the application architecture and approach to software development, so that this set of rules will not lose any of its validity in another ten years.
Let’s face it: the classic on-premise application is a monolith, consists of one artifact and is rolled out on one, sometimes several servers and connected to the network via load balancer or reverse proxy. There is nothing wrong with this, as this approach has worked excellently for decades. The only difference to the cloud application is that the runtime environment of a monolith is often “hand-crafted”, i.e., often manually prepared in such a way that the application can be operated. This can then range from server configurations to specially set up firewall configurations to correctly mounted network drives. This exact configuration of the runtime environment is required by the application, otherwise it cannot run. Cloud applications, on the other hand, have to be much more encapsulated, as they may be executed tomorrow in a completely different data center or be started several times in parallel. In addition, they are often constructed from several different artifacts that are interconnected and rolled out independently of each other. But focusing on the essentials, cloud applications are constructed from multiple “monoliths”, so roughly speaking they are exactly the same thing. So do the factors thought of for cloud applications also apply without restriction to classic applications? Let’s take a look:
The benefits of version control systems are well known, so the use of just such no longer scares anyone. Development teams, distributed or not, probably can’t imagine their work without Git, SVN or Mercurial. Also, most will agree that the same repository is rolled out to all environments. After all, the same software should run on all environments, why not take the same source?
However, the description of this factor also states that an application should consist of exactly one repository. Self-built dependencies should therefore either be archived in such a way that they can be resolved automatically by the Dependency Manager, or simply exist in the same repository.
In addition, this factor states that the same source code should be rolled out to all environments. This means that the same artifact should be used for the production environment as for all other environments. At this point at the latest, problems can arise in many a project. But how these can be handled is discussed in the following factors. So factor 1, the codebase, should not be a problem for monoliths.
Cloud applications are sometimes run on one server, sometimes on another. This can also lead to differences in operating systems. Therefore, they should not rely on any pre-installed libraries or other operating system occurrences. Instead, applications should declare their dependencies using a dependency declaration manifest and isolate themselves from their environment during execution using a dependency isolation tool to avoid unintentionally using dependencies. This also applies to operating system tools, for example ImageMagick or curl. If such tools are needed at runtime, they must be shipped with the application. Examples of such tools in the Ruby world are the combination of `Gemfile` and Bundler, in the Python world Pip and Virtualenv. In the Java world, tools such as Maven and Gradle have become widely used.
The advantages of following this factor should be obvious and are just as applicable to monoliths. Not being dependent on the specific setup of the underlying machine means easier onboarding of new colleagues in development, easier setup of new environment on customer side and allows to run on other operating systems at all. Especially in the field of web development, it often happens that servers are Unix-based systems and developers are stuck on Windows machines.
App configuration consists of everything that could change between deploys and environments. This includes, among other things, credentials to third-party systems, access to third-party services, and values that change per deploy. To specify default values that may only need to change on certain environments, these can either be compiled into the software, or specified in configuration files and later overridden via, for example, environment variables.
A good example of such mechanisms can serve the configuration mechanism of Spring Boot applications. In the chapter “Externalized Configuration” is described, in which order configuration values, which one usually puts in the `application.properties` or `application.yml`, can be overwritten from the outside.
With the help of this mechanism the software can be programmed in such a way that all standard values, which one entered in the central configuration file, are valid for the local development environment of the programmers. As soon as the software is then delivered, the application can be configured suitably for the respective area of application over environment variables, command line parameters or Java system properties, in order to call only a few.
This approach is completely independent of programming language and operating system, brings critical advantages in cloud applications, but can also generate very great added value in monolithic programs.
The authors of the twelve factor app refer to all services that consume an application over the network as backing services. They distinguish between services that are operated by the same administrators who maintain the application and services that are provided by third parties. Examples include the application’s own database on the one hand and binary data stores such as Amazon’s S3 or SMTP services on the other.
A software project that wants to satisfy the twelve factors treats these services, regardless of who is responsible for them, as a loosely connected resource. One should be able to exchange a locally created database, which is located on a neighboring VM, with a database from a cloud provider. The third factor, Config, in particular should make this possible: all access data is available in the configuration and can be exchanged by the operators of the software. The application only takes care of the technical nature of the network connection and draws on the configuration data to connect to the service. The service is now treated only as a connected resource. This factor plays a special role in the cloud environment, where communication across network boundaries is much faster. The idea that a service to be consumed is only accessible via a non-permanently available interface, which can additionally introduce latencies into the communication, must always be present in the back of the mind.
With monolithic systems, this predicted change in the operating environment is not necessarily a given – the software simply does not need to have the same level of flexibility. Nonetheless, it costs no extra effort,++ and is considered best practice, to build the system’s interfaces to rely only on the technically guaranteed properties of the technology, rather than assuming, for example, that the database will always reside on the same host and thus latencies will be close to zero.
According to the 12 Factor App, the Build Stage is responsible for converting a repository’s code into an executable artifact, also called a build. The Release Stage combines the artifact with the configuration of a deployment target to create a release. Finally, the Run Stage executes the Release. In addition, it is defined that a Release has a unique name, called a Release ID. A scheme is not specified, instead timestamps or incrementing numbers are recommended.
This procedure makes possible that the same state of a Repositories on all environments is used and clarifies again the importance of the third factor configuration. In addition, a rough structure of a build pipeline is already defined and the Deployprozess is touched on in such a way. At the latest here we get a first taste of the fact that experienced software developers have worked on this document. Each child is given a name and a clearly outlined task. Well-known best practices, which in themselves already represent distilled knowledge, are assigned to process steps and prove elegance in their simplicity.
In some technologies or existing legacy systems, the function of a release stage may not be implemented within a short period of time, but in the long term, such a structure will have a positive impact on factors such as the stability of releases – and not only in the cloud environment.
The application is to be executed in the Run Stage as one or more processes. In the simplest case, this means that the application is a simple script that is executed from the command line.
The most important aspect, however, is that such processes are stateless and do not share any data among themselves.
Being stateless means that processes always terminate completely over multiple runs and thus do not store any data “volatile” in memory. All data to be persisted must be stored in a backing service, such as a database.
Not sharing data with each other means that processes can operate independently of each other and must assume they are running in isolation from other processes. The only way processes can share data is to communicate with backing services.
If these criteria are met, not only has a clear application structure been achieved, but important processing steps, which are defined as processes, can be executed in parallel to achieve horizontal scalability.
12 Factor Apps are self-contained applications that open ports on their own to communicate over the network. This is in stark contrast to the idea of applications running inside already installed containers such as Apache modules or Java servlet containers.
One advantage of this approach is that server operators can manage the application at a very low technical level, since it appears at the operating system level as a process that does not depend on other processes, such as extra web servers that have been started. This makes many management tasks, such as automatic restarts and monitoring, easier to implement.
Another advantage is that by clearly delineating and exporting application services via network interfaces, any 12 Factor app can serve as a Factor IV backing service. This opens up new possibilities for networking different services.
This is certainly one of the points that is most likely to be ignored from the perspective of a monolith, since one of the core aspects of a monolith is that all the required capabilities are available in bundled form. However, there is nothing to prevent a monolith from being operated as a standalone application without a surrounding web server.
This factor again reinforces factor VI. Processes should be considered first-class objects in the architecture of the application. They form the primary component of the application structure. It is suggested to group processes by type, for example, web processes that process HTTP requests or background processes that perform regular tasks. Such processes can start child processes themselves, but thus can only scale a limited amount.
The biggest advantage is certainly that horizontal scalability at the application level can be achieved via this process model. If the prescribed conditions were observed, parallelism can be increased without problems in order to process more tasks.
In the cloud environment, this usually means that more virtual machines are used or more containers are started up in the cluster. In the monolith, this can mean that a configuration is adapted so that more background processes are started, for example. Here, scalability across multiple machines may not necessarily be a given, but it is then easier to work with an upgrade of the hardware.
Processes that meet the above conditions and are part of a 12 Factor App can be started and stopped at will for fast scalability and customizability, for example by changing configuration. They should also be able to start up and shut down quickly, within a few seconds, and the process model should be able to handle an abrupt process stop, such as when the host crashes.
This list of requirements is a clear admission of the conditions in the cloud environment. While a fast application start also suits a monolith very well and errors in the underlying hardware of the host or the proverbial pulling of the power plug should never lead to catastrophic process damage in the best case, the conditions in the on-premise area are usually considered more stable. Nonetheless, this point highlights some, albeit self-evident, worthwhile goals.
The authors describe that gaps can exist between environments for historical reasons. For example, it can take a long time for a new version to go live, different departments are responsible for development and deployment, and the tools and resources used differ between development and the production environment. A 12 Factor App aims to be rolled out via Continuous Deployment and thus close these gaps. The time gap is closed by automation, developers are at least involved when rolling out the applications and technologies are not used per environment but in the entire project.
This factor describes modern thinking and best practices rather than an approach that is specifically reserved for microservices or cloud applications. The DevOps line of thinking is clearly recognizable and if DevOps dictates one thing, it is that this philosophy is technology-independent.
12 Twelve-Factor apps do not care how their logs are stored or processed. Instead, they should simply send their output to `stdout` so that the environment can process it. This approach simplifies log handling for everyone involved, and should be the first choice for any form of program.
Even in the best-tested project that has the strictest quality processes, it is sometimes helpful for developers and operators to intervene in the running system via tools provided with the application. The authors recommend paying attention to the process model and all other factors for these administrative processes as well. For example, it is recommended to use the build tool of the respective technology for administrative processes, in Ruby for example rake or rails, in order to obtain the same advantages with regard to dependencies and isolation as with the application itself.
REPLs are explicitly mentioned as an alternative to the build tool. Languages such as Python or Clojure have supported this type of technology for a long time, Java also for several versions. However, the capabilities differ quite a bit and using it is certainly not accessible to everyone.
Closing Thoughts
After considering all the factors of the Twelve-Factor App methodology, a picture presents itself that on the one hand is clearly designed for microservices and cloud environments. But on second thought, it becomes clear that there were developers at work here who have seen quite a bit and made a mistake or two themselves. The fact that the addressee for the 12 Factor Apps is not the classic monolith is due to the spirit of the times and the trend in technological development.
So what we have here is a document that collects best practices and condenses them into twelve pieces of advice. Some of them will certainly be taken for granted, but it is helpful to have the formulations and to be able to refer to them when needed.
If you follow the advice that is given, you will get an application that can shine at the macro level with clear structures and a simple approach to hosting. The opposite of simplicity is complexity and this was already identified in 1968 at a conference held by a committee in Garmisch.
Comments
No Comments