Why Package the App and Its Automation Together?

In the last article, "What is a Modern Application," we talked about what those of us on the Chef Habitat project consider to be the essential qualities a modern application must have. The trend towards highly distributed service-oriented architectures has implications for how apps are managed and deployed. A more decentralized approach than what is commonly used today is required. By this we mean that the automation itself becomes service oriented, and applications move from being functional utilities with centralized command and control to self-configuring, intelligent services that can coordinate their behavior with each other.

The need for service-oriented automation motivates a central design principle of Chef Habitat: Chef Habitat is automation that travels with the app. The artifact used for deployment contains both the application and its automation.

Let’s put this in context. Scalability is really only achievable when intelligence itself is distributed. By scalability we mean more than the ability of an application to adapt to increased demand. We also mean human cognitive scalability. Operations teams must be able to understand what's going on in their systems. It's easier for them to do this if they only need to understand what's happening between a few services. It's much more difficult to look at a central orchestration point that describes the entire ecosystem.

Mark Burgess is the father of configuration management theory and the author of CFEngine. He distinguishes between a centralized approach, which he calls obligation-based, and a decentralized approach, which he calls promise-based. He asserts that promise-based systems scale much better, both from a technical perspective as well as from the perspective of cognitive load.

Obligation-based approaches, as applied to application management, assume that an application is "dumb," that it has no capacity for self-governance and that it must be managed by an overarching system, an orchestrator, which dictates how it should behave. We call this automation by side-effect, which is where we try to set up preconditions for an application—one that does not know it is being automated—and then try to recover from the inevitable failures. Such a system must anticipate every contingency and edge case. When systems were simpler and applications monolithic, this may have been possible but given the complexity of today's environments, obligation systems are themselves convoluted and brittle, resource intensive and, of course, represent a single point of failure.

Promise-based approaches assume that each component is an autonomous agent, with the ability to govern itself. Each agent publishes its intentions, which represent clear contractual boundaries of what the agent will attempt to provide. The overall model is one based on cooperation between independent entities.

Another benefit is that reliable systems can be built, even when the underlying foundation is unreliable. Promise-based systems assume failures will happen and that they are normal. Again, because intelligence is distributed, it is possible to compensate when some component fails because the remaining healthy components will self-organize and reconverge to satisfy the original promise that was made as a group.

An example of a decentralized approach that is, in fact, used by Chef Habitat, is a gossip protocol that disseminates state changes to other members of the Supervisor network. A component that is changing state—or an observer that notices a change in state—propagates that state change as a rumor to its peers. Eventually this state change is propagated to every component in the system. It's a way for components to build a global map from limited local interactions. A component only has to send to a subset of other nodes. In terms of failure, by combining the reachability data from many different components you can quickly determine when a particular component is down.

In contrast, a centralized approach to failure handling is to use a database from which all nodes query information. This presents scalability and performance problems for large distributed clusters; not only is the single database a bottleneck, but the speed at which all entities can be notified of state changes is very slow, since they need to periodically poll for new state data rather than have it sent to them.

Again, to summarize, rather than having a central overlord that enforces policy, it's far better for each component to understand what its standard of behavior is, to try and follow those standards, and to let other components know about how well it’s complying with those standards. That's the world of Chef Habitat. Chef Habitat organizes autonomous components into a group made up of pieces that can see each other and share policies. With Chef Habitat, the automation, or the intelligence to provision, configure, deploy and manage, travels with the application.

Summary of Benefits

To summarize, packaging the automation with the app is important if you want:

  • Scalability
  • Reliability
  • Flexible versioning and rollout
  • Simpler app development process
  • Ability to deploy anywhere
  • Ease of management


When intelligence is distributed, there is no single point of failure or bottleneck. A system that places the intelligence at the edges and defines a policy in all the individual, autonomous actors that are eventually going to converge on that policy, is an easier system to understand and manage.


When intelligence is distributed, you can build reliable systems on top of unreliable base layers. Because there is no single point of control, there is no single point of failure. An example is a leader/follower topology, where, when the current leader fails, a new leader is elected by the other components without any external directive.

Flexible versioning and rollout

When automation is integral to the application rather than imposed externally, flexible versioning and rollouts become much easier. Components can be separately versioned, and each can have its own rollout strategy. Different patterns for deploying applications are also possible. For example, investigative techniques such as canary deployments and A/B testing are much simpler with distributed automation.

Simpler app development process

Distributed intelligence simplifies the development process. Standard capabilities such as monitoring and health status can be implemented through a well-understood API that presents a standard external interface. Developers don't have to create their own management systems for the infrastructure, the runtime and the application itself. They can spend their time on features that actually improve the business. For teams that have adopted DevOps culture, a distributed model makes collaboration easier because boundaries are well understood. It also becomes possible to quickly iterate and test individual services and components. Finally, developers don't need to tie their application to a particular runtime environment.

Ability to deploy anywhere

Application automation frees applications from being tied to a particular environment. Packages can be run unmodified across a wide variety of runtime environments, from bare metal and virtual machines all the way to containers such as Docker, grid systems like Mesosphere or Kubernetes, or even PaaS systems such as Pivotal Cloud Foundry.

Ease of management

Packaging automation with the application makes the environment simpler to manage. When application automation, deployment coordination, and service discovery capabilities are built in, entities can coordinate with each other to deploy according to some defined policy rather than relying on a central orchestrator that forces entities, one at a time, to abide by that policy.