There are lots of ways DevOps can fail. For all the revolutionary promise of the idea, it takes a tricky cultural shift to get Development and Operations working together. Many companies—especially big ones—take a top-down approach. C-suite execs trumpet a Brand New DevOps Initiative, which everyone else resists, undermines, or ignores.
As a developer at a SaaS company, my success depends on Ops. Too often, Dev and Ops are divided by mistrust and rarely talk between releases. Bridging this gap requires finding a way to dial down the tensions that spring from differences in status and divergent incentives.
Here is an example I’ve seen too many times. A project is running late, so Dev leaves out operational features and the first deployment is a spectacular failure. Dev blames Ops, Ops blames Dev, and the CEO is called in to arbitrate. Dev is usually closer to the CEO, and it doesn’t help future relations that Ops is often left holding the bag.
Given deep-rooted mutual suspicion, it seems obvious that trust is key to making DevOps work. Trust between individual contributors can overcome mistrust higher up, even if the opposite is not true. A great relationship between the CIO and the engineering lead does not guarantee DevOps success. That’s why I think DevOps best practices need to begin at the bottom. The ideal starting point is a short project that adds a lot of business value, and is kept low profile until it succeeds.
One approach I’ve tried as a Dev lead is to get an Ops Ambassador appointed to my team. (We’re still trying to convince the SFPD to forgive his parking tickets). The Ops Ambassador attends weekly planning meetings and daily standups, which inspires holistic thinking. Dev is more likely to consider the difficulties of identifying bugs in production when the people affected are in the room, and Ops better understands why certain compromises were made. Recently, the Ops Ambassador helped us design a logging system with less spam. When a customer crisis cropped up, we could sift through the logs and find the problem faster.
To be sure, many people say DevOps fails because of technology rather than people. I often hear DevOps is impossible without better test automation or continuous integration. These technologies certainly help build higher quality features faster, but they don’t help fix problems quickly when bugs inevitably arise. Inviting Ops to be a trusted partner in development elevates unsexy items such as feature flags and logging from afterthoughts to integral parts of the design.
The value of trust extends beyond just Dev and Ops, of course. Other dyads of distrust plague many software companies, such as Dev versus Sales or Dev versus Support. One strength of AppDynamics is the depth of collaboration across these boundaries. We aspire to reach beyond DevOps to a DevOpsDocSupportSales joint focus on the customer. A few of my colleagues have similar thoughts on the possibilities of applying DevOps on a larger stage, you can read their thoughts on BizDevOps here.
