Full disclosure, when your company’s primary product is software, you know a little something about software development. In this post, I’ll reveal my epiphany on how the software development process can be applied to every other aspect of your business (and your life) — no matter your industry, no matter your position, no matter your department. [Spoiler alert: there’s a must-read book — "The Phoenix Project" — in your future!]
At xTuple, we selected “The Phoenix Project” for our in-house book club. I didn’t start out to write a review, but this atypical business book is too aha-moment good to keep to ourselves. While I’m not a developer (I work with some great ones, though), I learned how they approach their work and about cool tools such as [Kanban] for workflow visualization and management. Tools that can and will work for anyone!
“The Phoenix Project: A Novel about IT, DevOps, and Helping Your Business Win”
By Gene Kim, Kevin Behr and George Spafford
Imagine a fictitious novel about a group of very different individuals teaming up to conquer a dragon and rescue a village, except in this story the “warriors” are business professionals, the village is a failing manufacturing business (vertically integrated with eCommerce and retail components), and the dragon represents the problems facing the business, especially all of the myriad challenges of IT operations (inter-related with every other department as we find out!).
Any company — or any person — who works with, in, for, or around “DevOps” should read this humorous, yet often terrifying, novel. Boiling down our current business environment, and more specifically the global workforce, that means this book is for EVERYONE.
Nearly every business, literally, runs on — and is run by — DevOps. It’s core to the nervous system of the organization. Unless you’re a kid running a cash-only lemonade stand in the front yard.
What is DevOps?
In information technology terms, DevOps is a mashup of development (Dev) and operation (Ops). No longer “siloed,” today’s software culture merges these professionals into a single team, working across the entire application lifecycle, from development and testing to deployment to operations to quality assurance to security. Experience teaches us that working in a silo anywhere, i.e., not sharing information/knowledge with others, creates more problems than it solves.
The “Phoenix Project” Story and Why DevOps Matters
DevOps is the “manufacturing revolution” of our age. Using faux technology company, Parts Unlimited (PU), the authors illustrate how many corporate environments typically operate (dysfunctional!) and how any change, or lack thereof, can be transformational, and DevOps is at the core. Software is an integral part of every business, at every level. The story gives great insights from all perspectives, and it’s not just for technical types.
Our PU (pun intended) story opens on a catastrophe waiting to happen. A mid-level IT director (Bill), whose bosses have mysteriously “quit,” is promoted to VP of IT Operations, a position he neither covets nor wants to accept. His mission (think “Mission: Impossible”) — assigned directly by the at-first charming CEO — is to restore critical business operations and keep the company off the front-page news… all this amidst a years-overdue, over-scoped new product, Project Phoenix, and the company’s public financial confidence wavering.
Bill’s missive seems doomed to failure; Project Phoenix is PU’s “ride-or-die.” There’s company politics swirling around the project, and marketing head (and CEO heir apparent) Sarah is the primary antagonist, with an uncanny knack for blaming others for her screw-ups, at least according to IT.
Then the first problem arises… payroll systems are FUBAR (Fouled Up Beyond All Recognition). Bill’s edict is clear: save the company from itself.
Our fictional team starts out with all the earmarks of people who neither trust nor give any consideration to their teammates. Lack of communication is the least — and ultimately the crux — of their worries. Rather than banding together to solve problems, they blame-game and jump to inappropriate conclusions about one another. Personalities get in the way of professionalism. Condescension and corporate “theater” is rampant.
The discord between and among departments is blatant. The authors give us a taste of what IT Operations staffers think about the rest of the company:
- Marketing people are “art or music majors, not people with a technology background
- Salespeople promise customers the impossible and expect IT to just figure it out and deliver; they’re untrustworthy: “You know the saying, right? The way you can tell a vendor is lying is when their lips are moving.”
- VPs basically give PowerPoint presentations — to each other — all day long
- Developers “throw gas on the fire;” always crashing systems; mentally checked out or on vacation; often breaking things, then disappearing, leaving IT Operations to clean up the mess
- Information Security in the room is the best way to make sure something does NOT get done
One staffer muses that the popular British TV show “The IT Crowd” (one of my favorites!) accurately reflects IT Operations’ lot in life. The IT team — translating CIO as “Career Is Over” rather than Chief Information Officer — thinks promotion isn’t necessarily a good thing. Keeping your head low helps you avoid company bureaucracy. Made me think of the “floaters” on some reality TV shows.
As the PU story unfolds, suspicions mount, and conspiracy theories arise over how competitors keep getting the jump on them. Are loose lips on the loose?
And it gets worse. Written specifications do not get written. Despite the company’s high-tech marketing needs, IT / Operations aren’t consulted at all, and marketing often outsources IT purchases, circumventing the department altogether.
Handoffs, especially between Development and IT Operations, are never handled well. There’s the invariable ROUND-TUIT projects: IT upgrades and maintenance that are constantly put off as unforeseen emergencies and other priorities leave them behind. It’s tribal warfare.
The Assessment — The System is Broken
Bill takes his promotion — and all of the fires to be put out — in stride. His management by walking around style (and asking the hard questions) helps him put the company’s problems into “big picture” perspective as they pertain to the business itself.
He knows that operations shouldn’t really be working as firefighters, and while as a large company, PU has more people available to fight the crazy-paced daily “fire drills” than smaller companies, the issues remain the same. Everything is marked urgent. Yet, if everything is urgent, can anything be treated urgently?
Processes, procedures and systems for change management are in place but rarely followed. Lack of proper “everyone in the same room” planning is the norm. Required test environments don’t exist, supposedly due to lack of budget. Arbitrary go-live in- production dates are set with complete disregard for what actually needs to happen before deployment.
Support doesn’t flow from the top of the company’s chain of command. Updated equipment and training are ill-afforded luxuries. The company is basically flying blind. There is no situational awareness. The left hand doesn’t know what the right hand is doing.
Being proactive is an afterthought. Transparency is desperately needed.
Project delivery is driven by arbitrary — yet always urgent — drop-dead dates, due to external commitments made by the C-suite without a complete understanding of internal problems. Development then has to take outrageous and unacceptable shortcuts to deliver. The product almost assuredly will be unusable, and those clamoring for it will be seriously disappointed. Customer experience will be so bad that they’re driven to competitors. IT Operations will have to do whatever they can to make it work – and hide what is happening behind the scenes.
The team scrambles to not let perfect be the enemy of good. However, business impacts will be devastating; the enterprise isn’t operating at an enterprise level.
The newly released Phoenix software is blowing up.
Single Point of Failure — Who is your Brent?
Then there’s the “single point of failure” — the one indispensable employee who’s in the middle of every important project and pulled in a million different directions. Seemingly nothing can happen without their involvement. He/she can’t be replaced nor replicated. In “The Phoenix Project,” his name is Brent.
Production can’t be managed without a centralized list of projects: PU doesn’t know their demand priorities and commitments, status of work in process, and resource availability. IT Operations is in the middle of every flow of work, creating a bottleneck in work in process, project deliverables, outages and compliance.
At PU, spending time planning priorities had been considered a waste. They don’t follow the general rule that if something gets bumped up (in urgency), then something else has to be bumped down. They haven’t figured out how to reduce the amount of break-fix work so revenue-building project work can get done.
How to tackle your company’s single point of failure? The book starts with explaining the Theory of Constraints.
What is the Theory of Constraints?
The theory of constraints (TOC) is a management concept that just a few constraints (that single staffing point of failure, or bottleneck such as in a manufacturing production line) are all that is keeping you from achieving your goals. If you focus, you can identify what the constraint is and work around it. Think of the idiom “a chain is no stronger than its weakest link.”
But, if you make an improvement anywhere else, whether in front of or after the bottleneck, that constraint still exists. In our manufacturing scenario, work is piling up on the front-end, and work-in-process (WIP) is starved on the back-end of the bottleneck.
Managing processes in DevOps is not so different from manufacturing, running a factory, and vice versa. Every department can learn from each other. The same tools and techniques can be used to document processes, identify waste in time and/or materials, and improve the whole system.
One of PU’s board members mentored Bill. His character delivered useful focus for the task at hand by outlining the Four Types of Work and The Three Ways.
Understanding the Three Ways
The Three Ways describes the values and philosophies that frame processes, procedures and practices of every system. The goal of any business, any department, and every single employee is to truly understand — and appreciate — the system to improve their work and deliver better products and services:
- Systems Thinking — Control constraints and speed workflow, working from left to right (deliver faster results)
- Amplify Feedback — Shorten the feedback loop, working from right to left (deliver better quality)
- Culture of Experimentation and Learning — Embrace and encourage testing to more quickly apply lessons learned from failure (mitigate risk and deliver continual improvement)
Understanding the Four Types of Work
Compartmentalizing what we do and assigning metrics to each type of work helps us better identify bottlenecks, i.e., constraints:
- Business Projects — Work with a direct line of sight to the bottom line such as new products, new features
- Internal IT / Operations Projects — Work required to support Business Projects such as R&D (research and development), new architecture, infrastructure or engineering
- Changes — Modifications to scheduled “internal projects” work supporting “business projects”
- Unplanned Work — Bugs, defects, rework, crisis management, etc., that takes away from revenue-producing “business projects”
Clearly, the conclusion is that Internal Projects done well have a positive effect on Business Projects and also quash the amount of Unplanned Work.
Without giving away the ending of the book (Project Unicorn!), let’s just say that tensions escalate into territory that no business owner nor employee, manager or otherwise, wants to explore.
The Solution is Within Your Reach
- Automate recurring processes as much as you can, control and document the work (especially changes), and review, review, review
- Identify problems earlier, break down work into more manageable parts, check-in with all stakeholders to check results, make changes as they’re needed, and deliver deliverables more quickly
- Standardize for better, more repeatable, more accurate, faster results
“The Phoenix Project” made me look at lots of things in a new way. For example, everything is a system. A business, a manufacturing plant, a city’s transportation plan, even your family or your own body. The solutions to managing all of these “systems” are eerily similar. I can’t encourage everyone enough: read the book.
- Knowing your customers on a more intimate level can be the most valuable thing a company can do.
- Bureaucracy is inherent in any business. When the business grows bigger, new fail-safes are needed to ensure communication is seamless. If your business is small and people turn over, those enterprise-level communication problems also occur.
- Change management process means discipline — it’s painful to develop but it’s much worse if it does NOT exist. "You can't achieve the strategic until you've mastered the tactical."
- If someone asks how you fixed something, there’s nothing worse than saying “I don’t know.” Document! Next time it happens, we have the answer. Knowledge is power. Knowledge not shared is worthless, and worse – it creates chaos.
- If one department meets a milestone deliverable, but another group in the company is negatively impacted, don’t hold a “kegger” and send a broadcast email. It’s just bad form. You probably didn’t like it when your entire fifth grade class — except you — was invited to the cool kid’s birthday party. And if you were the cool kid, shame on you. Remember, no one is an island. It’s sink or swim together. Hold off on the party until everyone can celebrate together.
- If you have news to share, share it. Maybe the new change management process put in place just saved you from disaster — spread the news! Reinforce how something is working. Accolades, back-pats and thank-yous take little time and reap huge rewards in morale and enthusiasm. An “ounce of prevention is worth a pound of cure,” but the problem with prevention is that people rarely know about it. Communicate the wins!
- If everyone can understand how their job jeopardizes the company’s key performance indicators (KPIs), i.e., measures of overall business performance, rather than demoralized, the more likely they'll feel empowered and make better business/work decisions.
- Problems always arise. Don’t jump to conclusions. Don’t go off half-cocked and make the problem worse. Don’t back-stab or pass-the-buck or play the blame-game. Make sure processes are in place. Work the process. Proactively be a part of the solution and improve the process, as needed.
- Get to know — and understand — each other, personally as well as professionally. It builds respect, trust, and better working relationships. When getting to know someone, especially someone I work with (or interviewing to hire!), my new favorite question is: “What differentiates a good day at work from a bad day for you?”
At our last book club meeting, my xTuple co-workers chuckled as each of us seemingly identified with one or more of the characters, behavior traits and/or actions. Then we picked ourselves up, dusted off the fall-out, and talked about xTuple’s own internal progress in working toward more structure, better processes, and being more open to change to deliver better products and services to our customers.
xTuple values our company-wide interconnectedness and sharing different perspectives — internal with staff and external with customers and the broader open source community.
Stop the insanity and communicate. Create a defined sustainable process that everyone agrees on. Definitions matter. Understanding matters. Keep tweaking the process until it works. It can be done.