I find it nice to understand topics based on its historical background. For example, it’d be remiss to discuss OWASP Top 10 vulnerabilities without understanding the fundamentals in network protocols, or to discuss container security without understanding address spaces or syscalls. For rolling out Agile in the enterprise, we also have to understand CAPEX requirements and how the previous methodology, Waterfall, came to be.
Capital Expenditures (or CAPEX) are funds used to acquire assets, and are in contrast to Operating Expenses (or OPEX) which are incurred to operate assets. In a company, the requirements, design, and prototyping of software is an Expense item, whereas afterwards the actual development of software falls under CAPEX, and the software can be valued as part of the company’s assets. In other words, once management deicides to go ahead with a software project, it becomes CAPEX. CAPEX projects are typically budgeted a year in advance, often with monthly expenditure targets, removal of monthly budgets if that target is not met, and careful scrutiny of expenditures.
In relation to software development:
How these CAPEX requirements influence software development can be seen in early software development life-cycles. I’d propose that CAPEX are really what led to the early proliferation of Waterfall in software development. Imagine you wanted to meet your CAPEX requirements of planning & time budgets; a natural solution is to position someone more akin to management who’s responsible for the plan, and have them assign tasks to non-managing developers. In software development, the historical model was that this management-esque role is filled by a software architect. The architect does the software design, division of tasks, and assignment of tasks to individuals lower in the hierarchy.
When we draw out this organizational structure, we can explain the rise of modelling languages, namely UML, and the code generation tools which came to be in the 90s and early 00s. Code generation, off of modelling languages such as UML, seemed like a perfect fit: the software architect would do the design in UML, generate the interfaces from it, and have developers code in the details. When we take this organization model one step further, comes the outsourcing of component development, as we envisioned that development of components would be more akin to building cogs in an assembly line.
Stepping forward, we see a return to insourcing. Fundamentally, it turned out that software development wasn’t quite like running an assembly line. If I were to offer a rationale, software development is more like the designing and building of assembly lines: there are standards and processes sure, but it’s also important to have your developers, who know their individual domains very well, contribute to the design.
I also believe there is fundamental value in having people on your side. In the enterprise, we like to talk about this as alignment: making sure your teams share common goals and success criteria. Looking back at the code generation tools and outsourcing, the model is one of compartmentalization or silos. This has deleterious effects such as finger pointing, different goals, not to mention a hard argument to make for improving the overall quality of a software product.
In recent years, many large enterprises have moved back to a model of insourcing. Yet, we have to work on breaking those historical models of compartmentalization.
Agile methodologies can be brought in by a business for a wide variety of reasons, such as to address these silos, deliver faster, or some other reason.
I’m sure we’re all familiar with different Agile Methodologies by now: Scrum, Kanban, XP are all common terms thrown around. We’re all probably also familiar the the intended values of Agile:
But, what really are the core assumptions of Agile? That is, what is implied as true for an organization to implement Agile. I’d propose that these are:
These, I believe, are the fundamentals. I’d also propose that Agile pushes for the optimal use of your developers’ hours at any given point in time.
Remember, your enterprise may have just transitioned from an outsourcing model to an insourcing one. The artifacts, such as siloed teams, hierarchical staff architecture, differing incentives, are likely still present and as a result we have to push for the fundamentals (trust, inform not dictate) while meeting a mandate for rolling out Agile.
If you’re a large organization, these CAPEX requirements are also not going to go away. And, do they really need to? Planning in advance, forecasting blockers should all be parts of good software development practices anyways. With CAPEX requirements, the key consideration is meeting targets and the core concern of being under budget for a given month would be the main issue.
So we need to roll out Agile in an enterprise, what do we do?
A greater emphasis quarterly targets, over monthly targets, would be a good start. In a given shop, a developer may be assigned to multiple projects which are continually in flight and quarterly targets enable the developer to work on tasks when it is most optimally appropriate. Remember, we’re arguing that optimal use your developers’ time is a key part of Agile. For example, given Tasks A, B, C, D and Developers 1 on A, 2 on B and C, 3 on D. Let’s say Task B in wholly dependent on A, and D is wholly dependent on B, C. When an organization might have multiple projects in flight, while Developer 2 can work on C in the meantime, it is not necessarily optimal to work on C as Developer 3 would still be blocked.
This is an optimization problem: targeting a short time interval (1 month) may lead to sub-optimal decisions over a longer time interval (1 quarter). Of course, you also have to ensure that quarters are an acceptable medium for your CAPEX reporting.
This comes in two parts:
In an Agile model a software architect, from a Waterfall model, plays more of a senior developer role - their role in a hierarchy is more flat than from management. Project managers then exist to convey business priorities. The point is two-fold: it helps break within-team silos, and it makes teams wholly responsible to other teams.
We also have to adjust how time estimates are set for a given task. Historically, there isn’t so much time estimates as there are deadlines: the architect or project manager assigns tasks and sets deadlines. Your developers then have to take shortcuts to meet the deadlines, and hence a lower quality solution. When we see developers working 12 hour days in an “Agile” shop, this is because of there is a misalignment of Agile (trust your developers) vs. corporate structure (management setting deadlines). Instead, by enabling developers to set their own estimates we can account for better quality solutions, though realistically everyone also has to be mindful of quarterly targets.
Lastly is the hard problem of setting incentive structures, of which there are Short-Term Incentive Plans and Long-Term Incentive Plans. We use Short-Term Incentives to promote recruiting and performance, and the incentives take the form of bonuses from setting & meeting targets. We use Long-Term Incentives to encourage employees to stay, and these incentives take the form of vested stock options, and sometimes bonuses tied to tenure at a company.
With software development life cycles, the main concern is the performance pay: we want to incentivize employees in the short term, but typically also want measurable criteria to base these incentives on. For example, we want to encourage Time Management & Work Ethic, though the only measurable criteria for this might be meeting deadlines.
To take this further, you might, for example, set a target that feature X is to be delivered by time Y, but how would you define the quality of that feature when it’s delivered? Agile can exacerbate this measurability problem because now we have more iterative development cycles, and incentivising developers to address technical debt is hard because that core debt metric is tough for management to quantify.
Thus, in rolling out Agile we also must adjust incentive structures; we want to align both the company’s long-term future with our Developers’ short-term incentives. In rolling out Agile, there are a few key items we have to change: