Everybody, who I had a chance to work with, seems to be aware of the benefits of requirements traceability. However, no one seems to actually attempted to do this in a real enterprise solution project.
Recently there was a question asked in INCOSE group on the LinkedIn about this particular topic. The inquiry was if there any published work exists that explore quantification of the return on investment made into proper requirements traceability framework. The question sparked an amazing discussion, which I recommend to have a look at, but there are few insights that I’d like to cite here.
First, Mark Powell presented an anecdote from his own experiences (emphasis and comments are mine):
On a medium sized project ($180M) where I was CSE [Chief Systems Engineer — I.L.] a few years back, one of the customers offered up a design idea with the carrot that if we used his idea that he would increase the contract by $2M. Since I was CSE, we had a formal and baselined CM Plan [Configuration Management — I.L.], and a very robust and thorough schema for our requirements database that not only used parent-child traces, but traces across to all the related requirements (mostly ilities). The PM was all excited about the extra $2M, and everybody on the PM IPT (except me, I was not sure) thought we would actually make money on the deal and meet schedule, and should just do it carte blanche. But, as CSE, I insisted that we properly write a CR for the customer, properly assess the impacts to cost and schedule, all per our baselined CM plan. Most of the PM IPT, especially the project manager and some of the other customers in fact, was incensed with me and my insistence that we follow our own baselined project procedures, and just blindly make the project design change.
We used our requirement database tool to find all the requirements that would possibly be affected by this CR, based on our traces up, down, and across, and put together a change package for the pertinent engineers to assess the impacts to cost and schedule. After CM got all the impacts back from the engineers with signatures, and after we reconciled them to remove duplications and overlaps, the net impact of this proposed change was $67M and a 14 month schedule slip. The customer who offered up the suggested change promptly withdrew it.
What appeared to be simple changes to just a few requirements, we found through our traces within our tool and schema that it turned out to be a significant change afterall. Had we not had those traces, up, down, and across, and a good CM plan, we probably would have experienced $65M cost and 14 month schedule hickeys because we would have not identified all the impacts.
Those traces (and just plain old good SE) saved us. I see few projects today that use their requirements management tools properly, putting the investment into a complete and comprehensive schema for the tool, to have similar experiences as we did.
Any project that I’ve worked so far is dwarfed by what Mark considers ‘medium sized’, but I’m having very similar experiences, albeit downsized. The only difference was that my projects had no proper requirements management in place and, because of it, were actually behind the schedule and budged in a painful way. What I liked about this example is the illustration of the fact that for a complex system in development it is extremely hard to understand consequences of changing any single prior decision unless you conducted a rigorous analysis. And this analysis is only possible if there’s actual requirements and decisions database in place.
Furthermore, Benjamin Carlyle gave a thoughtful summary of what system engineering in general, and traceability in particular are about:
I would suggest that nearly all formal aspects of the system lifecycle are optional. You can choose to do or not to do any of these things. Just about everything we do in systems engineering is about containing risk and improving performance.
Instead of starting with a brief of “what’s the ROI on traceability, so I can decide whether or not to apply it to my project?” the brief should be “how risky is this project, so I can decide which SE tools to apply?”. Most formal SEs will work on a series of risky projects where clear communication, effective governance, and risk mitigation are critical, so most projects will employ many SE tools.
Traceability answers three questions:
* (top down) Have we answered every requirement,
* (bottom up) Are we building what is needed, something more, or something less, and
* (on change) What is the impact of this change?If the risk-weighted cost of getting the answers to these questions wrong outweighs the cost of doing the traceability for a given project then you’ll want to do the traceability, because on average you’ll make/save money by doing so. You could quantify this for a particular project by putting estimates on the cost for each of these questions, and the associated risk both with and without traceability.
The check-list Benjamin put together in his comment in my view is a really good set of questions to anyone involved in organizing project or programme of any noticeable size and complexity.
And, finally, Mark Powell again shared a fantastic insight on the essence of requirements from an unusual perspective:
We use requirements (contractual requirements language) in the practice of business to document decisions. The requirement is not the decision itself. Many SE’s are uncomfortable with this concept because it means they have to be accountable for the requirements.
Just a whole lot of folks in SE seem to think that the requirements are just out there floating in the ether, we just have to find them. A lot of folks in SE seem to think that the customer has all the requirements, we just have to get better in our elicitation skills to help the customer identify all the requirements. Neither notion could be further from the truth, and both are an abdication of SE responsibility.
In the practice of SE, we use requirements to document the decisions made on what problem we are going to solve, and to document the decisions on what solution we have chosen to solve it. SE is accountable for all of those decisions, and how well they are justified.
This is really a gem for me. The misconception that Mark talks about is very common. I believe, it’s a sign of great engineering discipline in the organization if there is a culture of thinking about requirements as important engineering deliverable that people have to be held accountable for.
In fact, reading this discussion alone gave me enough urge to revisit the way requirements are managed in my current projects. It’s a shame that financial sector is seriously lagging behind the curve of adoption for SE practices, especially in the times of economic downturn where everybody seems to be focusing on efficiency.
Enterprise Systems Engineering 