Displaying topic: "Quality Assurance and IV&V"

January 5, 2012

Scope Creep on Systems Projects

According to research conducted by Capers Jones the average amount that a software project’s scope expands from the time when the specifications are approved to the time when the system is delivered is:

  • 10% to 15% for small systems, 
  • 30% to 35 % for large systems.

This “scope creep” or expansion of the functionality usually occurs because either:

  • The systems users are able to persuade the systems developers to add various “bells and whistles” enhancements, or
  • Parts of the specifications were vague in the first place, and full exploration of their implications causes significant additional work to be added project.

The solution to scope creep has three parts:

  • Develop sufficiently complete and detailed specifications in the first place, to remove vagueness, ambiguities and room for misinterpretation,
  • Ensure that the customers (users of the future system) participate meaningfully in development of the specifications, and fully understand them and their implications, and
  • Carefully manage the change control process.

We will discuss what constitutes “sufficiently complete” specifications, meaningful participation, and management of the change control process in future posts.

September 26, 2011

Forgotten Dimensions of Risk

Earlier we discussed the difference between risks and issues.  In this post we’re going to talk about four aspects of risk you want to make sure and consider when dealing with risks.  We find these often missed in discussions about risk.  All of these help you determine the materiality of the risk and will help you prioritize the effort you put into responding to and managing the risk (more about managing risks in a later post).  The aspects to think about and capture are:

  • Impact: How big is the potential impact on the project?  We’ve seen this measured in a vague terms as “High, Medium, or Low” or as precise as a dollar amount or specific time the project will be delayed.  In our minds the more precise the estimate the better.  This is not necessarily because the more precise measure is more accurate but because the thought process required to come up with a precise estimate will help better define the risk.
  • Likelihood: What chance is there that this risk will turn into an issue for the project?  Are the “odds” high?  As with Impact, the more precise measurement the better, to a degree.  While high, medium, or low may be a bit light, 27.56% is an estimate that gives an illusion of precision.  Likelihood is predicting the future and false precision will give the impression you have a better handle on risk than you ever could.  In general we find risk expressed as a fraction with 4 or 5 in the denominator to be sufficient.  So for example theres a 1 in 4 (1/4) chance this risk will occur or a 4 in 5 chance.  This also provides sufficient detail should you choose to build a mathematical model for risk prioritization (more on mathematical models of risk in another post).
  • Immediacy: How soon will the risk occur?  Projects are often about prioritizing what’s important today.  Having a handle on when you think a risk will occur will help you prioritize when to start worrying about them.  This is the most difficult of the aspects to estimate and is the least precise.  Generally we think in terms of immediate (defined as within the next 3 months), mid-term (defined as between 3 and 6 months), and long term (greater than 6 months).
  • Triggering/Preceding Events: What visible events will precede the risk becoming an issue or indicate the risk is becoming an issue?  Be concrete, for example, if we see Milestone X is missed or have an indication it will slip it will indicate that the risk of late product delivery has turned into an issue.  Often risks will have several triggering or preceding events.  The biggest issue our clients sometimes have here is being too vague about defining the events.

There are, of course, other aspects of risk you want to capture when performing risk analysis and management on a project (for example: risk management strategy and tactics).  We’ll be more specific in a later post, but the aspects above are the most often missed on projects and some of the most beneficial in dealing with risks. Remember overall it’s less the list of risks you come up with in the end but the process of periodically identifying and analyzing risks that is of most value to the project.

July 9, 2011

Issue or Risk?

What’s the difference between an issue and a risk?  A lot of times people use these terms synonymously when, from a quality assurance perspective, they are quite different:

  • A risk is an event that may occur on a project that will have a negative impact (for example increase the time or cost to complete a project).  Note that a risk may occur. It is not a certainty (it has a less than 100% probability of occurring).  From a theoretical standpoint it is possible to have a risk that will have a positive impact on the project (shorten the time or cost of a project) but in quality assurance we typically don’t assess risks that could have a positive impact.  Optimizing project outcomes are left to project management.
  • An issue is an event that the project will or currently faces that will/does have a negative impact.  That is, an issue has a 100% probability of occurring.

The key differentiator here is risks are something that have some likelihood (probability) of occurring in the future.  An issue is something that will occur or is occurring.  Issues “will happen or are happening” and risks “may happen”.

A concrete example:

  • There is a risk a project may lose staff.  We may even be able to assign a probability this risk will occur based on expert opinion or past experience.  We can build a plan to address this risk should it occur.  The plan might dictate by staff position what we will do if a staff member leaves the project.  For example if we lose the project manager we will temporarily promote the deputy project manager to manager and begin the search for a replacement.  We can also do some things to lessen the likelihood a risk will occur.  In the “loss of staff” risk example above we might offer project completion bonuses to increase staff motivation to stay with a project through completion.
  • There is an issue if a staff member leaves the project and we have to act now to replace her.

Obviously identifying risks before they become issues and having plans in place to address them is preferable to having an issue arise for which we are not prepared.  In some cases it is even possible to lower the probability a risk will occur or even completely avoid the risk.

 

May 16, 2011

Benefits of IV&V

We are often asked “what are the benefits of performing Independent Validation and Verification (IV&V) on a software project”.  Though there are many, we routinely site four to our clients:

  • Early detection. IV&V identifies both problems and opportunities for improvement to processes and work products early on, so that they corrections and improvements can be made before the consequences become major. This has the result of reducing the total amount of work—and rework, which is especially expensive—that must be performed for the project to achieve its objectives. Note that IV&V does not just focus on problems and risks, but on solutions to problems and mitigation strategies and actions for risks. Every problem, risk or opportunity finding includes practical recommendations for solving the problem, mitigating the risk, or exploiting the opportunity.
  • Improved quality. IV&V improves the quality of project work products and processes. These improvements come not only through early detection of errors, but through pro-active prevention of errors: IV&V provides checklists, templates, and other instruments to improve the quality of project work form its inception.
  • Lower cost. By detecting errors, problems, and opportunities for improvement at the earliest possible points, IV&V can significantly reduce the over-all cost of an IT project. In an often sited study by NASA (A Case Study of IV&V Return on Investment -ROI; R.A. Rogers, D. McCaugherty, F. Martin; NASA; October 2000) estimates the return on investment for IV&V services is between 1.25-1.82.⁠ Despite the age of the study the figures appear valid (or even somewhat conservative) when applied to todays IV&V results.
  • Reduced management burden. By providing proven, professional techniques to evaluate project products and processes, IV&V can reveal risks, mitigation strategies, and opportunities for improving the efficiency and effectiveness of processes, greatly reducing the stress and “fire-fighting” typically associated with project management and freeing up managers’ time to focus on more important strategic issues.

Though there are other benefits these four alone typically justify the expense of performing IV&V on a software project.

April 19, 2010

Where do you devote your efforts on MMIS systems?

Medicaid Management Information Systems (MMIS) are some of (if not the) largest and most complex systems states have to deal with. We’ve been around long enough to know they need to be replaced every 10 to 15 years and are costly to develop. What most states don’t know is when developing systems of this size the bulk of the work goes into removing defects, problems that keep the system from functioning as it should. The graph below is based on information from Capers Jones (from his book “Applied Software Measurement”) for MMIS sized systems.

The graph shows most of the effort in large system development goes into removing defects. Defects can occur in the feasibility, requirements, analysis, design, and coding phases of a project. The sooner a defect is detected in the development life cycle the less expensive it is to correct. So, defects detected in requirements are relatively inexpensive to correct whereas a defect in code is significantly more expensive to correct. Once a system is in operation defects are very expensive to correct. Think about it: not only do you have to fix the code but potentially you fix documentation, retrain users, etc.

So this all begs the question – where do you put your effort and resources during the development of a large system like an MMIS? Focus on quality early (rather than waiting for the testing phase of a project) saves money.  A modest investment in quality assurance or independent validation and verification up front results in real dollar savings down the road.


February 21, 2010

Can Independent Verification and Validation (IV&V) be too rigorous?

Each proposal you receive for Independent Verification and Validation (IV&V) of a software project will state the vendor’s approach is “rigorous”.  Vendors fight to explain how their methodology is more “rigorous” than a competitor.  This is what you want right?  Isn’t rigor a good thing?  Maybe, but not when rigor:

Takes over for experience. Vendors will try and bury you in details about how rigorous their process is so you’ll hopefully overlook the fact they are staffing your project with individuals that have barely used software let alone written it, performed quality assurance on it, or managed it’s development.   The truth is effective and efficient IV&V is art as much as science.  You need qualified consultants as much as rigorous process.  Make sure you examine who will staff your project very closely, asked to see the results of each individual’s previous IV&V efforts.

Is used as marketing hype.  Vendors will try and convince you they are “scientists” or “engineers” performing science or engineering on your projects/software. After all, V&V was pioneered by NASA and, for software IV&V, the base standard is defined by the IEEE (Rocket Scientists and Engineers) so who better to perform it?  IV&V is, conceptually, pretty simple:

  • Identify the products and processes to undergo verification and validation (preferably before or in the early stages of development),
  • Determine the criteria with which each of those products and processes can be evaluated (starting with standards where available e.g. the IEEE-1012-2004 standard for software verification and validation),
  • Assess the products/processes while in production and upon completion to verify they meet the predefined criteria and note where they don’t,
  • Re-assess products/processes when deficiencies have been addressed.

IV&V is hard work that takes expertise (there are many nuances in developing criteria and evaluating products etc), independence, and yes a degree of rigor.  But it’s not “Rocket Science”.  We’ve seen a significant amount of rigor added to an IV&V process that adds little value to the client and seems to be there to convince the client that IV&V is engineering and they are hiring the best “engineer”.  There is no professional certification of IV&V “engineers” like there is for electrical or civil engineers.  Don’t buy into the hype that this line of reasoning purports.  Understand you need competent, experienced professionals but this is not a mystical art.

Is used as an excuse for lack of social and political savvy of vendor staff. Beyond competence in performing IV&V your consultants need communications skills and an understanding of the political landscape public sector projects face.  We’ve actually witnessed an IV&V vendor reporting to a state legislature stating strictly facts (“the project is 2.3 months behind schedule” and “523 out of 1232 test cases failed”) without explaining context (“critical features were added” and “we’re only a quarter of the way through testing”).  The poor agency receiving the “benefit” of IV&V spent the next month trying to keep funding for what was a thoroughly successful project.  When called on the carpet about this the IV&V vendor explained that their rigorous methodology required they describe only the facts not the context of those facts.  Just what you need, Joe Friday (“Just the facts ma’am”).

Creates unnecessary bureaucracy that just wastes your money. Sign-offs of intermediate IV&V products, large review meetings, and repeated reviews of materials provide you with little value.  Often this “rigor” is applied not for your benefit but for the consultants, to ensure their rear is covered (“but you approved the interim deliverable”) or to enhance consultant cash flow (“you approved the first interim IV&V checklist so we can bill you”).  This unnecessary bureaucracy simply costs you time and money.

IV&V helps your software projects succeed and should be rigorous – meaning thorough, focused, and disciplined.  However, be aware “rigor” can be used for the wrong reasons.  When working with your IV&V consultant or the next time you’re reviewing an IV&V proposal keep in mind excess “rigor” provides little if any value; is used to cover a vendor’s weaknesses or benefits the vendor not you; and unnecessarily raises the costs of IV&V services.

January 13, 2010

Playing Fast and Loose with Independence on IV&V Projects.

More States are recognizing the value of having an unbiased, outside, opinion expressed about their systems development projects.  Independent Verification and Validation (IV&V – for more information reference the IEEE-1012-2004 specification) is a standardized approach to providing these opinions that States are beginning to request contractor’s use. A key tenet of IV&V is “Independence”.  That is the ability for your IV&V contractor to give you an opinion about your systems project without being biased or unduly influenced.

We have seen many “IV&V” contractors play a game with Independence.  They avoid procurement conflict of interest rules by partnering with a systems integration (SI) vendor in one State on the construction or implementation of a project and bid in another State to provide IV&V over the same SI vendor.  Technically they aren’t working for the systems integration contractor in your State but in the larger picture they do receive money from the contractor they are charged with overseeing on your behalf.  This, obviously, negates the independence they profess to bring to your project.

IV&V vendors do need to remain technically smart about system integrators, system developers, software solution vendors, and fiscal agents offerings so we’re not suggesting there be no communication.  Just that no financial relationships exists between your IV&V vendor and these entities.

So what do you need? You need a consultant who does not contract with or have an obligation to system integrators, system developers, software solution vendors, and fiscal agents that will likely bid on your projects.   You need language in your IV&V procurements that guarantees this independence.