The graph shows most of the effort in large system development goes into removing defects. Defects can occur in the feasibility, requirements, analysis, design, and coding phases of a project. The sooner a defect is detected in the development life cycle the less expensive it is to correct. So, defects detected in requirements are relatively inexpensive to correct whereas a defect in code is significantly more expensive to correct. Once a system is in operation defects are very expensive to correct. Think about it: not only do you have to fix the code but potentially you fix documentation, retrain users, etc.

So this all begs the question – where do you put your effort and resources during the development of a large system like an MMIS? Focus on quality early (rather than waiting for the testing phase of a project) saves money.  A modest investment in quality assurance or independent validation and verification up front results in real dollar savings down the road.

Takes over for experience. Vendors will try and bury you in details about how rigorous their process is so you’ll hopefully overlook the fact they are staffing your project with individuals that have barely used software let alone written it, performed quality assurance on it, or managed it’s development.   The truth is effective and efficient IV&V is art as much as science.  You need qualified consultants as much as rigorous process.  Make sure you examine who will staff your project very closely, asked to see the results of each individual’s previous IV&V efforts.

Is used as marketing hype.  Vendors will try and convince you they are “scientists” or “engineers” performing science or engineering on your projects/software. After all, V&V was pioneered by NASA and, for software IV&V, the base standard is defined by the IEEE (Rocket Scientists and Engineers) so who better to perform it?  IV&V is, conceptually, pretty simple:

• Identify the products and processes to undergo verification and validation (preferably before or in the early stages of development),
• Determine the criteria with which each of those products and processes can be evaluated (starting with standards where available e.g. the IEEE-1012-2004 standard for software verification and validation),
• Assess the products/processes while in production and upon completion to verify they meet the predefined criteria and note where they don’t,
• Re-assess products/processes when deficiencies have been addressed.

IV&V is hard work that takes expertise (there are many nuances in developing criteria and evaluating products etc), independence, and yes a degree of rigor.  But it’s not “Rocket Science”.  We’ve seen a significant amount of rigor added to an IV&V process that adds little value to the client and seems to be there to convince the client that IV&V is engineering and they are hiring the best “engineer”.  There is no professional certification of IV&V “engineers” like there is for electrical or civil engineers.  Don’t buy into the hype that this line of reasoning purports.  Understand you need competent, experienced professionals but this is not a mystical art.

Is used as an excuse for lack of social and political savvy of vendor staff. Beyond competence in performing IV&V your consultants need communications skills and an understanding of the political landscape public sector projects face.  We’ve actually witnessed an IV&V vendor reporting to a state legislature stating strictly facts (“the project is 2.3 months behind schedule” and “523 out of 1232 test cases failed”) without explaining context (“critical features were added” and “we’re only a quarter of the way through testing”).  The poor agency receiving the “benefit” of IV&V spent the next month trying to keep funding for what was a thoroughly successful project.  When called on the carpet about this the IV&V vendor explained that their rigorous methodology required they describe only the facts not the context of those facts.  Just what you need, Joe Friday (“Just the facts ma’am”).

Creates unnecessary bureaucracy that just wastes your money. Sign-offs of intermediate IV&V products, large review meetings, and repeated reviews of materials provide you with little value.  Often this “rigor” is applied not for your benefit but for the consultants, to ensure their rear is covered (“but you approved the interim deliverable”) or to enhance consultant cash flow (“you approved the first interim IV&V checklist so we can bill you”).  This unnecessary bureaucracy simply costs you time and money.

IV&V helps your software projects succeed and should be rigorous – meaning thorough, focused, and disciplined.  However, be aware “rigor” can be used for the wrong reasons.  When working with your IV&V consultant or the next time you’re reviewing an IV&V proposal keep in mind excess “rigor” provides little if any value; is used to cover a vendor’s weaknesses or benefits the vendor not you; and unnecessarily raises the costs of IV&V services.

We have seen many “IV&V” contractors play a game with Independence.  They avoid procurement conflict of interest rules by partnering with a systems integration (SI) vendor in one State on the construction or implementation of a project and bid in another State to provide IV&V over the same SI vendor.  Technically they aren’t working for the systems integration contractor in your State but in the larger picture they do receive money from the contractor they are charged with overseeing on your behalf.  This, obviously, negates the independence they profess to bring to your project.

IV&V vendors do need to remain technically smart about system integrators, system developers, software solution vendors, and fiscal agents offerings so we’re not suggesting there be no communication.  Just that no financial relationships exists between your IV&V vendor and these entities.

So what do you need? You need a consultant who does not contract with or have an obligation to system integrators, system developers, software solution vendors, and fiscal agents that will likely bid on your projects.   You need language in your IV&V procurements that guarantees this independence.