Displaying topic: "Managing Your Consultant"

February 21, 2010

Can Independent Verification and Validation (IV&V) be too rigorous?

Each proposal you receive for Independent Verification and Validation (IV&V) of a software project will state the vendor’s approach is “rigorous”.  Vendors fight to explain how their methodology is more “rigorous” than a competitor.  This is what you want right?  Isn’t rigor a good thing?  Maybe, but not when rigor:

Takes over for experience. Vendors will try and bury you in details about how rigorous their process is so you’ll hopefully overlook the fact they are staffing your project with individuals that have barely used software let alone written it, performed quality assurance on it, or managed it’s development.   The truth is effective and efficient IV&V is art as much as science.  You need qualified consultants as much as rigorous process.  Make sure you examine who will staff your project very closely, asked to see the results of each individual’s previous IV&V efforts.

Is used as marketing hype.  Vendors will try and convince you they are “scientists” or “engineers” performing science or engineering on your projects/software. After all, V&V was pioneered by NASA and, for software IV&V, the base standard is defined by the IEEE (Rocket Scientists and Engineers) so who better to perform it?  IV&V is, conceptually, pretty simple:

  • Identify the products and processes to undergo verification and validation (preferably before or in the early stages of development),
  • Determine the criteria with which each of those products and processes can be evaluated (starting with standards where available e.g. the IEEE-1012-2004 standard for software verification and validation),
  • Assess the products/processes while in production and upon completion to verify they meet the predefined criteria and note where they don’t,
  • Re-assess products/processes when deficiencies have been addressed.

IV&V is hard work that takes expertise (there are many nuances in developing criteria and evaluating products etc), independence, and yes a degree of rigor.  But it’s not “Rocket Science”.  We’ve seen a significant amount of rigor added to an IV&V process that adds little value to the client and seems to be there to convince the client that IV&V is engineering and they are hiring the best “engineer”.  There is no professional certification of IV&V “engineers” like there is for electrical or civil engineers.  Don’t buy into the hype that this line of reasoning purports.  Understand you need competent, experienced professionals but this is not a mystical art.

Is used as an excuse for lack of social and political savvy of vendor staff. Beyond competence in performing IV&V your consultants need communications skills and an understanding of the political landscape public sector projects face.  We’ve actually witnessed an IV&V vendor reporting to a state legislature stating strictly facts (“the project is 2.3 months behind schedule” and “523 out of 1232 test cases failed”) without explaining context (“critical features were added” and “we’re only a quarter of the way through testing”).  The poor agency receiving the “benefit” of IV&V spent the next month trying to keep funding for what was a thoroughly successful project.  When called on the carpet about this the IV&V vendor explained that their rigorous methodology required they describe only the facts not the context of those facts.  Just what you need, Joe Friday (“Just the facts ma’am”).

Creates unnecessary bureaucracy that just wastes your money. Sign-offs of intermediate IV&V products, large review meetings, and repeated reviews of materials provide you with little value.  Often this “rigor” is applied not for your benefit but for the consultants, to ensure their rear is covered (“but you approved the interim deliverable”) or to enhance consultant cash flow (“you approved the first interim IV&V checklist so we can bill you”).  This unnecessary bureaucracy simply costs you time and money.

IV&V helps your software projects succeed and should be rigorous – meaning thorough, focused, and disciplined.  However, be aware “rigor” can be used for the wrong reasons.  When working with your IV&V consultant or the next time you’re reviewing an IV&V proposal keep in mind excess “rigor” provides little if any value; is used to cover a vendor’s weaknesses or benefits the vendor not you; and unnecessarily raises the costs of IV&V services.

January 13, 2010

Playing Fast and Loose with Independence on IV&V Projects.

More States are recognizing the value of having an unbiased, outside, opinion expressed about their systems development projects.  Independent Verification and Validation (IV&V – for more information reference the IEEE-1012-2004 specification) is a standardized approach to providing these opinions that States are beginning to request contractor’s use. A key tenet of IV&V is “Independence”.  That is the ability for your IV&V contractor to give you an opinion about your systems project without being biased or unduly influenced.

We have seen many “IV&V” contractors play a game with Independence.  They avoid procurement conflict of interest rules by partnering with a systems integration (SI) vendor in one State on the construction or implementation of a project and bid in another State to provide IV&V over the same SI vendor.  Technically they aren’t working for the systems integration contractor in your State but in the larger picture they do receive money from the contractor they are charged with overseeing on your behalf.  This, obviously, negates the independence they profess to bring to your project.

IV&V vendors do need to remain technically smart about system integrators, system developers, software solution vendors, and fiscal agents offerings so we’re not suggesting there be no communication.  Just that no financial relationships exists between your IV&V vendor and these entities.

So what do you need? You need a consultant who does not contract with or have an obligation to system integrators, system developers, software solution vendors, and fiscal agents that will likely bid on your projects.   You need language in your IV&V procurements that guarantees this independence.

December 9, 2009

Too many project risks?

Tired of getting reports from your quality assurance (QA) consultant that are just a long laundry list of all the things that might go wrong with your project? We’ve seen 20 page reports with over 200 “risks” listed.  The likelihood of many of those risks occurring is about the same as the likelihood of getting hit by a meteor while standing in your front yard.  Some are so immaterial that even if they happen they won’t affect the successful outcome of your project.  The long list of mostly irrelevant items overwhelms clients and erodes confidence in the QA process.  Usually the laundry list just  ends up getting ignored.

QA consultants often don’t understand their role or feel their role is simply to point out all possible risks.  Sometimes they are on autopilot just working to a checklist.  We’ve even seen QA consultants invested in a project failing because it validates all the things they’ve said in their reports. The real job of a QA contractor is to make your project successful. They should do this, in part, by advising you of the things that matter.

The first page of a QA report should give you the top three to five risks you need to attend to immediately.  There should be a strong rationale for choosing the “front page” risks (usually magnitude of impact, likelihood of occurrence, and immediacy of risk are the factors used).  The report should also provide pragmatic suggestions for managing or mitigating the top risks.  Demand the top issues are brought to your attention in a concise and useful manner.  Focus on resolving the most important things before you even look at the other 195 risks.

November 30, 2009

Setting expectations

Is your consultant having a difficult time meeting your expectations? Sometimes from a clients’ perspective it appears consultants are unresponsive to their needs and miss their expectations.  Typically, consultants focus on “deliverables”. These are usually written reports that when delivered to a client trigger a payment. At times consultants do this to the exclusion of resolving the problem the client sees as critical. This is done because consultants:

  • Get overly focused on the product they produce rather than the problem they are trying to solve; and
  • Want to make sure they meet their contractual obligations, typically outlined in a formal “statement of work”.

You focus on solving “the problem”.   However, sometimes that problem isn’t well defined or expectations about its resolution are inadequately communicated.  Your up front expectations for the consultants work are not typically discussed “up-front”. Further, as your project moves forward and more is learned, the original problem defined in the contractual “statement of work” may not really be the problem you need to solve. This leads to a situation where:

  • Consultants are not meeting the your expectations;
  • The problem being tackled isn’t clearly understood or worse is the wrong problem to solve;
  • Activities starts to vary from the contractual statement of work; and
  • Tensions between you and your consultant rise degrading the cooperative atmosphere required for a successful consulting engagement.

On projects where we find ourselves varying significantly from the contractual statement of work, don’t fully comprehend your expectations, or feel we aren’t on the same page regarding the problem we are trying to solve we will develop a “Deliverable Expectation Document” (DED). The intent of this document is to clearly define key expectations around the “deliverable” that results from our work.  The DED is a short statement that outlines the problem(s) we are trying to solve, what we think the end product of the task will look like (both the form and outline of the content), who the audience is for the end product, and where we are varying from the original statement of work.

The DED should be short (think a page or two) and concise. It must be developed as a collaborative effort between client and consultant. The DEDs are developed iteratively throughout a project just before work begins on a major task.  Over the course of a large project with many tasks you might develop many DEDs.  The process begins by imagining that a specific deliverable is complete and asking “what is different now than before we started.”  Most of the real value comes not from the document itself but from the focused collaboration between consultant and client required to develop it.

There are pitfalls to avoid and some downsides. Some consultants and clients want to specify the outline and format of the end product more rigidly or in greater depth up front than can practically be done. It is a mistake to turn the DEDs into the deliverables themselves. Creating a DED is time consuming requiring key staff to find the time to get together in person.  At other times DEDs are simply not necessary, for example, when you and your consultant are clearly in agreement on what needs to be done or when your consultant is really a “body for hire” (you direct all their activities).

We don’t suggest the use of DEDs on every project or for every deliverable. However, we have found them a valuable tool to avoid miscommunication about content, get the project focused on the problem that needs to be solved, and addresses concerns associated with variance from a contractual statement of work by providing a clear audit trail of decisions about scope . When appropriately applied they can help communicate expectations, save time, result in higher value products being delivered to you, and help maintain a collaborative productive relationship between you and your consultant.