Testing the Waters with a Proof of Concept – Part Two
Part Two – Proof of Concept in Vendor Selection
As discussed in a previous post, a Proof of Concept (POC) is equally valid in vendor
selection and as a implementation milestone. In this post, we look at the POC as part of the vendor evaluation. Conducting a POC in this scenario will allow the customer to:
- Evaluate key functional and usability requirements of a prospective system (with the understanding that some functionality might not be supported yet)
- Make a go/no go decision. If Go, then move to implementation, if NO GO, conduct a POC with an alternative vendor
- Evaluate the “fit” with the vendor, their responsiveness and understanding of the customer’s business prior to committing to purchase and implementation
What it is (and what it isn’t)
- It is a basis of vendor selection
- It is not part of the implementation
- It is conducted over a short period of time on key scenarios, typically the ones that have been highlighted as being of most concern for the business
- It should not require a full-blown configured system
- Should be a paid trial for the vendor with fees being deducted from the final price should the customer decide to go with the vendor
Prerequisites for Conducting a POC as Part of a Vendor Selection:
- There must be a clear understanding and communication between the customer and vendor that the POC is part of vendor selection rather than an implementation milestone.
- It is a good idea to have clearly documented business processes to ensure that they system supports current, or in the case of a transformation project, the to-be processes.
- The functional requirements must be documented and prioritized so that they can be used for evaluation testing.
- Identification of key business users to evaluate the system functionality, usability and business process support. It is fine for the solution to be technically elegant, but if it is not suited for the purpose of the business it will not be used.
- Identification of other potential vendors. If there are no other appropriate vendors, then conduct the POC as an implementation milestone.
- The POC should last no more than 2-4 weeks.
Creating the POC Environment and Running the POC
- Document key scenarios / Test Scripts with pass/fail criteria that will be used by the key business users in their evaluation testing.
- Develop an evaluation scorecard and scoring parameters that the key business users can use to score how well the vendor’s system satisfies the functional, technical and usability requirements. If the system provides the functionality and technical specifications required, but does not facilitate the customer’s business processes and workflows because it is not usable or requires numerous workarounds, this needs to be captured and scored.
- Conduct user training on the system as well as on the use of the scenarios/test scripts and the evaluation scorecard with the key business users. This should be a joint effort between the vendor and the customer
- The vendor and/or customer need to prepare a POC environment where the key business users can test the system away from the demands of their jobs and the system is set up with a test database (and, if appropriate test data), required interfaces, etc.
- Carefully select the test data to use as part of the POC so that it is meaningful and relevant to the test users. For example, it might be appropriate to use a combination of last month’s data pre-loaded on the environment, and in addition ask the users to enter current data during the POC.
- The key business users use the scenarios/test scripts to evaluate the system and record their results using the evaluation scorecard.
Outputs of the POC
After the testing and evaluation has been completed, the customer will decide whether to select the vendor or to engage in another POC with another vendor. Because of the effort required by both the vendor and customer, a POC for vendor selection should be entered into with only one or two vendors, with the preferred vendor going first.
Hi,
A few comments…
I think the aim of a POC is in part to qualify the vendors individual suitability, but also to compare vendors against each other? The statement: “Make a go/no go decision. If Go, then move to implementation, if NO GO, conduct a POC with an alternative vendor” implies that this is a sequential process and the first vendor in sequence to get a ‘Go’ is actually the winner. Does this mean that the next short-listed vendor due to participate, who may actually in fact be a better fit, is therefore dismissed?
A POC is commonly run in the latter stages of the bid process, usually with a short-list of 2-3 as a kind of ‘beauty-contest’, but the ultimate ‘Go’ decision is still subject to final financial negotiations. POC’s are often run concurrently, to reduce the elapsed time and cost impact. It will have a higher short-term impact on the customer resource and requires careful management but can be more efficient if properly run and coordinated.
2-4 weeks for a POC seems a reasonable guideline, but this is really subject to a number of factors:
o scale of the project
o scope of the POC; I have seen some POC’s that have attempted e2e processes, but they may (hopefully) decide that the vendors are very similar in many areas and may want to focus on any identified key differentials for the vendors in the short-list and/or key functional areas (this is in line with your earlier statement of: It is conducted over a short period of time on key scenarios, typically the ones that have been highlighted as being of most concern for the business)
o customer budget
o availability of project resource the customer has to manage this
o availability of user resource to participate; in my experience this is often a key constraint, more so as companies are increasingly tightening their belts and reducing head-count (presumably, this is where you can add value by acting in support of the users)
If possible, Non-functional requirements should also be considered and assessed in some way. It may be difficult at this early stage, as vendors may not be equipped to produce metrics for scalability (and presumably the POC will not be on Production-like hardware), but if possible some effort should be put to capturing this. If you can identify this up front, you can reduce the risk of hidden costs, delays to implementation and set some expectations for the user experience. Some of this can be difficult, as there will be elements, such as specific interfaces and functional areas, etc., which will have an impact and will not be available until the implementation stage, but perhaps throughput can be demonstrated in a similar or ‘typical’ scenarios or at least provide figures to show volumes handled at other sites. Truly representative simulated user load can be tricky, but setting expectations at this point can be a useful exercise in risk reduction.
Also related to Non-functional requirements, usability and Accessibility assessments is a feature that is increasingly becoming a criteria, certainly in many EU assessments. I believe this is also the same for US and is spreading. If the customer is focused on this, a short assessment for compatibility with tools such as Dragon or Jaws can be useful at this stage.
In general, a properly managed POC reduces risk (cost/time) to the project implementation and should accelerate the process to steady-state BAU which meets customer expectations.