Vendor Quality - what does it mean to you and your organization? Many of us today are dealing with third parties whether overseas or just around the corner. The software solutions that are delivered to us are key to our overall businesses. How many of you have experienced delays caused by the headaches of third party code? These delays can have many root causes such as poor requirements or a lack of understanding by the third party. This article will address what we can do specifically around testing to help increase the quality of third party code and reduce unwanted delays. There are many options and opinions regarding the quality of vendor code or lack thereof. Albeit not all vendor code is of poor quality; however it does imply room for improvement and the guidelines suggested here, applied with a little common sense, can make a difference.This article will focus on structuring pre-acceptance testing, but the work done here has paybacks for the overall quality of the delivered code.
Often the contractual arrangements between the organization and the third party specify a certain level of quality that needs to be delivered. If the code does not meet the specified criteria remedies are often prescribed. Invoking these remedies is not a preferred option because they may involve time delays, haggling and sometimes a souring of the relationship. The ideal option is that the code is delivered up front with a much higher level of quality and this can be achieved by working together.
To improve the quality the key point when dealing with your vendor is partnership building. Remember, your vendor can only be successful if the relationship develops a partnership or a "win -win" scenario, fostering quality and savings for both parties. Any business relationship which does not build and continue to build a strong partnership will in the long run degrade. Quality concerns will surface and could possibly end the partnership.
Information sharing is the key component to the QA program described to target "testing effectiveness". The basic framework is as follows: 1. Create Pre-acceptance test strategy 2. Review and walk-through the testing strategy 3. Develop test cases and expected results 4. Create execution plan and execute plan 5. Review results - acceptance of criteria 6. Product delivery Create Pre-acceptance Test Strategy - Joint EffortWorking together with your business partner-vendor to decide on the best approach for validating product upgrades/code enhancements. Take advantage of each others' strengths to determine success criteria. For example, an acceptable pass rate on jointly developed testing scenarios is a way to measure success. Focus your efforts on each others' strengths and expertise. Remember pre-acceptance is your way of saying to the vendor we will not accept the code to do our internal validations until it meets certain specific standards that we jointly agreed upon. It also allows the vendor to make sure that their code can pass many of the tests that your organization will execute before they deliver the code.
Review and walk-through the testing strategy - agree to execute plan.It may appear redundant; however a detailed walk-through of the jointly developed testing strategy and expected outcomes is recommended. Both parties will bring to the table assumptions or expectations that may not be clearly understood. A thorough review will uncover potential misunderstandings.
In some cases the code delivered are adjustments to an existing third party application. The vendor may have a core application that has been customized for a wide range of clients. In devising a test strategy many high level issues must be resolved such as • What is core functionality and what are the code adjustments? • Who tests what? • How do we know that the vendor has tested their core functionality? • Is there a vendor test plan? • What are the test cases? Are they documented and available for reviewing? • How will the vendor prove that the core test cases were executed? • What test cases will the 3rd party execute to test the code adjustments? • Are these code adjustments included in regression testing? • Who decides what test cases to test the adjustments are? • Who is the final approval? Develop test cases and expected results - joint effortThe creation of independent test cases may be necessary due to logistic reasons. However, sharing the testing scenarios will reduce the number of defects. If the vendor's testing team has access to these test cases and has the ability to execute them in their testing labs, you can expect less defects. The added benefit is that the vendor's testing team will also be in a position to highlight potential requirement gaps. In general, testing teams- vendors' understanding of the product (hands on daily) will verify design or product upgrades prior to any code development.
In some cases a more complete pre-acceptance test can be negotiated. Here the vendor software must pass specified tests before the organization accepts the code. This has several advantages. • Code will be delivered to an expected standard that will allow your organization to start testing without having to deal with defects which should have been found by the vendor. • The vendor has specific criteria that must be met and therefore can work more efficiently to meet them. • The amount of politics between the two parties is decreased. Create execution plan - joint effort Test planning in advance will reduce the "blame game" when dealing with system defects. Plan defect correction cycles in advance and worst case scenarios. Don't assume the happy path for execution based on the above guidelines. A jointly developed execution plan dealing with tough issues will continue to foster a positive partnership - when the going gets tough. Issues which need to be resolved vary from general to quite specific. Here are some examples of the type of decisions that are required to be negotiated: • Where the testing is to be executed, vendor site, offshore or in your organization? • Who will conduct the testing? • Will there be representatives from both organizations? • Can testing be executed remotely? • Will the vendor perform their tests and then your organization will repeat those test and others before accepting the code? • What data will be used for testing? • On what environment will the testing be completed and who is responsible for setting this up? • What is the turn around time for correcting defects? • Is it possible to automate these tests and who should do this? • How do we determine that pre-acceptance testing has been completed and the code is now deemed ready for delivery? Test automation if possible, would be much more efficient then manual tests. If the automated tests can be built by the vendor they can then be delivered along with the code for your organization's testing requirements. In many cases the test automation delivery will become the bases of your regression testing efforts. Beware of vendors claiming that they have significant test automation and ask for proof to avoid glorified smoke tests. If the client is providing the test automation, arrange to have access to the code while the vendor is performing integration and system testing. In that way automation can be written before pre-acceptance testing and used for both vendor system testing and PAT. Executing and Reviewing results - acceptance of criteria Inspection cycles have the advantage of ensuring quality releases. If expectations are planned in advance and jointly agreed-upon as suggested here, reviewing vendor testing results prior to delivery has many benefits. Defects or design issues can be addressed in the vendor's shop where corrective measure can be taken efficiently and can potentially reduce timeline impacts. Product delivery A planned re-test of all or partial pre-acceptance test cases in your environment is advisable if product set-up or hardware cannot be duplicated in your vendor's test lab. Having a development region for your vendor inside your shop will (with the assumption that this is possible within your environment) also increase the quality of the product. This will allow the vendor to fix defects and test them using your configurations and data. A well worked out structure and good communication is now the single most important factor to increase the chance for success. Given the global nature of many vendors and the ability of 24 hour development cycles, good communication is required as you may require access to people who are asleep. Conclusion:The ideal scenario is that both parties work and communicate without barriers - few barriers as possible. The above framework addresses testing aspects of our vendor relationships. This doesn't mean that implementing this approach will resolve all of your potential quality concerns. However, if your vendor relationship is suffering, this approach can begin to build a business partnership. In general, apply common sense and design a working solution with your vendor if you are suffering from poor vendor quality.
About the Author Joe Larizza, CSQA, is the Sr Manager, Quality Assurance | Retail and Wealth Mgmt Appls | Tech & Ops | RBC Royal Bank, and previously held the positions of QA Manager for CPP Investment Board, Director, QA for Loblaw Companies Inc., and Senior Manager, QA with RBC Dexia. He also has held support management roles with International Financial Data Services and its sister companies. During his career, he has undertaken a number of strategic initiatives, including the expansion of testing programs and establishment of testing standards and procedures, implementation of a Quality Metrics program, evaluation of an IT Division against the Capability Maturity Model, and implementation of Automated Testing using the Behaviour Model and Data Driven scripts. Mr. Larizza has earned a reputation for competency and excellent leadership ability in the field of quality assurance and testing of software. Joe Larizza is President of the Board of Directors for the Toronto Association of Software Quality and volunteers for the Quality Assurance Institute of Canada. He is a Certified Quality Analyst and holds a Bachelor of Arts Degree in Economics, and the Canadian Securities Course.
2012 Toronto, Canada