For most applications that now are under development, white and black box testing was defined long ago and does not cover all the possible scenarios that must be tested. For example, how can a tester design a test case where a process on a web server is down or that may use numerous global variables that require concurrent input from multiple sources?

By implementing gray box testing, we simply reduce the overall cost of system defects - and prevent more from passing the testing stage. Our intention is to help you explore the different testing techniques. We do not expect or hope that you implement all of the different possible gray box testing methods. Use common sense, and take on what you can handle and successfully employ.

Black box testing and traditional waterfall methodologies are being replaced to meet the growing demand on our testing teams. CIOs are asking "ROI" questions and questioning the value of our testing teams - low skilled black box testing teams are not the wave of the future. We can improve the quality of our testing efforts through better planning. Doing things right the first time and asking ourselves key questions. Where is the value in what I am doing? Is there a better way towards building quality products? And if so, how do I get there?

First, you need to figure out where you are- yes, rate yourself. How do you plan to get anywhere unless you know where you are, you just might be there! The Juran Theoretical Quality Model is a great place to start and will help you determine if you need to make changes.

Juran Theoretical Quality Model

Software Black Box Testing Black box testing also known as functional testing is a software testing technique whereby the tester does not know the internal workings of the software.

Black box testing can best be described as placing your code in a box and creating your test cases independent of the code. This is the most common method for testers, as most of us don’t read code! One of our struggles with black box testing is the lack of documentation, which prevents us from creating effective test cases. We often get “code” dumped in our laps and are subsequently told to test.

As previously mentioned, black box testing is a part of the testing process as a whole. It is a complementary method and excels in determining certain specific errors that can be defined in the following categories:

In addition, black box testing occurs toward the end of the coding process, and is therefore used to test the following:

Testing is done under a structural testing strategy and requires complete access to the object's structure, that is, the source code. [B. Beizer, 1995 p8], The following are well known white box, code-based test case selection techniques:

Software white box testing technique is most commonly used by individuals who can read and write code. White box testing has many advantages such as increased testing coverage and reduced duplicate testing. The down side is the loss of focus of the testing team. For example, is the software user friendly and does it meet the business requirements? White box testers are so focussed on the code that they lose their objectivity. The other disadvantage is the cost of white box testing which tends to be higher due to the increased skill levels.

Have you heard the expression "It doesn’t matter whether a cat is black or white so long as it catches mice"? The same holds true for testing. gray box testing will increase our testing coverage by allowing you to focus on all of the layers of any complex system through the combination of all existing white and black box techniques.

Cem Kanner defines gray box testing as involving inputs and outputs, but test design is educated by information about the code or the program operation of a kind that would normally be out of view of the tester. Gray box testing can be seen as the blending of structural and functional testing methods throughout the entire testing procedure.

Gray-box testing examines the activity of back-end components during test case execution. There are two types of problems that can be encountered during gray-box testing. The first is when a component encounters a failure of some kind, causing the operation to be aborted. For example, an edit check to allow dollars does not accept dollar amounts, i.e. "AAA". The second is when the test executes in full, but the content of the results is incorrect. Example: calculations - produces a number but it is incorrect.

The idea of beginning your testing activities with a list of variables used in the software is not new. You may have heard the term CRUD (created, read, update and deleted) method. Basically it starts with developers defining all the variables that exist in their programs. Each variable will have an inherent technical risk, business risk and can be used with different frequency during its’ life cycle. All of this information is summarized in tables 1 and 2, which then drives the design of test cases as illustrated in the following example.

From the chart above, a testing analyst can quickly ascertain that the business and technical aspect of the code, deleting and saving records requires testing. It would not be uncommon for a black box tester to concentrate his or her efforts solely on updating record functionality; hence, little return from their effort - more system defects/higher cost down the road.

Regression testing -testing that is performed after making a functional improvement or repair to the program. Its purpose is to determine if the change has regressed other aspects of the program [Glenford J.Myers, 1979]. This can be accomplished by executing the following testing strategies.

Running existing test cases- 80% of allowed time (automated testing really helps here). Do not forget to create matrix tables as above to choose priority of test cases during execution. Exploratory Testing - 20% of allowed time

Pattern testing is best accomplished when historical data of previous system defects are analyzed. Your analysis will include specific reasons for the defect, which will require system analysis. Unlike black box testing where you are tracking the type of failures that are occurring, gray box testing digs within the code and determines why the failure happened. This information is extremely valuable, as future design of test cases will be proactive in finding the other failures before they hit production. Remember, having coding structure in place influences gray box test case design. Therefore, take advantage of powerful technique and design test cases that your developers and black box testers would not cover.

Orthogonal Array Testing is a statistical testing technique implemented by Taguchi. This method is extremely valuable for testing complex applications and e-comm products. The e-comm world presents interesting challenges for test case design and testing coverage. The black box testing technique will not adequately provide sufficient testing coverage. The underlining infrastructure connections between servers and legacy systems will not be understood by the black box testing team. A gray box testing team will have the necessary knowledge and combined with the power of statistical testing, an elaborate testing net can be set-up and implemented.

The theory -Orthogonal Array Testing (OAT) can be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases. OAT is an array of values in which each column represents a variable - factor that can take a certain set of values called levels. Each row represents a test case. In OAT, the factors are combined pair-wise rather than representing all possible combinations of factors and levels.

By assigning values for each factor and then extrapolating for combined pairing, our total number of test cases drops to nine from 27. This simple but effective technique maximizes the required testing coverage.

Summary

We have covered a number of different testing techniques that can be successfully deployed. Our intent is to broaden your scope of thinking and provide you with a starting point to move forward. Check out the experts in the biography and talk to people who share your vision of a well-balanced and highly skilled testing team. Reduce the cost of system defects and thereby increase the quality of your software.

Bio:
Alex Samurin is a QA Analyst in Toronto, Canada. Mr. Samurin volunteers for The Quality Assurance Institute (QAI) as well as the Toronto Association of Systems & Software Quality.

Joseph (Joe) Larizza, is a Project and QA Manager, Banking and Investment Industry at CGI. Mr. Larizza is a member of the Board of Directors for the Toronto Association of Systems & Software Quality and volunteers for theQuality Assurance Institute. He is a Certified System Quality Analyst and holds a Bachelor of Arts Degree in Economics.