Explore the World of Gray Box Testing
Explore the World of Gray Box Testing?

If you ask most "Testers" a question about gray box testing, they likely will give you a strange look - especially if all they have known is black box testing. However, this should not stop you from exploring this powerful testing technique.

Software development shops in general see testing teams as a necessary evil to completing projects. There is an increase in the number of sophisticated testing teams being utilized to fulfill project demands. Testers are growing up and their skill levels are maturing. We are witnessing a shift from black box to gray box testing.

As we all well know, system defects have costs:
  • Cost to prevent the defect
  • Cost to find the defect
  • Cost to report the defect
  • Cost to fix the defect
  • Cost to distribute the fix

    For most applications that now are under development, white and black box testing was defined long ago and does not cover all the possible scenarios that must be tested. For example, how can a tester design a test case where a process on a web server is down or that may use numerous global variables that require concurrent input from multiple sources?

    By implementing gray box testing, we simply reduce the overall cost of system defects - and prevent more from passing the testing stage. Our intention is to help you explore the different testing techniques. We do not expect or hope that you implement all of the different possible gray box testing methods. Use common sense, and take on what you can handle and successfully employ.

    Black box testing and traditional waterfall methodologies are being replaced to meet the growing demand on our testing teams. CIOs are asking "ROI" questions and questioning the value of our testing teams - low skilled black box testing teams are not the wave of the future. We can improve the quality of our testing efforts through better planning. Doing things right the first time and asking ourselves key questions. Where is the value in what I am doing? Is there a better way towards building quality products? And if so, how do I get there?

    First, you need to figure out where you are- yes, rate yourself. How do you plan to get anywhere unless you know where you are, you just might be there! The Juran Theoretical Quality Model is a great place to start and will help you determine if you need to make changes.

    Juran Theoretical Quality Model

    Software Black Box Testing Black box testing also known as functional testing is a software testing technique whereby the tester does not know the internal workings of the software.

    Black box testing can best be described as placing your code in a box and creating your test cases independent of the code. This is the most common method for testers, as most of us don’t read code! One of our struggles with black box testing is the lack of documentation, which prevents us from creating effective test cases. We often get “code” dumped in our laps and are subsequently told to test.

    The following are well known black box requirements- based test case selection techniques:
  • Equivalence Classing
  • Partition Analysis
  • Cause-Effect Graphing
  • Factor/Action matrix (FAM) Technique
  • Heuristic Test Case Selection

    As previously mentioned, black box testing is a part of the testing process as a whole. It is a complementary method and excels in determining certain specific errors that can be defined in the following categories:

  • Incorrect or missing functions
  • Interface errors
  • Errors in data structures or files
  • Performance errors
  • Initialization or termination problems

    In addition, black box testing occurs toward the end of the coding process, and is therefore used to test the following:

  • The functional validity of the software.
  • The classes of errors applicable to the specific software.
  • The boundaries of error classes
  • Requirements.
  • Output.
    Software Black Box Testing is not a bad thing. This software testing technique works and has advantages and disadvantages.

    Advantages

  • Tests are unbiased because the designer and the tester are independent of each other
  • Tester does not need programming knowledge
  • Tests are done from the point of view of the client, not the designer/coder
  • Test cases can be designed as soon as the specifications are complete

    Disadvantages

  • Tests can be redundant if the coder has already run the test cases
  • Test cases are difficult to design - very difficult when coding is done ahead of specifications/design documentation
  • Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested

    Software White Box Testing (synonym glass-box).

    Testing is done under a structural testing strategy and requires complete access to the object's structure, that is, the source code. [B. Beizer, 1995 p8], The following are well known white box, code-based test case selection techniques:

  • Statement/Path Coverage
  • Data Flow Coverage
  • Domain Testing

    Software white box testing technique is most commonly used by individuals who can read and write code. White box testing has many advantages such as increased testing coverage and reduced duplicate testing. The down side is the loss of focus of the testing team. For example, is the software user friendly and does it meet the business requirements? White box testers are so focussed on the code that they lose their objectivity. The other disadvantage is the cost of white box testing which tends to be higher due to the increased skill levels.

    Why We Need Gray Box Testing

    Have you heard the expression "It doesn’t matter whether a cat is black or white so long as it catches mice"? The same holds true for testing. gray box testing will increase our testing coverage by allowing you to focus on all of the layers of any complex system through the combination of all existing white and black box techniques.

    Software Gray Box Testing

    Cem Kanner defines gray box testing as involving inputs and outputs, but test design is educated by information about the code or the program operation of a kind that would normally be out of view of the tester. Gray box testing can be seen as the blending of structural and functional testing methods throughout the entire testing procedure.

    Gray-box testing examines the activity of back-end components during test case execution. There are two types of problems that can be encountered during gray-box testing. The first is when a component encounters a failure of some kind, causing the operation to be aborted. For example, an edit check to allow dollars does not accept dollar amounts, i.e. "AAA". The second is when the test executes in full, but the content of the results is incorrect. Example: calculations - produces a number but it is incorrect.


    We will introduce a sample of gray box testing techniques below in order to provide you with the general overview.
  • Matrix Testing
  • Strategy for Gray box Regression testing
  • Pattern Testing
  • Orthogonal Array Testing

    Matrix Testing

    The idea of beginning your testing activities with a list of variables used in the software is not new. You may have heard the term CRUD (created, read, update and deleted) method. Basically it starts with developers defining all the variables that exist in their programs. Each variable will have an inherent technical risk, business risk and can be used with different frequency during its’ life cycle. All of this information is summarized in tables 1 and 2, which then drives the design of test cases as illustrated in the following example.

    Example: Client name and address input-recall screen

    Client name and address input-recall table

    Further information is extracted to complete the testing matrix and is summarized in the table 2 for the example in table 1.

    Risk Matrix

    From the chart above, a testing analyst can quickly ascertain that the business and technical aspect of the code, deleting and saving records requires testing. It would not be uncommon for a black box tester to concentrate his or her efforts solely on updating record functionality; hence, little return from their effort - more system defects/higher cost down the road.


    Software Regression Testing

    Regression testing -testing that is performed after making a functional improvement or repair to the program. Its purpose is to determine if the change has regressed other aspects of the program [Glenford J.Myers, 1979]. This can be accomplished by executing the following testing strategies.

  • Retest all: Rerun all existing test cases
  • Retest Risky Use Cases: Choose baseline tests to rerun by risk
  • Retest By Profile: Choose baseline tests to rerun by allocating time in proportion to operational profile
  • Retest Changed Segment: Choose baseline tests to rerun by comparing code changes. (White box strategy)
  • Retest within Firewall: Choose baseline tests to rerun by analyzing dependencies. (White box strategy)

    As a gray box tester, you can mix all the above strategies. You may want to use the following percentage breakdown for regression testing.

    Running existing test cases- 80% of allowed time (automated testing really helps here). Do not forget to create matrix tables as above to choose priority of test cases during execution. Exploratory Testing - 20% of allowed time

    Pattern Testing

    Pattern testing is best accomplished when historical data of previous system defects are analyzed. Your analysis will include specific reasons for the defect, which will require system analysis. Unlike black box testing where you are tracking the type of failures that are occurring, gray box testing digs within the code and determines why the failure happened. This information is extremely valuable, as future design of test cases will be proactive in finding the other failures before they hit production. Remember, having coding structure in place influences gray box test case design. Therefore, take advantage of powerful technique and design test cases that your developers and black box testers would not cover.

    Analysis Template will include:
  • The problem addressed
  • Applicable situation
  • The problem that can be discovered
  • Related problems
  • Solution for developers that can be implemented
  • Generic test cases

    New test cases will include:
  • Security related test cases
  • Business related test cases
  • GUI related test cases
  • Language related test cases
  • Architecture related test cases
  • Database related test cases
  • Browser related test cases
  • Operating System related test cases

    Orthogonal Array Testing: Taguchi Method

    Orthogonal Array Testing is a statistical testing technique implemented by Taguchi. This method is extremely valuable for testing complex applications and e-comm products. The e-comm world presents interesting challenges for test case design and testing coverage. The black box testing technique will not adequately provide sufficient testing coverage. The underlining infrastructure connections between servers and legacy systems will not be understood by the black box testing team. A gray box testing team will have the necessary knowledge and combined with the power of statistical testing, an elaborate testing net can be set-up and implemented.

    The theory -Orthogonal Array Testing (OAT) can be used to reduce the number of combinations and provide maximum coverage with a minimum number of test cases. OAT is an array of values in which each column represents a variable - factor that can take a certain set of values called levels. Each row represents a test case. In OAT, the factors are combined pair-wise rather than representing all possible combinations of factors and levels.

    A simple example: Three Factors and Three Levels Array is shown in table 3.

    Three Levels Array

    Now given the example above, a tester would have to create 3³ or 27 test combinations as shown in table 4.

    Number of test cases table

    By assigning values for each factor and then extrapolating for combined pairing, our total number of test cases drops to nine from 27. This simple but effective technique maximizes the required testing coverage.

    Summary

    We have covered a number of different testing techniques that can be successfully deployed. Our intent is to broaden your scope of thinking and provide you with a starting point to move forward. Check out the experts in the biography and talk to people who share your vision of a well-balanced and highly skilled testing team. Reduce the cost of system defects and thereby increase the quality of your software.

    Bio:
    Alex Samurin is a QA Analyst in Toronto, Canada. Mr. Samurin volunteers for The Quality Assurance Institute (QAI) as well as the Toronto Association of Systems & Software Quality.

    Joseph (Joe) Larizza, is a Project and QA Manager, Banking and Investment Industry at CGI. Mr. Larizza is a member of the Board of Directors for the Toronto Association of Systems & Software Quality and volunteers for theQuality Assurance Institute. He is a Certified System Quality Analyst and holds a Bachelor of Arts Degree in Economics.

    Bibliography

    1- Nguyen, Hung Q. Testing Applications on the Web. Canada: Robert Ipsen, 2001.
    2- McGregor, John D. and Sykes, David A. Practical Guide to Testing Object-Oriented Software. Canada: Addison-Wesley, 2001.
    3- Poston, Robert M. Automating Specification-Based Software Testing. USA: IEEE Computer Society Press, 1996.
    4- Cem Kaner, James Bach, Bret Pettichord. Lessons Learned in Software Testing. John Wiley & Sons 2002.
    5- Kaner, Fank, and NguyenTesting Computer Software. USA: International Thomson Computer Press, 1998.
    6- Boris Beizer. Black-Box Testing : Techniques for Functional Testing of Software and Systems
    7- Robert V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools (The Addison-Wesley Object Technology Series). October, 1999 Addison-Wesley Pub Co
    Published by Toronto Association of Systems and Software Quality - July 2003

    search comp.software.testing Frequently Asked Questions (FAQ) for more information about software testing

    © July 2003 Alex Samurin geocities.com/xtremetesting/ © 2009 eXtremeSoftwareTesting.com
    If you have navigated to this page from another site, and you would like to go to our home page, please click:
    Extreme Software Testing