Software Testing

eXtreme Testing Is Elementary, My Dear Watson.

Getting started with Xtreme testing
Be not astonished at new ideas; for it is well known to you that a thing
does not therefore cease to be true because it is not accepted by many.
Baruch Spinoza (1632-1677)

The following recommendations highlight the main ideas which black/gray box
tester needs to know and implement during transition to Extreme Testing.
I presume that you have already read an article or a book on Extreme
Programming (XP).
If not, you can read about it on the Web at www.xprogramming.com and in
Extreme Programming Explained by Kent Beck. Please pay attention on the main
two practices of XP "write tests first" and "pair programming."
Excellent abstract about XP you can find in the article " Programming at the
Edge of Chaos" by Brian Foote and Laura Hill. They say about designers of XP:
"They simply had to remove a few roadblocks, and allow it to unfold in front
of them. They didn't as much have to craft it, as make sure they got out of
its way. They simply had to observe what worked, and what didn't, and take
pieces of what worked, and fit them together. The brilliance was in simply
having had the boldness and brashness to do so. This is far from a simple thing."
The concept of XP is the brightest set of guidelines in software development.
Xtreme testing must follow the same rules as XP (when these rules make sense from
testing point of view) and take the best from research and practical experience.
The concept of Xtreme testing must be based on the fact that there is no single
approach that will work for all companies. Nor should it expect that all the
pieces from one methodology are the best of the breed. The right mix of good
practices will differ for each individual project or company. But some of them
(like write test first) come along with any methodology. I prefer "design test
first". Good test must be designed like a good software.
Xtreme testing must eliminate, as much as possible, the unnecessary elements of
building software. Remember: a relatively small increase in methodology size or
density adds a relatively large amount to the project cost. XP is the less
complicated in implementation methodology comparing to others. It shows
better results if you ask for return on investment (ROI). Any increase
in methodology has to show some proven ROI before company even begins to
consider it. You have to create a set of guidelines for implementing
Xtreme testing. Some of these guidelines may reflect other methodologies.

Recommendation for implementing Xtreme Testing

Remember an old Chinese proverb:
"There are many mountains up to heaven and many pathways up each mountain."

My opinion on this topic is little different from common.
1. Define what is considered to be Quality for your system and then begin
to build Xtreme testing process to assure you meet the standard you set.
My suggestion is that quality for Xtreme testing project should be defined
as achieving the target, increasing the testability and minimizing the
variability of the system under the test. The fact that minimizing the
variability of the system under the test increases its quality has been
proved long ago. Designing unit tests first forces you to build modular,
unit testable objects (see the vision page) and is only the first step
for minimizing the system variability and increasing testability. Read
Taguchi if you would like to find more about minimizing the variability
of the system under the test. It is possible to minimize variability of
the system during requirements defining; architectural design, GUI design
and coding . Minimizing variability during test cases design helps to save
time in a future maintenance of the test suite.
Remember that:
* Xtreme testing is a major source of quality feedback
* Xtreme testing is a key to customer satisfaction.
2. XP suggests to works in pairs. Are two brains better than one? Sure.
However, is productivity increased by two? Instead of working in pairs
as XP suggests for developers, use rotation team (even for this activity
you need to schedule additional time for switching) where a less experienced
tester starts the cycle. Rotation can be based for example on the iteration
cycles.
3. Xtreme testing embodies communication with focus on simplicity: use
person-to-person communication instead of written documents wherever possible.
Cooperation between developers and testers through all development life
cycle is essential.
One of the rules is: Add document, or process only when you really need them,
not in anticipation of need.
If you still have to create a document - it should contain essentials only.
* Are you afraid that a tester will leave in the middle of the project?
Schedule an additional time for him to prepare an extensive documentation.
4. In fact extreme programming/testing provides better risk management than
many other methods.

Let us begin Xtreme testing (set of guidelines):
"The whole is more than the sum of its parts."
Aristotle, in Metaphysics (384 - 322 B.C.)

* Define the target.
* Define the minimum necessary documentation for testing that must
be created.
* Design test first. ( Try to specify as many test cases as possible
before the code is written.)
* Let developers write unit tests according to XP. Define 100% line
coverage of code for unit testing as a must. This means only that developers'
work is finished, not that code is bugs free.
* Try to eliminate redundancy of TC for unit and system testing.
* Try to avoid frequent builds for bugs fixes and changes after promotion.
An every night build is the good practice only during development process.
Try to avoid (as much as you can) supporting different versions of applications
in production..
* Try to cover with test cases all layers of the system. (define TC related to
the system architecture and environment)
* Try to make all testing processes more robust.
* Allow working together developers and testers (deep integration.)
I mean here not pairs of developers or testers but developers and testers
working in one group.

And try to make some innovations like:
* Using a rotation team.
* Implementing a gray box testing.
* Participating of testers on all architecture related meetings and walkthrough.
* Developers reviewing TC running by testers and vice versa.
* Testers involving in the business analysis and requirements review. This can
help to increase testability and minimize the variability of the system under test.
* Use UML model and diagrams for defining TC: This is not the spirit of XP, but
from my experience it helps to create better TC. Unfortunately, special
diagrams and notations for testers do not exist yet. I believe that easily
understandable graphical TC is the only right way for improvement
the testing process.
Class diagram for unit test cases
Sequence diagram for integration test cases
Use Case and Activity diagram for system test cases
* Use orthogonal array technique to reduce the number of TC.
* Next step will be creating a built-in diagnostic for any of:
External system
Third party libraries
Database
Configuration parameters
Built-in verification and calculation of a control example for the
end-to-end scenario - is a good decision for some systems.

Allow developers to concentrate on:
* Developing new and running existing unit test cases.
* Applying analysis tools such as code coverage directly into your JUnit tests
* Generating automatically unit tests from class definitions and UML models.

Allow tester (i.e. black box tester) to concentrate on:
* Integration testing of the system.
* Business cycle testing (system behavior in certain time frames).
* Concurrency testing.
* Integration testing of any third party components.
* Specification based testing, scenario testing.
* Database testing using CRUD matrix. (gray box testing method)
* Other testing types like: load, performance, installation, volume, user
documentation, disaster recovery etc.

Conclusion:
"Trust, but verify" (the old Russian proverb).
Ronald Reagan, speaking to Mikhail Gorbachev.

We know that any good idea potentially can be misused.
If we ruled out ideas because of their potential misuse, we wouldn't advance
anywhere. Many different development and testing models have been proposed
in books, journals and on conferences, each one having its own assumptions.
The concept of XP is the brightest of them. My experience shows that
"design test first" practice not only helps to create better software
but also allows manage changes quickly. So let us be careful, trust in
Xtreme testing and believe that competence and attitude of individual testers
are the most important components of a successful project.