Black-Box Test case Reduction Technique

Just want to share about a good technique of black box test reduction which I have implemented in few of my projects in different companies. This method is known as Input-Output Analysis.

Motivation for this Research
After the initial releases in a project, testers use to end up with a large set of test cases which is really hard to run with a daily or weekly build. An efficient QA will set the priority of all test cases in a test suite in order to manage them as per build schedules, even then it will be difficult to manage the complete test runs. Moreover additional information from the software implementation often improves and add on to the black-box test cases.
So here is a thought process to find a way to get information from the software implementation to verify our test models, before the software is released. And that’s how I started down this path for the said methodology named as Input Output Analysis.....

Observations on Black-Box Testing

• Testing from the black-box perspective generates a large number of tests which is usually difficult to run with each and every build.
• Testers don't want to repeat those tests for daily builds for which functionality is now stable.
• Mostly test teams are facing the resource limitations; hence number of tests must be reduced.
• Test reduction done randomly or based on “engineering judgment” may lead to ineffective or redundant tests.

Combinatorial Testing

• For programs with multiple inputs, one must decide how to test different combinations of selected test data values so that test coverage for a valid set of test data is quite high.
• The most thorough approach is to test all combinations - this is quite often impossible due to the large number of tests.
• However, not to consider testing different input combinations could lead to an unacceptable number of undetected software faults.

Combinatorial Explosion
If n = 10 and 10 test data values are selected for each variable, the total number of combinatorial test cases is 10^10.




Current Approaches

• Testers may try to identify “important” combinations to test using specifications, historical fault data, and engineering judgment. This technique is not a fault prof method.
• Orthogonal arrays, or other combinatorial design techniques, may be used to tests pair-wise combinations of inputs.
• But these combinatorial tests are difficult to determine impact on fault-detection capability of the test set using these techniques.

Proposed Technique Input/OutPut Analysis method
1) Current techniques for test reduction do not consider program output variables.
2) For a class of programs, not all program inputs affect all program outputs. It means that all input variables are not contributing to the required output for a functionality.
I propose the use of Input-Output (IO) Analysis to determine which program inputs affect program outputs. This information can be used to reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set.

Simple Example using IO AnalysisInput data selected using equivalence partitioning:Total Number of Combinatorial Tests: 2 * 4 * 2 = 16

IO Analysis Applied


Optimization Problem

• Create the smallest possible test set which covers all data combinations in the individual output test sets.
• Test set optimized using “Don’t Care” (DC) conditions. Test data which is not relvent can be replaced by a flag (say DC).

T(W) T (Z) T(Optimized)(1, DC, TDC) (DC, N, DC) (1, N, TDC)
(1, DC, BDM) (DC, S, DC) (1, S, BDM)
(3, DC, TDC) (DC, E, DC) (3, E, TDC)
(3, DC, BDM) (DC, W, DC) (3, W, BDM)
Example:- Customize Tools menu ScreenNow I will show how I have implemented this test case reduction method in the below displayed screen:-


IO Analysis Applied
Total areas to be checked:-
1) Menus Related
2) Icons Related
Following are total controls :-
>> 5 checkboxes = 2*2*2*2*2 = 32 values
>> 1 list boxes = 6 entries
>> 1 button = 2 Values
>> Total possible test cases = 32*6*2 = 432
Now Following are the combinations which can be created:-
» Menu TC= CB1+CB2+LB
» Icons TC= CB3+CB4+CB5+B
» Control set= CB1+CB2+CB3+CB4+CB5+LB+B
Reducing test cases
»T(Menu)
(1, 1 , DC, DC, DC, None,1)
(1, 0, DC, DC, DC, Random,0)
(0, DC, DC, DC, DC, Unfold,1)
(1, 1 , DC, DC, DC, Slide,0)
(1, 0, DC, DC, DC, Explode,1)
(0, DC, DC, DC, DC, Fadein, 0)= 6 test cases
»T(Icon)
(DC, DC, 1, 1, 1, DC, 0 )
(DC, DC, 1, 1, 0, DC, 1 )
(DC, DC, 1, 0, DC, DC,0)
(DC, DC, 0, 1, 1, DC, 1)
(DC, DC, 0, 1, 0, DC, 0)
(DC, DC, 0, 0, DC, DC, 1) = 6 test cases

T(Menu) U T(Icon)= 6+6=12 test cases
(1, 1, 1, 1, 1, None,(1,0))
(1, 0, 1, 1, 0, Random, (1,0))
(0, 0, 1, 0, 0, Unfold, (1,0))
(1, 1 , 0, 1, 1, Slide, (1,0))
(1, 0, 0, 1, 0, Explode, (1, 0))
(0, 0, 0 , 0, 0, Fadein,(1, 0))
Total test case reduction = 97%
Test case can be reduced further by using the combinotorial testing methods in the above result set but some test cases needs to be dropped in that case.
Concusion -> For the class of program and type of problem defined, input-output analysis can reduce the number of combinatorial tests while “maintaining” the fault-detection capability of the test set.
-> Research continues on identifying program types and testing situations where this approach is applicable.

Automation/Manual Testing Tips

1) I always works on Cockroach theory of testing which means if you are able to find one bug in a application at one particular functionality, try to search for many more bugs in that module itself. This is because bugs are like cockroaches who always live in a group. If you able to find one, changes are quite likely that you will be able to find many more at the same place.

2) Try to automate as much as possible. It is going to make the life miserable in the start while creating the test scripts, but you will enjoy rest of you project tenure once the automation is over.

3) Divide each and every step of a test case in a separate file, test script while doing automation. It will help you to reuse various test scripts. Try to create a tracker while starting of the automation so that you are able to track the test scripts depending upon the functionality.

4) For a large application, divide the automatic test runs in such a way that multiple machines are used so that team can get the results within a short span of time.

5) Integrate the automation test suite with Cruise Control (or any other build management system) so that automation batch will get started automatically once the build is out. [In my next blog, I will talk about the automation framework which I have build for my current project and is integrated with cruise control as well].