Software testing is a time consuming and tedious process. Most usual testing is test-it-to-death. There must exist criteria which should be followed to decide when to stop testing.
It is required to ensure that developed software must work correctly. Before the developed software is delivered it must be error free. Test-it-to-death is not a fruitful approach, so software testing undergoes different testing phases having stopping rules that decides when to stop testing but still ensuring quality of the software.
Stopping rules are based on a well defined set of rules used for testing. One of the most common criteria to test software is – statement coverage and branch coverage known as coverage criteria. When using coverage testing it is not possible to decide whether further coverage can detect errors or not, for this software testers keep on switching to different testing strategies. To switch to a new testing startegy there must exist some criteria so that it can be ensured that new testing technique was employed since the previous testing technique was saturated.
Each testing technique used to test software is decided on two factors:
- Cost which is based on cost of generating test and validation cost
- Yield which depends on elements covered and bugs found
Using appropriate stopping criteria saves simulation cycle time and ensures that the objective of the test process is achieved. Developing stopping criteria is a software engineering research topic. Binomial distribution is one of the approaches to decide whether the coverage testing should be applied further or not. If the binomial distribution reveals that there exists a low probability of finding error in the next test run then testing stops.
Usage model is another criteria to determine when to stop testing. Usage model works on Markov chains which decide the next state depending on the present state. Usage models are considered to be more reliable than other models.
Empirical-Bayesian estimator is another model which can be used to decide when to stop testing. In software testing, testing of some programming constructs affects other programming constructs. For example, in programming testing of some branches may prevent testing of other branches this case arises in if-then-else condition.
In the testing process, test inputs are grouped into test suits. Test inputs are grouped into problem specific or code specific. These test inputs are then used in testing and checked for coverage. For each check if the desired level of coverage is reached then testing stops and if desired level of coverage is not reached then testing continues. When coverage gets saturated then testing stops.
Question again arises when do we stop testing, the objective function is to find an optimal point at which we should stop testing. The optimal stopping point of testing must maintain the quality of the software and cost of the software. The software must include the cost of test set generation, cost of execution and cost of validating the software.
Software testing stopping rule algorithm
Following is the algorithm for stopping the software testing:
- For testing sequential steps groups of test inputs are used.
- If testing the given sequential steps exceeds the economic criterion which depends on different parameters used in the evaluation of cost of the software then testing is continued with the existing software testing strategy otherwise a different software testing strategy is used.
- If the cost to stop software testing is greater than the cost to continue software testing then software testing is continued.
- If the cost of stopping software testing is less than the cost of continuing software testing then software testing is discontinued.
- Apart from this Bayes model which makes decisions based on the previous results of software testing, it should be used to decide when to stop software testing.
Techniques used in Software Testing
Testing is an important step in the life cycle of software. There exists different kinds of testing each differ in nature and objective. Few of the testing are given below, Table 1 list software testing:
List of Software Testing
Unit Testing | This is the lowest form of testing. It tests modules or components or units of software. |
Integration Testing | This testing is used when two or more modules are combined together to a large module. |
System Testing | This testing is done to test the quality of the developed system as one unit. System testing checks whether the developed system meets the specified requirement of the system or not. |
Acceptance Testing | Acceptance testing is conducted when the developed system is given to the user to work on. This testing is done by the user rather than the tester. |
Two analysis are conducted to assure software quality:
Static Analysis: Static Analysis determines software quality based on predefined methods without executing the software. Static Analysis technique is conducted to check for program analysis, symbolic analysis and model checking.
Dynamic Analysis: Dynamic analysis conducts actual execution with real data as well as unreal data. Techniques of dynamic testing include input synthesis, testing procedures, and automating environment testing.
Apart from static analysis and dynamic analysis functional technique and structural technique is also used. Functional technique and Structural technique of software testing test for different quality aspects of the developed software. Functional Testing and Structural testing is given below:
Functional Testing: This is also known as Black box Testing. Black box testing use test cases developed on design specification of the software. Test cases are based on design specification, dry run values, and results obtained by simulated execution of the software.
Structural Testing: Structural testing is also known as “white box” testing. White box testing is target oriented, it targets specific statements, program branches or paths. A coverage criteria is developed to test expected results. Coverage consists of path coverage, branch coverage and data-flow coverage.
Conclusion:
Software testing is for designers, developers and managers. Research is being carried out to find the effective method to perform software testing. Software testing is done to find errors in the design and implementation of the software. Research is going on to find how much testing is enough. To find optimal testing loops fundamental research is going on. Thus criteria to find how testing is enough for software testing must posses the following qualities::
- The criteria must be accepted by the research community.
- Criteria must not be dependent on other areas of software engineering.
- Developed criteria must possess widespread practical scope.
- Developed criteria must have a set of classes and well established techniques.
There must exist some stopping rule based on statistics to find testing techniques that can cover large amounts of code and take the decision to stop testing. Conducting testing by setting stop criteria results in efficient testing. Setting up stop criteria is a mixture of methods from graph theory, programming language, reliability assessment and reliability-testing theory. Results of theoretical research are used to set stopping criteria. Stopping criteria must use rational techniques to bound the execution of sequence of actions.
Stopping criteria must also consider functional and structural techniques. There exists no software testing strategy that guarantees that there exists no error in the program. Stopping criteria of software testing must include edge strategy to test every edge in the program. Stopping criteria must also include Probe insertion techniques. Probe insertion is a useful testing technique.
Software testing technique must ensure that each path must be traversed at least once. To ensure this decision making in stopping criteria must be dependent on the path testing approach.
Developing a well defined stopping criteria for software testing is a topic of research.
