CN112685327B - Method for generating failure test case of model domain - Google Patents
Method for generating failure test case of model domain Download PDFInfo
- Publication number
- CN112685327B CN112685327B CN202110115994.1A CN202110115994A CN112685327B CN 112685327 B CN112685327 B CN 112685327B CN 202110115994 A CN202110115994 A CN 202110115994A CN 112685327 B CN112685327 B CN 112685327B
- Authority
- CN
- China
- Prior art keywords
- test case
- failure
- index
- failed
- failure test
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The invention relates to a method for generating failure test cases in a model domain, which comprises the steps of defining a minimum suspicious set, extracting common features from all the failure test cases, and then mutating information of non-common features on the basis of keeping the common features of the failure test cases for each failure test case in the model domain, so as to generate a new failure test case in the model domain. And finally, merging the failure test cases of the newly added model domain into the original test case set and positioning defects. The method uses a variant nearest neighbor algorithm to generate the failure test case vector of the model domain, thereby achieving the aim of improving the defect positioning effect.
Description
Technical Field
The invention relates to a failure test case generation method, in particular to a failure test case generation method of a model domain.
Background
During software development and maintenance, software debugging is aimed at finding and repairing errors present in software. It is a time and labor consuming process that often requires significant effort and time from the developer. In order to reduce the cost of debugging, researchers have proposed many ways to help the debugger find defects in the program. A typical procedure for locating program defects is: assuming a program P having a test case set T constructed from an input field, the program P generates an information matrix of a model field after executing all test cases in T, see fig. 1. And then calculating the coverage information of the model domain by using a suspicious value calculation formula, and finally obtaining a statement suspicious value list arranged in a descending order. The coverage information of this model field is typically represented in the form of a matrix.
The test case set T plays a crucial role in the defect localization process. It contains two test cases of different characteristics: successful test cases and failed test cases. In reality, however, the number of failed test cases that lead to software failure is much smaller than the number of successful test cases. Existing studies have shown that failed test cases tend to be more helpful in locating defects. The problem of class imbalance among test cases can affect the final result of defect localization. Therefore, to improve the positioning performance, more failed test cases must be generated to alleviate the class imbalance problem.
One way to generate a failure test case is to start from the input field. However, inputs that cause program failure typically account for only a small portion of the input field. Therefore, researchers have made a great deal of effort to generate failed test cases, but in view of the difficulty of generating in the input domain, existing test case generation methods rarely start with failed test cases.
Disclosure of Invention
Aiming at the problems existing in the prior art, the technical problems to be solved by the invention are as follows: how to generate valid failed test cases that can be used to locate software defects.
In order to solve the technical problems, the invention adopts the following technical scheme: a failure test case generation method of model domain,
the method comprises the following steps:
s10: the method comprises the steps of obtaining data, giving a program P containing defects, wherein a group of original test cases T are provided, and the output of each test case in the group of test cases T is known;
giving each test case in the T to P for execution to obtain statement coverage information of each test case in the program P;
meanwhile, for each test case, comparing the known output of the test case with the actual output obtained after the program P is run, if the actual output of each test case is the same as the known output, the label of the test case is 1, otherwise, the label of the test case is 0;
the labels of all test cases form a label vector;
statement coverage information and labels of all test cases in the program P form an original coverage matrix of an original test case set, and the original coverage matrix is recorded as TOrig;
s20: initializing variables, namely copying an original coverage matrix TOrig to an output matrix TNew, and initializing the output matrix;
finding statement coverage information and labels corresponding to the failed test cases in the original coverage matrix and label vectors, and forming a failed test case coverage matrix TOrigF by the statement coverage information and labels corresponding to all the failed test cases;
calculating the successful test case quantity Pnum, the failed test case quantity Fnum and the difference value Fnewnum between the successful test case quantity and the failed test case quantity in the original test case set, wherein Fnewnum=Pnum-Fnum;
assigning an experience value to the nearest neighbor number k of the program P as an initial value;
s30, finding k nearest neighbors of each failed test case, and finding k nearest failed test cases to the current failed test case as nearest neighbors for each failed test case, and storing the nearest k nearest neighbors in an array nnarray;
s40, generating a failure test case on a model domain;
initializing the value of index to 1;
s41: selecting a random integer between 1 and k, and marking the random integer as r;
s42: extracting index-th failure test case t from TOrigF i The method comprises the steps of carrying out a first treatment on the surface of the In nnarray index In the method, the (t) of the (r) nearest neighbor failure test case is taken out r ；
S43: using the formula: t is t i +rand(0,1)*|t r -t i Generating a new failure test case, and setting a new failure test case label as 1;
s44: the newly generated failure test case is marked as t new And t is taken new Adding the end of TNew to obtain TNew index ；
S45: let index value be index+1; judging whether the value of index satisfies the value of index (Fnum+1) to be 0, if so, making the value of index to be 1, and executing the next step; if not, executing the next step;
s46: judging whether index is greater than Fnewum, if so, returning to S41, otherwise, outputting TNew index 。
As an improvement, the method for finding k nearest failure test cases to the current failure test case in S30 includes: and calculating Euclidean distances between the current failure test case and all the failure test cases, arranging the obtained Euclidean distances in sequence from large to small, and taking the failure test cases corresponding to the first k Euclidean distances as k nearest neighbors of the current failure test case.
Compared with the prior art, the invention has at least the following advantages:
the invention provides a failure test case generating method of a model domain, which uses a variant nearest neighbor algorithm to generate a failure test case vector of the model domain, thereby achieving the aim of improving the defect positioning effect.
Drawings
FIG. 1 is a process for locating a program defect typical in the prior art.
FIG. 2 is a schematic diagram of a program spectral information model of a model domain.
FIG. 3 is a schematic diagram of a virtual test case for use in defect localization based on deep learning.
Fig. 4 is a structural diagram of a defect localization technique based on deep learning.
Fig. 5 is an example illustrating the principles of the method of the present invention.
FIGS. 6 (a) -6 (d) are graphs comparing EXAM values for 8 defect localization methods using MTF-Gen and localization methods without MTF-Gen.
FIG. 7 (a) is a rim comparison of used MTF-Gen versus unused MTF-Gen for all subject procedures over 8 defect localization methods; FIG. 7 (b) is a rim comparison of the use of MTF-Gen versus the non-use of MTF-Gen for all defect localization methods at the 8 subject program.
Detailed Description
The present invention will be described in further detail below.
The invention achieves the purpose of enhancing the test case set by generating the vector of the failure test case on the coverage matrix, and realizes that the generation of the failure test case on the coverage matrix faces two challenges.
The first is to select which information in the model domain to generate in order to better promote the accuracy of the positioning. The data in the model field that covers all failed test case features (the smallest suspicious fragment), which refers to the statement that all failed test cases cover, has a great effect on locating defects. For a single defect program, we can intuitively judge that the minimum suspicious fragment definitely contains an error statement. This means that the test case vector generated from the model field containing the smallest suspicious fragment can be labeled as a failed test case. Another challenge is how much data to generate. A class balanced test case set is more advantageous for locating defects, and typically performs beyond a class unbalanced test case set. Thus, we continue to generate failed test cases until it has the same number as successful test cases.
The invention solves the two challenges and provides an MTF-Gen (maximum Transmission factor-Gen) model-level failure test case generation method. The method uses a variant nearest neighbor algorithm to generate a failure test case vector of the model domain, thereby achieving the purpose of improving the defect positioning effect. The MTF-Gen uses an information model in which a vector records the execution information of a test case, showing which statements the test case covers. Some specific values are used to represent different weights. The MTF-Gen generates a new failure test case by covering the minimum suspicious segment in the model domain and utilizing the existing failure test case.
Specifically, for failed test cases in a model domain, the MTF-Gen selects one of the nearest neighbors from all failed test cases and finds their intersection, which contains the set of sentences (the smallest suspicious fragments) that are covered by all failed test cases. To preserve common features in the intersection, the MTF-Gen generates a new failed test case by simply changing the different portions of the two failed test cases. Thus, this new failed test case containing the smallest suspicious fragment is helpful for computing defects. Finally, MTF-Gen continuously generates new failure test cases until a test case set with balanced categories is obtained, and the set contains the same number of successful test cases and failure test cases.
Unlike conventional methods for generating test cases from input fields, the method for generating test case vectors for failure in model fields is simpler and more efficient, and does not require program execution to obtain a failed tag.
A failure test case generation method of a model domain comprises the following steps:
s10: the method comprises the steps of obtaining data, giving a program P containing defects, wherein a group of original test cases T are provided, and the output of each test case in the group of test cases T is known;
giving each test case in the T to P for execution to obtain statement coverage information of each test case in the program P;
meanwhile, for each test case, comparing the known output of the test case with the actual output obtained after the program P is run, if the actual output of each test case is the same as the known output, the label of the test case is 1, otherwise, the label of the test case is 0;
the labels of all test cases form a label vector;
statement coverage information and labels of all test cases in the program P form an original coverage matrix of an original test case set, which is marked as TOrig, and see FIG. 2; if the program has n sentences and m test cases, the dimension of the coverage matrix is m×n, the dimension of the label vector is m×1, and the dimension of the matrix harrig is m×1.
S20: initializing variables, namely copying an original coverage matrix TOrig to an output matrix TNew, and initializing the output matrix;
finding statement coverage information and labels corresponding to the failed test cases in the original coverage matrix and label vectors, and forming a failed test case coverage matrix TOrigF by the statement coverage information and labels corresponding to all the failed test cases;
calculating the successful test case quantity Pnum, the failed test case quantity Fnum and the difference value Fnewnum between the successful test case quantity and the failed test case quantity in the original test case set, wherein Fnewnum=Pnum-Fnum;
and assigning an experience value to the nearest neighbor number k of the program P as an initial value.
S30, finding k nearest neighbors of each failed test case, and finding k nearest failed test cases to the current failed test case as nearest neighbors for each failed test case, and storing the nearest k nearest neighbors in an array nnarray; i.e., each failed test case t j All correspond to an array of nearest neighbor storing k j 。
Specifically, the method for finding k failure test cases closest to the current failure test case in S30 includes:
the Euclidean distance between the current failure test case and all the failure test cases is calculated, and the obtained Euclidean distance is in the order from large to small
And (3) sequencing, namely taking the failure test cases corresponding to the first k Euclidean distances as k nearest neighbors of the current failure test case.
S40, generating a failure test case on the model domain.
Initializing the value of index to 1;
s41: selecting a random integer between 1 and k, and marking the random integer as r;
s42: extracting index-th failure test case t from TOrigF i The method comprises the steps of carrying out a first treatment on the surface of the In nnarray index In the method, the (t) of the (r) nearest neighbor failure test case is taken out r ；
S43: using the formula: t is t i +rand(0,1)*|t r -t i Generating a new failure test case, and setting a new failure test case label as 1;
s44: the newly generated failure test case is marked as t new And t is taken new Adding the end of TNew to obtain TNew index ；
S45: let index value be index+1; judging whether the value of index satisfies the value of index (Fnum+1) to be 0, if so, making the value of index to be 1, and executing the next step; if not, executing the next step;
s46: judging whether index is greater than Fnewum, if so, returning to S41, otherwise, outputting TNew index 。
Data covering the smallest suspicious set (common features for all failed test cases) is advantageous for improving positioning accuracy. Intuitively, the test case does not cause the program to fail until the real defect statement is executed, so the minimum suspicious set certainly contains the real defect statement.
If a true defect statement is overridden in a vector labeled with a failed test case (labeled 1), then the defect localization technique increases the suspicious value of the statement. Because the newly generated test case vector contains the minimum suspicious set, the real defect statement is covered, and the test case vector is marked as a failed test case, therefore, the suspicious value of the suspicious statement can be improved, namely the positioning accuracy is improved.
Thus, the main idea of MTF-Gen is to use the information vectors of existing failed test cases (those labeled with the failed test case labels) to synthesize a new information vector that covers the smallest suspicious set. In particular, for each failed test case, the MTF-Gen uses an oversampling technique that synthesizes a minority of data to generate a new failed test case information vector from the k nearest neighbor failed test case information vectors. The intersection of the k nearest neighbor failed test cases (i.e., the intersection of the features) contains the smallest suspicious set. The MTF-Gen calculates the intersection of the k nearest neighbors and generates a new information vector on the basis that contains the common feature of the nearest neighbors, namely the smallest suspicious set. Thus, the MTF-Gen marks this new information vector as a failed test case and adds it to the information model.
The MTF-Gen should confirm how many failed test cases were generated to best improve the efficacy of defect localization. In fact, the class-balanced test case set is more beneficial to improving the accuracy of defect positioning, and the positioning algorithm based on the class-balanced test case set often exceeds the algorithm based on class imbalance in accuracy. Therefore, the MTF-Gen continuously generates the number of test case information vectors with failure labels in a model domain until we obtain a class-balanced test case set, wherein the number of failed test case vectors is equal to the number of successful test case vectors, which means that we have the same number of failed test cases and successful test cases.
The principle of the method of the invention is illustrated below:
the principle of the information domain test case generation method MTF-Gen is further described with an example. Fig. 5 shows a program P with 16 sentences, P containing a defective sentence s3. The cells under each statement in the figure record information whether the statement is covered by the test case, 1 indicates covered, and 0 indicates uncovered. The right-most table indicates success or failure of the test case, 1 indicates a failed test case, and 0 indicates a successful test case. We can see that there are 6 test cases in the example, where t1 and t6 are failed test cases.
We selected GP02 and MLP-FL as representatives, representing the existing spectrum-based and deep learning-based defect localization methods, respectively. Because there are only two failed test cases t1 and t6 in the test case set, they are nearest neighbors to each other. The MFT-Gen calculates the difference between t1 and t6 and the nearest neighbor by using the 16 th line method in the algorithm 1, and continuously generates new failed test case information vectors until the information vectors of the failed test cases and the information vectors of the successful test cases have the same quantity. In fig. 5, t7 and t8 are new failed test cases. Since the MTF-Gen maintains the intersection of all failed test case information vectors, i.e., the minimum suspicious set, the newly generated test case information vector also contains this minimum suspicious set, defined as { s1, s2, s3, s7, s8, s9, s12, s13} in FIG. 5. Because t7 and t8 also contain the smallest suspicious set, we can ignore whether there are such test cases in the input domain in reality, and consider it as a test case that also fails in the model domain. We set the tags of t7 and t8 to 1 according to step S43 in the method, indicating that it is a failed test case. After adding t7 and t8 to the original test case set, we obtain a new test case set containing 4 successful test cases and 4 failed test cases.
As shown in fig. 5, there are a total of four lists arranged in descending order. In the case of the original test case set as input, the GP02 outputs { s7, s8, s9, s12, s10, s11, s14, s15, s16, s13, s1, s2, s3, s6, s4, s5}, and the MLP outputs { s5, s4, s16, s11, s15, s6, s13, s12, s2, s8, s9, s3, s14, s10, s1, s7}. When the new test case set added at t7 and t8 is used as input, the GP02 outputs { s1, s8, s14, s3, s16, s5, s10, s12, s6, s11, s9, s13, s7, s15, s2, s4}, and the MLP outputs { s7, s8, s9, s12, s10, s1, s2, s3, s13, s14, s11, s15, s16, s5, s4, s6}. We can see that in the case of the original test case as input, the ranking of the defect sentence s3 in both the GP02 and MLP-FL localization models is 13 th and 12 th, respectively. After the new test case set generated by MTF-Gen is used, GP02 (MTF-Gen) and MLP (MTF-Gen) respectively rank defect sentences s3 by 8 th and 7 th, and better defect positioning accuracy is obtained.
And (3) experimental verification:
the selected dataset was from three websites, comprising 8 programs, each program having a different defect version and being all real defects, wherein python, gzip, libtiff programs were derived from ManyBugs, http:// repairbenchmarks.cs.umass.edu/ManyBugs /); the space program is derived from SIR (http:// SIR. Un. Edu/portal/index. Php); chart, math, lang and time programs are derived from Defect4J (http:// Defect 4J. Org). Table 1 describes the characteristics of these procedures.
TABLE 1 characterization of the subject
| Program name | Brief description of the drawings | Number of versions | Number of lines of code (thousands of lines) | Number of test cases | |
| python | |
8 | 407 | 355 | |
| | Data compression | 5 | 491 | 12 | |
| | Image processing | 12 | 77 | 78 | |
| space | ADL interpreter | 35 | 6.1 | 13585 | |
| chart | JFreeChart package | 26 | 96 | 2205 | |
| math | Math program package | 106 | 85 | 3602 | |
| mockito | Unit test frame | 38 | 6 | 1075 | |
| time | Joda-Time package | 27 | 53 | 4130 |
To verify the effectiveness of the present invention, the present method was applied to the 8 most advanced defect localization methods and compared to the effect of defect localization methods that did not employ the present method. Including 2 deep learning-based defect localization methods (see fig. 3 and 4) and 6 program spectrum-based defect localization methods. The calculation formula of 6 defect localization methods based on program spectra is shown in table 2. The 2 defect positioning methods based on deep learning are CNN-FL and MLP-FL, and a convolutional neural network and a multi-layer perceptron are respectively used as components of a deep learning component layer.
Table 2 suspicious value calculation formulas
The failure test case generation method of the model domain is simply called MTF-Gen.
Two widely used evaluation criteria will be used here: EXAM and RIMP. The EXAM is defined as the percentage of sentences that have been checked when the true error sentence was found. Lower values of EXAM indicate better defect localization performance. RImp is defined as the sum of the number of all statements that MTF-Gen finds error checks for all versions of a program divided by the sum of the number of all statements that another defect localization method finds error checks for all versions of the program. For MTF-Gen, a lower rim value represents better localization performance.
Experimental results show that under the EXAM standard, the curve of the defect localization method using MTF-Gen is always above the curve of the defect localization method not using MTF-Gen, which indicates that MTF-Gen improves the efficacy of the defect localization technique.
Under the rim standard, in all defect localization methods, all scores for rim were below 100%. This shows that the efficacy of localization using MTF-Gen was improved on all experimental procedures for each defect localization method. Taking the MLP-FL as an example, the score of RImp was 63.17%, which indicates that the total number of sentences examined by the MLP-FL using MTF-Gen when all defects of 8 subject programs were found was 63.17% of the total number of sentences examined by the MLP-FL not using MTF-Gen when all defects of 8 subject programs were found. In other words, the number of sentences to be inspected when all defects in all subject procedures were inspected was reduced by 36.83% (100% -63.17% = 36.83%) as compared to the MLP-FL using the MTF-Gen and the MLP-FL not using the MTF-Gen. We can see that the rim score using MTF-Gen ranges from 49.63% for GP03 to 98.11% for ER 5. This means that the statement using the defect localization method of MTF-Gen has a saving value of 50.37% (100% -49.63% = 50.37%) of GP03 at the maximum and 1.89% (100% -98.11% = 1.89%) of ER5 at the minimum. In all experimental procedures, all scores for rim were below 100%, which means that MTF-Gen had an improvement in localization efficacy over all experimental procedures. Taking chart as an example, the rim score is 68.71%, which means that in the chart program, the total sentence number required to check all the defects of chart by using the MTF-Gen is 68.71% of the total sentence number required to check all the defects of chart by not using the MTF-Gen. This also means that MTF-Gen can save 31.29% (100% -68.71% = 31.29%) of the number of sentences to be checked on program object chart. The rim value of MTF-Gen on all subject programs ranged from 52.09% of mockito 98.92% of libtiff, which indicated that the maximum saving of MTF-Gen on the subject program was 47.91% (100% -52.09% = 47.91%) of mockito and the minimum saving was 1.08% (100% -98.92% = 1.08%) of libtiff.
Overall, the savings value for MTF-Gen was 23.53% on average over all defect localization methods and all subject procedures. Thus, based on rim, a significant increase in the efficacy of defect localization after the use of MTF-Gen can be seen.
Since rim only shows a specific boost ratio, which is only a general effect boost display, some specific details may be omitted, we used the Wilcoxon signed rank test for statistical analysis in order to further verify the effectiveness of the present invention. Wilcoxon signed rank test non-parametric statistics were used to test the differences between a pair of data F (x) and G (y) that do not follow a normal distribution. Given a parameter σ, we use the p-values of 2-labeled and 1-labeled to draw conclusions. For the p value of 2-charged, if p Σ is equal to or greater than p, then assume H0: there was no significant difference between F (x) and G (y). Otherwise, accept hypothesis H1: f (x) and G (y) are significantly different. There are two cases for the p value of 1-charged: 1-pinned (right) and 1-pinned (left), respectively. For 1-strained (right), if p Σ, then assume H0: f (x) and G (y) are not accepted as Better by comparison; otherwise assume H1: the comparison of F (x) and G (y) results in Better being accepted. For 1-tailed (left), if p Σ, then assume H0: f (x) and G (y) are not accepted as Worse; otherwise, assume that the comparison of H1 to F (x) and G (y) results in Worse being accepted.
FIGS. 6 (a) -6 (d) show graphs comparing EXAM values for 8 defect localization methods using MTF-Gen and localization methods without MTF-Gen. In fig. 6 (a) -6 (d), the abscissa represents the percentage of the inspected sentence, and the ordinate represents the percentage of the number of defective versions that have been found among the numbers of all defective versions. "one dot in fig. 6 (a) -6 (d) indicates the percentage of the number of defects detected when a statement of a certain scale is detected, to the total number of defects. From FIGS. 6 (a) -6 (d), it can be seen that the curve of the defect localization method using MTF-Gen is always above the curve of the defect localization method not using MTF-Gen, indicating that MTF-Gen improves the efficacy of the defect localization technique.
To further demonstrate the results of the experiment, we used rim to evaluate the effectiveness of MTF-Gen. FIG. 7 shows a comparison of RImp using MTF-Gen in both cases, FIG. 7 (a) is a comparison of RImp using MTF-Gen versus not using MTF-Gen for all subject procedures over 8 defect localization methods, and FIG. 7 (b) is a comparison of RImp using MTF-Gen versus not using MTF-Gen for all defect localization methods over 8 subject procedures.
As shown in fig. 7 (a), in all defect localization methods, all scores of rim were 100% or less. This shows that the efficacy of localization using MTF-Gen was improved on all experimental procedures for each defect localization method. Taking the MLP-FL as an example, the score of RImp was 63.17%, which indicates that the total number of sentences examined by the MLP-FL using MTF-Gen when all defects of 8 subject programs were found was 63.17% of the total number of sentences examined by the MLP-FL not using MTF-Gen when all defects of 8 subject programs were found. In other words, the number of sentences to be inspected when all defects in all subject procedures were inspected was reduced by 36.83% (100% -63.17% = 36.83%) as compared to the MLP-FL using the MTF-Gen and the MLP-FL not using the MTF-Gen. We can see that the rim score using MTF-Gen ranges from 49.63% for GP03 to 98.11% for ER 5. This means that the statement using the defect localization method of MTF-Gen has a saving value of 50.37% (100% -49.63% = 50.37%) of GP03 at the maximum and 1.89% (100% -98.11% = 1.89%) of ER5 at the minimum.
In FIG. 7 (b), all scores for RImp were below 100% in all experimental procedures, which means that MTF-Gen had an improvement in localization efficacy over all experimental procedures. Taking chart as an example, the rim score is 68.71%, which means that in the chart program, the total sentence number required to check all the defects of chart by using the MTF-Gen is 68.71% of the total sentence number required to check all the defects of chart by not using the MTF-Gen. This also means that MTF-Gen can save 31.29% (100% -68.71% = 31.29%) of the number of sentences to be checked on program object chart. The rim value of MTF-Gen on all subject programs ranged from 52.09% of mockito 98.92% of libtiff, which indicated that the maximum saving of MTF-Gen on the subject program was 47.91% (100% -52.09% = 47.91%) of mockito and the minimum saving was 1.08% (100% -98.92% = 1.08%) of libtiff.
In the experiments, we compared the Wilcoxon signed rank test for 8 defect localization methods with and without MTF-Gen. The experiment used EXAM as a benchmark to compare F (x) and G (y), each test used 2-tailed and 1-tailed detection with sigma of 0.05. In particular, assuming that there is a defect localization method FL1, we assume that a series of EXAM values of FL1 of MTF-Gen at all defective versions of all programs are used as F (x), and that a series of EXAM values of FL1 of MTF-Gen at all defective versions of all programs are not used as G (y). Thereafter, in the 2-charged assay, FL1 with MTF-Gen and FL1 without MTF-Gen had SIMILAR localization efficacy when receiving H0 with a significance level sigma of 0.05, resulting in SIMILAR. In the 1-labeled (right) test, FL1 using MTF-Gen was less efficient in localization than FL1 without MTF-Gen when H1 with a significance level sigma of 0.05 was accepted, resulting in WORSE. In the 1-labeled (left) test, FL1 using MTF-Gen was BETTER localized than FL1 without MTF-Gen when H1 with a significance level sigma of 0.05 was accepted, resulting in BETTER.
In short, the invention provides a method for generating the failure test case of the model domain, which has reasonable and sufficient interpretability. The method is applied to the defect positioning method of the current mainstream, can obviously improve the accuracy of positioning, can be applied to actual application scenes, and contributes to improving the efficiency of finding the defect position by developers.
Table 3 Wilcoxon sign rank test
Table 3 summarizes the Wilcoxon signed rank test comparison over all defect localization methods and Wilcoxon signed rank test comparison over all experimental procedure subjects. For example, in the comparison of defect localization methods, MLP-FL (MTF-Gen) obtained 8 BETTER (8/8=100%) compared to MLP-FL. In comparison on experimental procedures, gzip (MTF-Gen) obtained 4 BETTERs (4/8=50%), 4 SIMILARs (4/8=50%), 0 WORSE (0/8=0%), compared with gzip. As can be seen from table 3, MTF-Gen yielded a total of 76 bet ters (76/128=59.37%), 52 SIMILARs (52/127= 40.63%) and 0 WORSE. Thus, based on the above results and analysis, we can conclude that MTF-Gen significantly improves the efficacy of defect localization, showing potential from modeling and test case generation.
Finally, it is noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the technical solution of the present invention, which is intended to be covered by the scope of the claims of the present invention.
Claims (2)
1. The method for generating the failure test case of the model domain is characterized by comprising the following steps:
s10: the method comprises the steps of obtaining data, giving a program P containing defects, wherein a group of original test cases T are provided, and the output of each test case in the group of test cases T is known;
giving each test case in the T to P for execution to obtain statement coverage information of each test case in the program P;
meanwhile, for each test case, comparing the known output of the test case with the actual output obtained after the program P is run, if the actual output of the test case is the same as the known output, the label of the test case is 1, otherwise, the label of the test case is 0;
the labels of all test cases form a label vector;
statement coverage information and labels of all test cases in the program P form an original coverage matrix of an original test case set, and the original coverage matrix is recorded as TOrig;
s20: initializing variables, namely copying an original coverage matrix TOrig to an output matrix TNew, and initializing the output matrix;
finding statement coverage information and labels corresponding to the failed test cases in the original coverage matrix and label vectors, and forming a failed test case coverage matrix TOrigF by the statement coverage information and labels corresponding to all the failed test cases;
calculating the successful test case quantity Pnum, the failed test case quantity Fnum and the difference value Fnewnum between the successful test case quantity and the failed test case quantity in the original test case set, wherein Fnewnum=Pnum-Fnum;
assigning an experience value to the nearest neighbor number k of the program P as an initial value;
s30, finding k nearest neighbors of each failed test case, and finding k nearest failed test cases to the current failed test case as nearest neighbors for each failed test case, and storing the nearest k nearest neighbors in an array nnarray;
s40, generating a failure test case on a model domain;
initializing the value of index to 1;
s41: selecting a random integer between 1 and k, and marking the random integer as r;
s42: extracting index-th failure test case t from TOrigF i The method comprises the steps of carrying out a first treatment on the surface of the In nnarray index In the method, the (t) of the (r) nearest neighbor failure test case is taken out r ；
S43: using the formula: t is t i +rand(0,1)*|t r -t i Generating a new failure test case, and setting a new failure test case label as 1;
s44: the newly generated failure test case is marked as t new And t is taken new Adding the end of TNew to obtain TNew index ；
S45: let index value be index+1; judging whether the value of index satisfies the value of index (Fnum+1) to be 0, if so, making the value of index to be 1, and executing the next step; if not, executing the next step;
s46: judging whether index is greater than Fnewum, if so, returning to S41, otherwise, outputting TNew index 。
2. The method for generating a failure test case in a model domain according to claim 1, wherein: the method for finding k nearest failure test cases from the current failure test cases in the step S30 is as follows:
and calculating Euclidean distances between the current failure test case and all the failure test cases, arranging the obtained Euclidean distances in sequence from large to small, and taking the failure test cases corresponding to the first k Euclidean distances as k nearest neighbors of the current failure test case.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110115994.1A CN112685327B (en) | 2021-01-28 | 2021-01-28 | Method for generating failure test case of model domain |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110115994.1A CN112685327B (en) | 2021-01-28 | 2021-01-28 | Method for generating failure test case of model domain |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112685327A CN112685327A (en) | 2021-04-20 |
| CN112685327B true CN112685327B (en) | 2023-06-27 |
Family
ID=75459414
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110115994.1A Active CN112685327B (en) | 2021-01-28 | 2021-01-28 | Method for generating failure test case of model domain |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112685327B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113836027B (en) * | 2021-09-28 | 2023-07-21 | 重庆大学 | Method for generating failure test case by using generation type network |
| CN114416845A (en) * | 2022-01-19 | 2022-04-29 | 平安好医投资管理有限公司 | Big data testing method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103019943A (en) * | 2013-01-04 | 2013-04-03 | 北京航空航天大学 | Software error positioning method catering to rare case of failed test cases |
| CN103970676A (en) * | 2013-01-29 | 2014-08-06 | 帝斯贝思数字信号处理和控制工程有限公司 | Computer-implemented method and system for data management of product variants in control unit development |
| CN105117331A (en) * | 2015-08-17 | 2015-12-02 | 浪潮(北京)电子信息产业有限公司 | Error-location-oriented coincidence correctness test case identification method and device |
| CN106909502A (en) * | 2017-01-13 | 2017-06-30 | 河南工业大学 | Accidental correctness test case recognition methods and software error localization method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9032361B2 (en) * | 2011-12-15 | 2015-05-12 | Tata Consultancy Services Limited | Agile unit and regression testing framework for domain specific languages |
-
2021
- 2021-01-28 CN CN202110115994.1A patent/CN112685327B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103019943A (en) * | 2013-01-04 | 2013-04-03 | 北京航空航天大学 | Software error positioning method catering to rare case of failed test cases |
| CN103970676A (en) * | 2013-01-29 | 2014-08-06 | 帝斯贝思数字信号处理和控制工程有限公司 | Computer-implemented method and system for data management of product variants in control unit development |
| CN105117331A (en) * | 2015-08-17 | 2015-12-02 | 浪潮(北京)电子信息产业有限公司 | Error-location-oriented coincidence correctness test case identification method and device |
| CN106909502A (en) * | 2017-01-13 | 2017-06-30 | 河南工业大学 | Accidental correctness test case recognition methods and software error localization method |
Non-Patent Citations (3)
| Title |
|---|
| "Test Case Evaluation and Input Domain Reduction strategies for the Evolutionary Testing of Object-Oriented software";José Carlos Bregieiro Ribeiro 等;《Information and Software Technology》;第51卷(第11期);第1534-1548页 * |
| "基于输出域的测试用例自动生成方法研究";尤枫 等;《计算机研究与发展》;第541-549页 * |
| "增强上下文的错误定位技术";张卓 等;《软件学报》;第266-281页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112685327A (en) | 2021-04-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11423311B2 (en) | Automatic tuning of artificial neural networks | |
| Zhang et al. | CNN-FL: An effective approach for localizing faults using convolutional neural networks | |
| CN112685327B (en) | Method for generating failure test case of model domain | |
| Lei et al. | How test suites impact fault localisation starting from the size | |
| CN108491302B (en) | Method for detecting spark cluster node state | |
| CN113705092B (en) | Disease prediction method and device based on machine learning | |
| US10802825B2 (en) | Cognitive software porting assistant system | |
| CN113011509B (en) | Lung bronchus classification method and device, electronic equipment and storage medium | |
| US11630758B2 (en) | Artificial intelligence enabled output space exploration for guided test case generation | |
| CN109409504A (en) | A kind of data processing method, device, computer and storage medium | |
| CN113900923A (en) | System and method for checking similarity of binary functions of cross-instruction set architecture | |
| CN118012767B (en) | Multi-coverage-rate-use-case-oriented generation and multi-method-fusion-oriented software bug positioning method | |
| CN111831556A (en) | Software multi-fault decoupling and parallel positioning method and device | |
| US20210287131A1 (en) | Machine learning model accuracy fairness | |
| CN116432570A (en) | Method and device for generating test case of chip and storage medium | |
| CN116243883A (en) | Test excitation generation method, test method and related equipment | |
| CN113177387A (en) | Pixel layout generation method and device of display panel | |
| CN112420125A (en) | Molecular attribute prediction method and device, intelligent equipment and terminal | |
| CN111104400A (en) | Data normalization method and device, electronic equipment and storage medium | |
| Zhang et al. | Improving fault localization using model-domain synthesized failing test generation | |
| CN114721922A (en) | Performance evaluation method of server cluster, computing equipment and storage medium | |
| CN116486151A (en) | Image classification model training method, image classification method, device and storage medium | |
| CN115167868A (en) | Code compiling method, device, equipment and computer storage medium | |
| CN110096447B (en) | Software and hardware automatic testing method based on image recognition | |
| CN115828848A (en) | Font generation model training method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |