CN113535486A - Method and device for testing performance of machine - Google Patents
Method and device for testing performance of machine Download PDFInfo
- Publication number
- CN113535486A CN113535486A CN202010289797.7A CN202010289797A CN113535486A CN 113535486 A CN113535486 A CN 113535486A CN 202010289797 A CN202010289797 A CN 202010289797A CN 113535486 A CN113535486 A CN 113535486A
- Authority
- CN
- China
- Prior art keywords
- performance
- test results
- machine
- target test
- calculating
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 188
- 238000000034 method Methods 0.000 title claims abstract description 49
- 238000011056 performance test Methods 0.000 claims abstract description 58
- 238000004364 calculation method Methods 0.000 claims abstract description 17
- 238000010586 diagram Methods 0.000 claims description 24
- 230000001186 cumulative effect Effects 0.000 claims description 15
- 238000013507 mapping Methods 0.000 claims description 15
- 238000004590 computer program Methods 0.000 claims description 9
- 238000010998 test method Methods 0.000 claims description 7
- 230000015556 catabolic process Effects 0.000 description 10
- 238000006731 degradation reaction Methods 0.000 description 10
- 230000009467 reduction Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000035945 sensitivity Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000012827 research and development Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 230000002354 daily effect Effects 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3452—Performance evaluation by statistical analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/2273—Test methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Life Sciences & Earth Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Probability & Statistics with Applications (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method and a device for testing the performance of a machine, and relates to the technical field of computers. One embodiment of the method comprises: repeatedly executing performance tests on the machine to obtain a plurality of target test results; calculating the multiple target test results based on the performance reference value, and judging the performance of the machine according to the calculation results; the performance reference value is obtained by adopting the following method: and repeatedly executing the performance test on the machine to obtain a plurality of benchmark test results, thereby calculating to obtain the performance benchmark value. The implementation mode can solve the technical problem that a large number of false alarms exist.
Description
Technical Field
The invention relates to the technical field of computers, in particular to a method and a device for testing performance of a machine.
Background
For a system with a client-server architecture, the performance of a server interface may affect the response time of a user operation and also affect the throughput Per unit time (QPS) of the system. In order to monitor whether the performance of the server-side interface is reduced along with the evolution of the system, the performance test is generally performed regularly, and the performance test is automated under the condition that the tool ecology is perfect, and is operated every week or even every day.
Such performance tests typically record the maximum throughput per second of a single machine, i.e., the throughput per second of a machine when the CPU, memory, etc. of the machine reaches a relatively high critical state. Specifically, the existing method is: recording the maximum throughput per second for a certain period of time, taking an average as a reference, periodically and automatically executing a test, if the maximum throughput per second is found to be reduced, indicating that the performance is reduced, if the reduction amplitude reaches a set threshold value, alarming, informing research and development personnel, confirming and checking by the research and development personnel, checking whether the performance is reduced or not, if the performance is reduced, checking the reason and solving.
In the process of implementing the invention, the inventor finds that at least the following problems exist in the prior art:
whether a large number of false alarms exist in performance reduction is judged by setting a threshold value, a large amount of time is wasted for research personnel to investigate, the frequency of the false alarms is increased, the research personnel can reduce attention to alarm, and the real performance reduction can be missed.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for testing performance of a machine, so as to solve the technical problem of a large number of false alarms.
To achieve the above object, according to an aspect of an embodiment of the present invention, there is provided a method for testing performance of a machine, including:
repeatedly executing performance tests on the machine to obtain a plurality of target test results;
based on the central limit theorem, calculating the multiple target test results through performance reference values, and judging the performance of the machine according to the calculation results;
the performance reference value is obtained by adopting the following method:
and repeatedly executing the performance test on the machine to obtain a plurality of benchmark test results, thereby calculating to obtain the performance benchmark value.
Optionally, repeatedly performing the performance test on the machine to obtain a plurality of benchmark test results, so as to calculate the performance benchmark value, including:
repeatedly executing performance tests on the machine to obtain a plurality of benchmark test results;
and calculating to obtain a performance mean value and a performance standard deviation according to the plurality of test results.
Optionally, the number of the benchmark test results is more than 2 times the number of the target test results, and the number of the target test results is greater than or equal to 30.
Optionally, based on the central limit theorem, calculating the multiple target test results through performance reference values, and determining the performance of the machine according to the calculation results, where the method includes:
calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation and the number of the target test results;
and calculating the average value of the target test results, and comparing the performance threshold value with the average value of the target test results so as to judge the performance of the machine.
Optionally, calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation, and the number of the target test results, including:
subtracting a preset performance probability threshold value from 1 to obtain an accumulated probability;
mapping on a normal distribution diagram according to the cumulative probability to obtain a multiple of a standard deviation;
and calculating a performance threshold according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of the target test results.
Optionally, calculating a performance threshold according to the multiple of the standard deviation, the performance mean and the performance standard deviation includes:
solving the square root of the number of the target test result;
multiplying the performance average value by the multiple of the standard deviation and dividing the result by the square root of the target test result to obtain a performance difference value;
and subtracting the performance difference value from the performance average value to obtain a performance threshold value.
Optionally, comparing the performance threshold with a mean value of the target test results to determine the performance of the machine, including:
and if the average value of the target test results is smaller than the performance threshold value, sending an alarm to indicate that the performance of the machine is reduced.
In addition, according to another aspect of the embodiments of the present invention, there is provided an apparatus for testing performance of a machine, including:
the test module is used for repeatedly executing performance tests on the machine to obtain a plurality of target test results;
the judging module is used for calculating the multiple target test results through the performance reference values based on the central limit theorem and judging the performance of the machine according to the calculation results;
the performance reference value is obtained by adopting the following method:
and repeatedly executing the performance test on the machine to obtain a plurality of benchmark test results, thereby calculating to obtain the performance benchmark value.
Optionally, repeatedly performing the performance test on the machine to obtain a plurality of benchmark test results, so as to calculate the performance benchmark value, including:
repeatedly executing performance tests on the machine to obtain a plurality of benchmark test results;
and calculating to obtain a performance mean value and a performance standard deviation according to the plurality of test results.
Optionally, the number of the benchmark test results is more than 2 times the number of the target test results, and the number of the target test results is greater than or equal to 30.
Optionally, the determining module is further configured to:
calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation and the number of the target test results;
and calculating the average value of the target test results, and comparing the performance threshold value with the average value of the target test results so as to judge the performance of the machine.
Optionally, the determining module is further configured to:
subtracting a preset performance probability threshold value from 1 to obtain an accumulated probability;
mapping on a normal distribution diagram according to the cumulative probability to obtain a multiple of a standard deviation;
and calculating a performance threshold according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of the target test results.
Optionally, the determining module is further configured to:
solving the square root of the number of the target test result;
multiplying the performance average value by the multiple of the standard deviation and dividing the result by the square root of the target test result to obtain a performance difference value;
and subtracting the performance difference value from the performance average value to obtain a performance threshold value.
Optionally, the determining module is further configured to:
and if the average value of the target test results is smaller than the performance threshold value, sending an alarm to indicate that the performance of the machine is reduced.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including:
one or more processors;
a storage device for storing one or more programs,
when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any of the embodiments described above.
According to another aspect of the embodiments of the present invention, there is also provided a computer readable medium, on which a computer program is stored, which when executed by a processor implements the method of any of the above embodiments.
One embodiment of the above invention has the following advantages or benefits: because the technical means of calculating a plurality of target test results through the performance reference value based on the central limit theorem and judging the performance of the machine according to the calculation result is adopted, the technical problem of a large number of false alarms in the prior art is solved. The embodiment of the invention accurately judges the performance of the machine based on the central limit theorem, improves the alarm accuracy of the automatic performance test, improves the sensitivity of performance monitoring, and can effectively monitor the performance reduction to a smaller extent.
Further effects of the above-mentioned non-conventional alternatives will be described below in connection with the embodiments.
Drawings
The drawings are included to provide a better understanding of the invention and are not to be construed as unduly limiting the invention. Wherein:
FIG. 1 is a schematic diagram of a main flow of a method of testing machine performance according to an embodiment of the invention;
FIG. 2 is a schematic view of a main flow of a method of testing the performance of a machine according to one referential embodiment of the present invention;
FIG. 3 is a schematic view of a main flow of a method of testing the performance of a machine according to another referential embodiment of the present invention;
FIG. 4 is a schematic diagram of the main modules of an apparatus for testing machine performance according to an embodiment of the present invention;
FIG. 5 is an exemplary system architecture diagram in which embodiments of the present invention may be employed;
fig. 6 is a schematic block diagram of a computer system suitable for use in implementing a terminal device or server of an embodiment of the invention.
Detailed Description
Exemplary embodiments of the present invention are described below with reference to the accompanying drawings, in which various details of embodiments of the invention are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
The existing automatic performance test usually sets a threshold value in order to judge whether the performance is reduced, and an alarm is given when a certain operation does not reach the threshold value, so that the performance is prompted to be reduced. However, the conditions of the machine are different when the test is run each time, the result of the performance test is good when the load of the machine is light, the result of the performance test is poor when the load of the machine is heavy, the result of the performance test is likely to fail to reach the threshold value due to the condition of the machine, if the result of the performance test fails to reach the threshold value each time, a large number of false alarms are generated, a large amount of time is wasted for research and development personnel to perform investigation, the number of false alarms is increased, the research and development personnel can reduce the attention to the alarm, and the real performance degradation can be missed.
In order to reduce the problem of false alarm, some systems adjust the threshold value, so that the alarm is only given out when the performance of a single test result is greatly reduced, however, the method can only reduce the false alarm to a small extent, and cannot really solve the problem of false alarm, and can only detect the performance reduction to a large extent, so that the problem cannot be solved, and the result is worse.
Fig. 1 is a schematic diagram of a main flow of a method of testing machine performance according to an embodiment of the present invention. As an embodiment of the present invention, as shown in fig. 1, the method for testing the performance of a machine may include:
and 101, repeatedly executing performance tests on the machine to obtain a plurality of benchmark test results, so as to calculate and obtain a performance benchmark value.
First, the performance test is repeatedly executed for the machine for N times, the test result (such as the performance parameter of the throughput per second QPS) of each time is recorded, and then the performance reference value of the machine is calculated according to the test results. In the embodiment of the invention, different virtual machines with the same configuration can be selected to run the test, so that the distribution of the test result is in accordance with the actual situation as much as possible.
Optionally, step 101 may comprise: repeatedly executing performance tests on the machine to obtain a plurality of benchmark test results; and calculating to obtain a performance mean value and a performance standard deviation according to the plurality of test results. For example, 1000 performance tests may be repeatedly performed on the machine, the test result (i.e., benchmark test result) of each time is recorded, then the test result is fitted by using Common library of Apache (an Apache library, which may perform statistics and probability-related calculations), and it is found to be well in line with normal distribution, and the performance mean m and the performance standard deviation μ of the machine are calculated.
The running code is as follows:
it should be noted that, the execution times N of the performance test is not limited in the embodiment of the present invention, and may be 500 times, 2000 times, 3200 times, or the like, and the execution times may be determined according to actual needs by a tester, and then the performance mean m and the performance standard deviation μ are calculated according to the test result of each time.
And 102, repeatedly executing performance tests on the machine to obtain a plurality of target test results.
After the performance reference values (including the performance mean m and the performance standard deviation μ) are obtained, the performance test is repeatedly performed again on the machines of the same configuration, thereby obtaining a plurality of target test results. Optionally, the number (N) of the benchmark test results is more than 2 times the number (N) of the target test results, and the number of the target test results is greater than or equal to 30.
Statistically, the sample mean is perfectly matched when the size n of the sample is greater than 30 (m,
) Is turning toAnd (4) state distribution. If the conditions do not allow 30 times per day, the results of several days can be accumulated and calculated after 30 times. Optionally, each time the test is executed, the test result is written into a text file in an apend manner, when the file reaches n lines, it indicates that the performance test has been executed n times, at this time, step 103 is performed, and it is determined whether the performance has been degraded, and the file is emptied. Alternatively, n may be 30, 35, 40, 48, 50, or the like, which is not limited in this embodiment of the present invention.
And 103, calculating the multiple target test results through the performance reference values based on the central limit theorem, and judging the performance of the machine according to the calculation results.
After n performance tests are performed on the machine, the n performance tests are calculated based on the performance reference value obtained in the step 101, and the performance of the machine is judged according to the calculation result, so that whether an alarm is needed or not is judged.
Optionally, step 103 may comprise: calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation and the number of the target test results; and calculating the average value of the target test results, and comparing the performance threshold value with the average value of the target test results so as to judge the performance of the machine.
As can be seen from the central limit theorem, the sampling distribution of the sample mean is the distribution formed by all the sample means, the sampling distribution of the sample mean is symmetrical in shape, and as the sample amount n increases, the sampling distribution of the sample mean tends to be normal distribution whether the original population follows normal distribution or not, wherein the distribution is expected to be the population mean m, the variance is 1/n of the population variance, and the standard deviation is the population standard deviation
Optionally, comparing the performance threshold with a mean value of the target test results to determine the performance of the machine, including: if the average value of the multiple target test results is less than the performance thresholdA value, an alarm is issued to indicate a performance degradation of the machine. Assuming that the same performance test is run on the same machine at different times, the test results conform to a normal distribution, with an expectation of m and a standard deviation of μ. On this basis, according to the central limit theorem, if the same test is performed n times per day, the mean of the n test results obeys the expectation that m standard deviations are
Is normally distributed. The mean of the n test results is calculated after n times of daily execution, and the cumulative probability of the mean in the normal distribution can be calculated. If the probability is less than a certain set value, an alarm is issued.
Optionally, calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation, and the number of the target test results, including: subtracting a preset performance probability threshold value from 1 to obtain an accumulated probability; mapping on a normal distribution diagram according to the cumulative probability to obtain a multiple of a standard deviation; and calculating a performance threshold according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of the target test results. Optionally, calculating a performance threshold according to the multiple of the standard deviation, the performance mean and the performance standard deviation includes: solving the square root of the number of the target test result; multiplying the performance average value by the multiple of the standard deviation and dividing the result by the square root of the target test result to obtain a performance difference value; and subtracting the performance difference value from the performance average value to obtain a performance threshold value.
First, the text file in step 102 is read to obtain n test results, and then the mean m2 of the n test results is calculated. Since the mean value is in accordance with (m,
) The cumulative probability of m2 can be calculated according to the normal distribution of (1). The running code is as follows:
NormalDistribution normalDistribution＝new NormalDistribution(m,miu/Math.sqrt(30))；
double cumn＝normalDistribution.cumulativeProbability(m2)；
the performance threshold may be calculated using the following formula:
wherein, gateNum is a performance threshold, m is a performance mean, x is a multiple of a standard deviation obtained by mapping cumulative probability on a normal distribution diagram, mu performance standard deviation, and n is the number of target test results.
Thus, a new probability distribution is constructed, expected to be m, with a standard deviation of
Then, a performance probability threshold value is set, and when the probability of performance degradation reaches the value, an alarm is given.
Assuming that the set performance probability threshold is gate, the performance threshold gateNum can be calculated by the following procedure: informal distribution. invertepcumulant profile (1-gate), alarm if the actual mean (m2) is less than this value.
Assuming that m is 1000, μ is 100, and the threshold of performance probability is set to 84.1%, which means that if there is 84.1% of performance degradation probability, the alarm will be given, and the corresponding cumulative probability is 1-0.841-0.159, and since the value of cumulative probability 0.159 on the normal distribution diagram is 1 times of standard deviation, if the average value of n times is less than that of n times
It indicates that there is a 84.1% probability of performance degradation, at which time it should be alarmed. It should be noted that the threshold of 84.1% is only an example, and the actual system does not necessarily use this number and can be flexibly set.
Then, step 102 and step 103 may be performed periodically, that is, the performance test is performed periodically, and the test result of each time is recorded, when the recorded result reaches n times, the average value of the n test results is calculated, and if the number is smaller than the previously calculated performance threshold value gateNum, it indicates that the probability of performance degradation exceeds the gate, and an alarm is required.
The embodiment of the invention adopts the central limit theorem to judge whether the result of the automatic performance test indicates that the performance is reduced, and has the following advantages:
1) the alarm accuracy is greatly improved. Assume that the mean of 30 data at a time is
The corresponding probability of performance degradation is 84.1%. If the existing method is adopted and the result of each performance test is used for calculation, the probability of performance reduction corresponding to the number is 57.2%, and it can be seen that the alarm accuracy rate is remarkably improved by the embodiment of the invention.
2) The sensitivity of performance degradation monitoring is greatly improved. By adopting the existing method, if the alarm is required to be given when the probability of performance reduction reaches 90%, the alarm can be given only when the test result is less than 1.28 standard deviations. By adopting the method provided by the embodiment of the invention, the alarm can be sent out only if the test result is less than 0.23 standard deviation, and the sensitivity is greatly improved.
According to the various embodiments described above, it can be seen that the technical problem of a large number of false alarms in the prior art is solved by the technical means of calculating the performance reference values of a plurality of target test results based on the central limit theorem and judging the performance of the machine according to the calculation results in the embodiments of the present invention. The embodiment of the invention accurately judges the performance of the machine based on the central limit theorem, improves the alarm accuracy of the automatic performance test, improves the sensitivity of performance monitoring, and can effectively monitor the performance reduction to a smaller extent.
Fig. 2 is a schematic diagram of a main flow of a method of testing the performance of a machine according to one referential embodiment of the present invention. As another embodiment of the present invention, as shown in fig. 2, the method for testing the performance of a machine may include:
step 201, repeatedly executing N times of performance tests on the machine to obtain N reference test results.
First, the machine is repeatedly tested for N times, and the test results are recorded. Different virtual machines with the same configuration can be selected to run the test, so that the distribution of the test results is in accordance with the actual situation as far as possible.
And step 202, calculating to obtain a performance mean value and a performance standard deviation according to the N test results.
The test results can be fitted by using a Common math library of Apache, so that the performance mean value m and the performance standard deviation mu of the machine can be calculated.
And 203, calculating the performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation and the number n of target test results.
Specifically, a preset performance probability threshold is subtracted from 1 to obtain an accumulated probability; mapping on a normal distribution diagram according to the cumulative probability to obtain a multiple of a standard deviation; and calculating a performance threshold according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of the target test results. Alternatively, the performance threshold may be calculated using the following formula:
wherein, gateNum is a performance threshold, m is a performance mean, x is a multiple of a standard deviation obtained by mapping cumulative probability on a normal distribution diagram, mu performance standard deviation, and n is the number of target test results. Assuming that the same performance test is run on the same machine at different times, the test results conform to a normal distribution, with an expectation of m and a standard deviation of μ. On this basis, according to the central limit theorem, if the same test is performed n times per day, the mean of the n test results obeys the expectation that m standard deviations are
Is normally distributed. The mean of the n test results is calculated after n times of daily execution, and the cumulative probability of the mean in the normal distribution can be calculated. If this is the caseIf the individual probability is less than a certain set value, an alarm is sent out.
And step 204, repeatedly executing n times of performance tests on the machine to obtain n items of mapping test results.
And (3) periodically and repeatedly executing n times of performance tests on the machine, writing a test result into a text file in an apend mode when the test is executed each time, indicating that the n times of performance tests are executed when the file reaches n rows, entering the next step at this time, judging whether the performance is reduced, and emptying the file. Wherein the number (N) of the benchmark test results is 2 times or more of the number (N) of the target test results, and the number of the target test results is 30 or more.
Step 205, calculating a mean m2 of the n-entry mapping test results.
Step 206, comparing the performance threshold gateNum with the average m2 of the target test results, thereby determining the performance of the machine.
Then, step 204, step 205 and step 206 may be performed periodically, that is, the performance test is performed periodically, and the test result of each time is recorded, when the recorded result reaches n times, the average value of the n test results is calculated, and if the number is less than the previously calculated performance threshold value gateNum, it indicates that the probability of performance degradation exceeds the gate, and an alarm is required.
In addition, in one embodiment of the present invention, the detailed implementation of the method for testing the performance of the machine is described in detail above, and therefore the repeated description is not repeated here.
Fig. 3 is a schematic diagram of a main flow of a method of testing the performance of a machine according to another referential embodiment of the present invention. As another embodiment of the present invention, as shown in fig. 3, the method for testing the performance of a machine may include:
step 301, presetting a performance probability threshold and a number n of times for repeatedly executing a performance test.
And step 302, subtracting a preset performance probability threshold value from 1 to obtain an accumulated probability.
And step 303, mapping on the normal distribution diagram according to the cumulative probability to obtain a multiple of the standard deviation.
And step 304, calculating a performance threshold according to the multiple of the standard deviation, the performance average value of the machine, the performance standard deviation of the machine and the number n of times of repeatedly executing the performance test.
The performance mean value and the performance standard deviation of the machine can be obtained by adopting the following method: the machine is repeatedly tested for N times, and the test result of each time is recorded. Different virtual machines with the same configuration can be selected to run the test, so that the distribution of the test results is in accordance with the actual situation as far as possible. Then, a Common math library of Apache can be used for fitting the test results, so that the performance mean value m and the performance standard deviation mu of the machine are calculated.
Finally, the performance threshold is calculated using the following formula:
wherein, gateNum is a performance threshold, m is a performance mean, x is a multiple of a standard deviation obtained by mapping cumulative probability on a normal distribution diagram, mu performance standard deviation, and n is the number of target test results.
Assuming that the set performance probability threshold is gate, the performance threshold gateNum can be calculated by the following procedure: informal distribution. inverteCumulative Prohability (1-gate).
Optionally, the number (N) of benchmark test results is more than 2 times the number (N) of target test results.
And 305, repeatedly executing n times of performance tests on the machine to obtain n items of mapping test results.
Optionally, the number of the target test results is greater than or equal to 30. Statistically, the sample mean is perfectly matched when the size n of the sample is greater than 30 (m,
) Is normally distributed. If the conditions do not allow 30 times per day, it is also possibleTo accumulate results for several days and to calculate 30 times later. Optionally, each time the test is executed, the test result is written into a text file in an apend manner, when the file reaches n lines, it indicates that the performance test has been executed n times, at this time, step 103 is performed, and it is determined whether the performance has been degraded, and the file is emptied. Alternatively, n may be 30, 38, 42, or 55, etc., which is not limited in this embodiment of the present invention.
Step 306, calculating the mean value of the n items of mapping test results.
Step 307, judging whether the mean value of the n-entry mapping test results is smaller than the performance threshold value; if yes, go to step 308; if not, go to step 305.
At step 308, an alarm is issued to indicate a performance degradation of the machine.
In addition, in another embodiment of the present invention, the detailed implementation of the method for testing the performance of the machine is described in detail above, and therefore the repeated description is not repeated here.
FIG. 4 is a schematic diagram of the main modules of an apparatus for testing the performance of a machine according to an embodiment of the present invention, as shown in FIG. 4, the apparatus 400 for testing the performance of a machine includes a testing module 401 and a judging module 402; the test module 401 is configured to repeatedly perform a performance test on a machine to obtain a plurality of target test results; the judging module 402 is configured to calculate the multiple target test results through a performance reference value based on a central limit theorem, and judge the performance of the machine according to the calculation result; the performance reference value is obtained by adopting the following method: and repeatedly executing the performance test on the machine to obtain a plurality of benchmark test results, thereby calculating to obtain the performance benchmark value.
Optionally, repeatedly performing the performance test on the machine to obtain a plurality of benchmark test results, so as to calculate the performance benchmark value, including:
repeatedly executing performance tests on the machine to obtain a plurality of benchmark test results;
and calculating to obtain a performance mean value and a performance standard deviation according to the plurality of test results.
Optionally, the number of the benchmark test results is more than 2 times the number of the target test results, and the number of the target test results is greater than or equal to 30.
Optionally, the determining module 402 is further configured to:
calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation and the number of the target test results;
and calculating the average value of the target test results, and comparing the performance threshold value with the average value of the target test results so as to judge the performance of the machine.
Optionally, the determining module 402 is further configured to:
subtracting a preset performance probability threshold value from 1 to obtain an accumulated probability;
mapping on a normal distribution diagram according to the cumulative probability to obtain a multiple of a standard deviation;
and calculating a performance threshold according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of the target test results.
Optionally, the determining module 402 is further configured to:
solving the square root of the number of the target test result;
multiplying the performance average value by the multiple of the standard deviation and dividing the result by the square root of the target test result to obtain a performance difference value;
and subtracting the performance difference value from the performance average value to obtain a performance threshold value.
Optionally, the determining module 402 is further configured to:
and if the average value of the target test results is smaller than the performance threshold value, sending an alarm to indicate that the performance of the machine is reduced.
According to the various embodiments described above, it can be seen that the technical problem of a large number of false alarms in the prior art is solved by the technical means of calculating the performance reference values of a plurality of target test results based on the central limit theorem and judging the performance of the machine according to the calculation results in the embodiments of the present invention. The embodiment of the invention accurately judges the performance of the machine based on the central limit theorem, improves the alarm accuracy of the automatic performance test, improves the sensitivity of performance monitoring, and can effectively monitor the performance reduction to a smaller extent.
It should be noted that, in the embodiment of the apparatus for testing the performance of a machine according to the present invention, the above-mentioned method for testing the performance of a machine has been described in detail, and therefore, the repeated description is omitted here.
Fig. 5 illustrates an exemplary system architecture 500 for a method or apparatus for testing machine performance to which embodiments of the present invention may be applied.
As shown in fig. 5, the system architecture 500 may include terminal devices 501, 502, 503, a network 504, and a server 505. The network 504 serves to provide a medium for communication links between the terminal devices 501, 502, 503 and the server 505. Network 504 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
The user may use the terminal devices 501, 502, 503 to interact with a server 505 over a network 504 to receive or send messages or the like. The terminal devices 501, 502, 503 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 501, 502, 503 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 505 may be a server providing various services, such as a background management server (for example only) providing support for shopping websites browsed by users using the terminal devices 501, 502, 503. The background management server may analyze and otherwise process the received data such as the item information query request, and feed back a processing result (for example, target push information, item information — just an example) to the terminal device.
It should be noted that the method for testing the performance of the machine provided by the embodiment of the present invention is generally executed by the server 505, and accordingly, the apparatus for testing the performance of the machine is generally disposed in the server 505. The method for testing the performance of the machine provided by the embodiment of the present invention may also be executed by the terminal devices 501, 502, and 503, and accordingly, the device for testing the performance of the machine may be disposed in the terminal devices 501, 502, and 503.
It should be understood that the number of terminal devices, networks, and servers in fig. 5 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Referring now to FIG. 6, a block diagram of a computer system 600 suitable for use with a terminal device implementing an embodiment of the invention is shown. The terminal device shown in fig. 6 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present invention.
As shown in fig. 6, the computer system 600 includes a Central Processing Unit (CPU)601 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. In the RAM603, various programs and data necessary for the operation of the system 600 are also stored. The CPU 601, ROM 602, and RAM603 are connected to each other via a bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The following components are connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program performs the above-described functions defined in the system of the present invention when executed by the Central Processing Unit (CPU) 601.
It should be noted that the computer readable medium shown in the present invention can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present invention, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present invention, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The modules described in the embodiments of the present invention may be implemented by software or hardware. The described modules may also be provided in a processor, which may be described as: a processor includes a test module and a determination module, where the names of the modules do not in some way constitute a limitation on the modules themselves.
As another aspect, the present invention also provides a computer-readable medium that may be contained in the apparatus described in the above embodiments; or may be separate and not incorporated into the device. The computer readable medium carries one or more programs which, when executed by a device, implement the method of: repeatedly executing performance tests on the machine to obtain a plurality of target test results; based on the central limit theorem, calculating the multiple target test results through performance reference values, and judging the performance of the machine according to the calculation results; the performance reference value is obtained by adopting the following method: and repeatedly executing the performance test on the machine to obtain a plurality of benchmark test results, thereby calculating to obtain the performance benchmark value.
According to the technical scheme of the embodiment of the invention, because the technical means of calculating the multiple target test results through the performance reference value based on the central limit theorem and judging the performance of the machine according to the calculation result is adopted, the technical problem of a large number of false alarms in the prior art is solved. The embodiment of the invention accurately judges the performance of the machine based on the central limit theorem, improves the alarm accuracy of the automatic performance test, improves the sensitivity of performance monitoring, and can effectively monitor the performance reduction to a smaller extent.
The above-described embodiments should not be construed as limiting the scope of the invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions can occur, depending on design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A method of testing machine performance, comprising:
repeatedly executing performance tests on the machine to obtain a plurality of target test results;
based on the central limit theorem, calculating the multiple target test results through performance reference values, and judging the performance of the machine according to the calculation results;
the performance reference value is obtained by adopting the following method:
and repeatedly executing the performance test on the machine to obtain a plurality of benchmark test results, thereby calculating to obtain the performance benchmark value.
2. The method of claim 1, wherein repeatedly performing performance tests on the machine to obtain a plurality of benchmarking results to calculate a performance benchmark value comprises:
repeatedly executing performance tests on the machine to obtain a plurality of benchmark test results;
and calculating to obtain a performance mean value and a performance standard deviation according to the plurality of test results.
3. The method according to claim 2, wherein the number of pieces of the benchmark test results is 2 times or more the number of pieces of the target test results, and the number of pieces of the target test results is 30 or more.
4. The method of claim 2, wherein calculating the plurality of target test results by a performance benchmark based on a central limit theorem, and determining the performance of the machine based on the calculation comprises:
calculating a performance threshold according to a preset performance probability threshold, the performance mean, the performance standard deviation and the number of the target test results;
and calculating the average value of the target test results, and comparing the performance threshold value with the average value of the target test results so as to judge the performance of the machine.
5. The method of claim 4, wherein calculating a performance threshold based on a predetermined performance probability threshold, the performance mean, the performance standard deviation, and the number of target test results comprises:
subtracting a preset performance probability threshold value from 1 to obtain an accumulated probability;
mapping on a normal distribution diagram according to the cumulative probability to obtain a multiple of a standard deviation;
and calculating a performance threshold according to the multiple of the standard deviation, the performance mean, the performance standard deviation and the number of the target test results.
6. The method of claim 5, wherein calculating a performance threshold based on the multiple of the standard deviation, the performance mean, and the performance standard deviation comprises:
solving the square root of the number of the target test result;
multiplying the performance average value by the multiple of the standard deviation and dividing the result by the square root of the target test result to obtain a performance difference value;
and subtracting the performance difference value from the performance average value to obtain a performance threshold value.
7. The method of claim 4, wherein comparing the performance threshold to a mean of the plurality of target test results to determine the performance of the machine comprises:
and if the average value of the target test results is smaller than the performance threshold value, sending an alarm to indicate that the performance of the machine is reduced.
8. An apparatus for testing machine performance, comprising:
the test module is used for repeatedly executing performance tests on the machine to obtain a plurality of target test results;
the judging module is used for calculating the multiple target test results through the performance reference values based on the central limit theorem and judging the performance of the machine according to the calculation results;
the performance reference value is obtained by adopting the following method:
and repeatedly executing the performance test on the machine to obtain a plurality of benchmark test results, thereby calculating to obtain the performance benchmark value.
9. An electronic device, comprising:
one or more processors;
a storage device for storing one or more programs,
the one or more programs, when executed by the one or more processors, implement the method of any of claims 1-7.
10. A computer-readable medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010289797.7A CN113535486A (en) | 2020-04-14 | 2020-04-14 | Method and device for testing performance of machine |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010289797.7A CN113535486A (en) | 2020-04-14 | 2020-04-14 | Method and device for testing performance of machine |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113535486A true CN113535486A (en) | 2021-10-22 |
Family
ID=78120063
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010289797.7A Pending CN113535486A (en) | 2020-04-14 | 2020-04-14 | Method and device for testing performance of machine |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113535486A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101964997A (en) * | 2009-07-21 | 2011-02-02 | 中国移动通信集团黑龙江有限公司 | Method and device for carrying out early warning on network performance |
| CN103139007A (en) * | 2011-12-05 | 2013-06-05 | 阿里巴巴集团控股有限公司 | Method and system for detecting application server performance |
| CN104598361A (en) * | 2013-10-31 | 2015-05-06 | 华为技术有限公司 | Performance monitoring method and device |
| CN106201829A (en) * | 2016-07-18 | 2016-12-07 | 中国银联股份有限公司 | Monitoring Threshold and device, monitoring alarm method, Apparatus and system |
| US20170059475A1 (en) * | 2015-08-25 | 2017-03-02 | Bwt Property, Inc. | Variable Reduction Method for Spectral Searching |
| CN108959042A (en) * | 2017-05-19 | 2018-12-07 | 北京神州泰岳软件股份有限公司 | Base-line data calculation method in a kind of baseline management |
| CN109726086A (en) * | 2017-10-30 | 2019-05-07 | 北京京东尚科信息技术有限公司 | The method and apparatus of testing server performance |
| CN109783385A (en) * | 2019-01-14 | 2019-05-21 | 中国银行股份有限公司 | A kind of product test method and apparatus |
| CN110231447A (en) * | 2019-06-10 | 2019-09-13 | 精锐视觉智能科技（深圳）有限公司 | The method, apparatus and terminal device of water quality abnormality detection |
-
2020
- 2020-04-14 CN CN202010289797.7A patent/CN113535486A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101964997A (en) * | 2009-07-21 | 2011-02-02 | 中国移动通信集团黑龙江有限公司 | Method and device for carrying out early warning on network performance |
| CN103139007A (en) * | 2011-12-05 | 2013-06-05 | 阿里巴巴集团控股有限公司 | Method and system for detecting application server performance |
| CN104598361A (en) * | 2013-10-31 | 2015-05-06 | 华为技术有限公司 | Performance monitoring method and device |
| US20170059475A1 (en) * | 2015-08-25 | 2017-03-02 | Bwt Property, Inc. | Variable Reduction Method for Spectral Searching |
| CN106201829A (en) * | 2016-07-18 | 2016-12-07 | 中国银联股份有限公司 | Monitoring Threshold and device, monitoring alarm method, Apparatus and system |
| CN108959042A (en) * | 2017-05-19 | 2018-12-07 | 北京神州泰岳软件股份有限公司 | Base-line data calculation method in a kind of baseline management |
| CN109726086A (en) * | 2017-10-30 | 2019-05-07 | 北京京东尚科信息技术有限公司 | The method and apparatus of testing server performance |
| CN109783385A (en) * | 2019-01-14 | 2019-05-21 | 中国银行股份有限公司 | A kind of product test method and apparatus |
| CN110231447A (en) * | 2019-06-10 | 2019-09-13 | 精锐视觉智能科技（深圳）有限公司 | The method, apparatus and terminal device of water quality abnormality detection |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109471783B (en) | Method and device for predicting task operation parameters | |
| US20160224400A1 (en) | Automatic root cause analysis for distributed business transaction | |
| CN106874135B (en) | Method, device and equipment for detecting machine room fault | |
| EP3168748A1 (en) | System and method for monitoring performance of applications | |
| CN109976971B (en) | Hard disk state monitoring method and device | |
| CN110727560A (en) | Cloud service alarm method and device | |
| CN114238058A (en) | Monitoring method, apparatus, device, medium, and program product | |
| CN113342619A (en) | Log monitoring method and system, electronic device and readable medium | |
| CN110727563A (en) | Cloud service alarm method and device for preset customer | |
| CN112256548B (en) | Abnormal data monitoring method and device, server and storage medium | |
| CN116701123A (en) | Task early warning method, device, equipment, medium and program product | |
| CN114024867B (en) | Network anomaly detection method and device | |
| CN112148705A (en) | Data migration method and device | |
| CN113535486A (en) | Method and device for testing performance of machine | |
| CN112131077B (en) | Positioning method and positioning device for fault node and database cluster system | |
| CN112132722B (en) | Government hot line quantity trend abnormity determining method and device, electronic equipment and medium | |
| CN115202973A (en) | Application running state determining method and device, electronic equipment and medium | |
| CN111427878B (en) | Data monitoring alarm method, device, server and storage medium | |
| CN113254325A (en) | Test case processing method and device | |
| CN113760874A (en) | Data quality detection method and device, electronic equipment and storage medium | |
| CN114089712B (en) | Data processing method and device | |
| CN111679885A (en) | Method, device, medium and electronic equipment for determining virtual machine drift | |
| CN118331823B (en) | Method and system for managing and monitoring alarm of space engineering business operation log | |
| CN117130945B (en) | Test method and device | |
| CN113778978B (en) | Big data quality control method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |