WO2019175985A1

WO2019175985A1 - Performance evaluation device, performance evaluation method, and performance evaluation program

Info

Publication number: WO2019175985A1
Application number: PCT/JP2018/009802
Authority: WO
Inventors: 泰子日野
Original assignee: 三菱電機株式会社
Priority date: 2018-03-13
Filing date: 2018-03-13
Publication date: 2019-09-19
Also published as: JPWO2019175985A1

Abstract

A performance evaluation device (200), wherein a distribution recording unit (230) calculates the probability distribution of a processing time from the start of a process that corresponds to each request (600) in a server (100) when the server (100) is actuated. A pattern recording unit (220) records a second pattern different from a first pattern that is the arrival pattern of a request (600) in the server (100) when the server (100) is actuated. An evaluation unit (240) calculates, from the probability distribution of the processing time calculated by the distribution recording unit (230) and the second pattern recorded by the pattern recording unit (220), the probability distribution of a response time from the arrival of each request (600) in the server (100).

Description

Performance evaluation apparatus, performance evaluation method, and performance evaluation program

The present invention relates to a performance evaluation apparatus, a performance evaluation method, and a performance evaluation program.

In the technique described in Patent Document 1, an access pattern to the server during operation is recorded. After that, the load based on the recorded access pattern is replayed so that the access frequency increases, and the access frequency when the response time measured at the same time exceeds the required response time is the limit performance value of the server. As recorded.

JP 2004-318454 A JP 2017-224226 A

Generally, even when the average arrival rate of requests per unit time is equal, the response time for the request varies depending on the arrival pattern of the request. For this reason, in the performance test for confirming whether the information system to be evaluated satisfies the performance requirement, it is necessary to consider the influence of the arrival pattern in order to improve the test accuracy.

When using a method of regenerating the load based on the arrival pattern during operation as in the technique described in Patent Document 1, the arrival pattern of the request must be reproduced when performing the performance test, and the test efficiency is poor. Performance tests cannot be performed on arrival patterns of requests that cannot be reproduced.

The present invention aims to improve both the accuracy and efficiency of the performance test.

A performance evaluation apparatus according to an aspect of the present invention is provided.
Calculates the probability distribution of processing time from the start of processing according to each request in the server when the server that executes processing by repeatedly receiving requests is operated, and records the probability distribution of processing time in the memory A distribution recording unit,
A pattern recording unit that records, in the memory, a second pattern that is an arrival pattern different from a first pattern that is an arrival pattern of a request in the server when the server is operated;
An evaluation unit that calculates a probability distribution of response time from arrival of each request in the server from the probability distribution recorded by the distribution recording unit and the second pattern recorded by the pattern recording unit;

In the present invention, the probability distribution of the response time is calculated from the probability distribution of the processing time when the server is operated and the arrival pattern different from the arrival pattern of the request when the server is operated. Therefore, it is possible to perform a performance test in consideration of the influence of the arrival pattern without reproducing the arrival pattern. Therefore, according to the present invention, both the accuracy and efficiency of the performance test are improved.

1 is a block diagram showing an overall configuration of a system according to a first embodiment. FIG. 2 is a block diagram showing a hardware configuration of a server and a performance evaluation device according to the first embodiment. FIG. 2 is a block diagram showing a functional configuration of a server and a performance evaluation device according to the first embodiment. 5 is a flowchart showing operations of the server and the performance evaluation device according to the first embodiment. The flowchart which shows the detailed operation | movement of step S120 of FIG. The figure which shows the example of arrival time data. The figure which shows the example of arrival pattern data. The flowchart which shows the detailed operation | movement of step S150 of FIG. The figure which shows the example of processing time data. The figure which shows the example of processing time distribution data. The flowchart which shows the detailed operation | movement of step S170 of FIG. 5 is a flowchart showing a detailed operation of step S190 in FIG. FIG. 4 is a block diagram showing functional configurations of a server and a performance evaluation device according to Embodiment 2. 9 is a flowchart showing operations of a server and a performance evaluation device according to Embodiment 2. The flowchart which shows the detailed operation | movement of step S250 of FIG. The figure which shows the example of processing time data.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals. In the description of the embodiments, the description of the same or corresponding parts will be omitted or simplified as appropriate. The present invention is not limited to the embodiments described below, and various modifications can be made as necessary. For example, two or more embodiments among the embodiments described below may be combined and executed. Alternatively, among the embodiments described below, one embodiment or a combination of two or more embodiments may be partially implemented.

Embodiment 1 FIG.
This embodiment will be described with reference to FIGS.

*** Explanation of configuration ***
With reference to FIG. 1, the overall configuration of the system according to the present embodiment will be described.

The server 100 is a computer that is a target of a performance test, that is, an evaluation target. The server 100 includes an execution unit 110 and a measurement unit 120.

The execution unit 110 is an element that executes a process that is a target of a performance test. The measurement unit 120 is an element that receives an execution command from the performance evaluation apparatus 200 and measures the characteristics of the execution unit 110.

The performance evaluation apparatus 200 is a computer that performs a performance test. The performance evaluation device 200 is connected to the input device 300 and the output device 400.

The input device 300 is a device that receives input from the user and transmits the input to the performance evaluation device 200. The output device 400 is a device that outputs data received from the performance evaluation device 200 to a user.

The performance evaluation apparatus 200 issues an execution command to the measurement unit 120 and receives the arrival time data 500 from the measurement unit 120. The performance evaluation apparatus 200 transmits a request 600 to the execution unit 110 and receives processing time data 700 from the measurement unit 120. The performance evaluation device 200 transmits the result calculated based on the received data to the output device 400.

The server 100 and the performance evaluation apparatus 200 can be replaced with a virtual machine.

The server 100 may be composed of a plurality of computers when there are a plurality of execution units 110. In that case, each computer includes an execution unit 110 and a measurement unit 120.

The performance evaluation apparatus 200 may be the same computer as the server 100, but is preferably a separate computer so as not to affect the processing performance of the execution unit 110.

The input device 300 and the output device 400 are not essential components, but are useful for enhancing the effects of the present embodiment.

The configuration of the server 100 will be described with reference to FIG. 2 and FIG.

The server 100 includes a processor 101 and other hardware such as a memory 102, an auxiliary storage device 103, and a network interface 104. The processor 101 is connected to other hardware via a signal line, and controls these other hardware.

The server 100 includes an execution unit 110 and a measurement unit 120 as functional elements. The measurement unit 120 includes a first measurement unit 121 and a second measurement unit 122. The functions of the execution unit 110 and the measurement unit 120 are realized by software.

The processor 101 is a device that executes a server program. The server program is a program that realizes the functions of the execution unit 110 and the measurement unit 120. The processor 101 is a CPU, for example. “CPU” is an abbreviation for Central Processing Unit.

The memory 102 and the auxiliary storage device 103 are devices that store server programs. The memory 102 is, for example, RAM, flash memory, or a combination thereof. “RAM” is an abbreviation for Random Access Memory. The auxiliary storage device 103 is, for example, an HDD, a flash memory, or a combination thereof. “HDD” is an abbreviation for Hard Disk Drive. The auxiliary storage device 103 can be replaced with an external disk. In that case, the external disk is connected to the server 100 by the network interface 104.

The network interface 104 includes a receiver that receives data input to the server program and a transmitter that transmits data output from the server program. The network interface 104 is, for example, a communication chip or a NIC. “NIC” is an abbreviation for Network Interface Card.

The server program is loaded from the auxiliary storage device 103 to the memory 102 and executed by the processor 101.

The server 100 may include a plurality of processors that replace the processor 101. The plurality of processors share the execution of the server program. Each processor is, for example, a CPU.

Data, information, signal values, and variable values used, processed, or output by the server program are stored in the memory 102, the auxiliary storage device 103, or a register or cache memory in the processor 101. Such data includes arrival time data 500, request 600, and processing time data 700.

The server program is a program that causes a computer to execute the procedures performed by the execution unit 110 and the measurement unit 120 as an execution procedure and a measurement procedure, respectively. The server program may be provided by being recorded on a computer-readable medium, may be provided by being stored in a recording medium, or may be provided as a program product.

The server 100 may be composed of one computer or a plurality of computers. When the server 100 is configured by a plurality of computers, the execution unit 110 may be provided in at least one computer, but the measurement unit 120 is provided in each computer.

The configuration of the performance evaluation apparatus 200 will be described with reference to FIGS.

The performance evaluation apparatus 200 includes a processor 201 and other hardware such as a memory 202, an auxiliary storage device 203, a network interface 204, an input interface 205, and an output interface 206. The processor 201 is connected to other hardware via a signal line, and controls these other hardware.

The performance evaluation apparatus 200 includes an input unit 210, a pattern recording unit 220, a distribution recording unit 230, an evaluation unit 240, and an output unit 250 as functional elements. The pattern recording unit 220 includes a calculation unit 221 and a correction unit 223. The distribution recording unit 230 includes a transmission unit 231 and a calculation unit 232. The evaluation unit 240 includes a first calculation unit 241 and a second calculation unit 243. The functions of the input unit 210, pattern recording unit 220, distribution recording unit 230, evaluation unit 240, and output unit 250 are realized by software.

The processor 201 is a device that executes a performance evaluation program. The performance evaluation program is a program that realizes the functions of the input unit 210, the pattern recording unit 220, the distribution recording unit 230, the evaluation unit 240, and the output unit 250. The processor 201 is, for example, a CPU.

The memory 202 and the auxiliary storage device 203 are devices for storing a performance evaluation program. The memory 202 is, for example, RAM, flash memory, or a combination thereof. The auxiliary storage device 203 is, for example, an HDD, a flash memory, or a combination thereof. The auxiliary storage device 203 can be replaced with an external disk. In that case, the external disk is connected to the performance evaluation apparatus 200 via the network interface 204.

The network interface 204 includes a receiver that receives data input to the performance evaluation program and a transmitter that transmits data output from the performance evaluation program. The network interface 204 is, for example, a communication chip or a NIC.

The input interface 205 is an interface connected to the input device 300. The input interface 205 is a USB port, for example. “USB” is an abbreviation for Universal Serial Bus. The input device 300 is operated by a user to input data to the performance evaluation program. The input device 300 is, for example, a mouse, a keyboard, a touch panel, or some or all of these.

The output interface 206 is an interface connected to the output device 400. The output interface 206 is, for example, a USB port. The output device 400 presents data output from the performance evaluation program to the user. The output device 400 is, for example, a display such as an LCD, a printer, or a combination thereof. “LCD” is an abbreviation for Liquid Crystal Display.

The performance evaluation program is loaded from the auxiliary storage device 203 to the memory 202 and executed by the processor 201.

The performance evaluation apparatus 200 may include a plurality of processors that replace the processor 201. The plurality of processors share the execution of the performance evaluation program. Each processor is, for example, a CPU.

Data, information, signal values, and variable values used, processed, or output by the performance evaluation program are stored in the memory 202, the auxiliary storage device 203, or a register or cache memory in the processor 201. Such data includes arrival time data 500, request 600 and processing time data 700, as well as arrival pattern data 222, processing time distribution data 233, response time distribution data 242, performance characteristic value 244 and performance requirement data 260. is there.

The performance evaluation program is a computer in which the procedures performed by the input unit 210, pattern recording unit 220, distribution recording unit 230, evaluation unit 240, and output unit 250 are input procedure, pattern recording procedure, distribution recording procedure, evaluation procedure, and output procedure, respectively This is a program to be executed. The performance evaluation program may be provided by being recorded on a computer-readable medium, may be provided by being stored in a recording medium, or may be provided as a program product.

The performance evaluation apparatus 200 may be composed of a single computer or a plurality of computers. When the performance evaluation apparatus 200 is configured by a plurality of computers, the functions of the input unit 210, the pattern recording unit 220, the distribution recording unit 230, the evaluation unit 240, and the output unit 250 are realized by being distributed to each computer. May be.

The server 100 and the performance evaluation apparatus 200 are connected by

respective network interfaces

104 and 204.

*** Explanation of operation ***
The operations of the server 100 and the performance evaluation device 200 according to the present embodiment will be described mainly with reference to FIG. The operations of the server 100 and the performance evaluation apparatus 200 correspond to the performance evaluation method according to the present embodiment.

In step S110, the input device 300 confirms whether or not there is an execution command related to the arrival pattern recording to the performance evaluation device 200.

In step S120, the pattern recording unit 220 of the performance evaluation apparatus 200 records the arrival pattern of the request 600 received by the execution unit 110 of the server 100 during operation as a probability distribution.

Specifically, the pattern recording unit 220 has an arrival pattern different from the first pattern that is the arrival pattern of the request 600 in the server 100 when the server 100 that repeatedly receives the request 600 and executes the process is operated. A second pattern is recorded in the memory 202 as arrival pattern data 222.

As a method for determining the second pattern, any method may be used, but in the present embodiment, a method for determining the second pattern by calculating the first pattern and correcting the calculation result is used. .

In the present embodiment, the pattern recording unit 220 calculates the probability distribution of the arrival time intervals of the requests 600 in the server 100 from the data indicating the arrival times of the requests 600 in the server 100 as the first pattern. Then, the pattern recording unit 220 calculates, as the second pattern, a probability distribution having different characteristics such as average or variance from the calculated probability distribution of arrival time intervals.

As a method for determining the second pattern, a method of accepting an input of the second pattern via the input interface 205 is selectively used in the present embodiment.

As the data indicating the arrival time of each request 600 in the server 100, the arrival time data 500 is used as follows.

The first measurement unit 121 of the measurement unit 120 receives an execution command from the input unit 210, measures the time when the request 600 arrives at the execution unit 110, and transmits the measurement result as arrival time data 500. The calculation unit 221 of the pattern recording unit 220 receives the arrival time data 500 from the first measurement unit 121, calculates an arrival pattern based on the received data, and stores it in the memory 202 as arrival pattern data 222. The correction unit 223 of the pattern recording unit 220 corrects the arrival pattern data 222 based on the information received from the input unit 210 or stores the information received from the input unit 210 in the memory 202 as the arrival pattern data 222. . Note that the input unit 210 converts information input from the user via the input device 300 into elements in the measurement unit 120, elements in the pattern recording unit 220, elements in the distribution recording unit 230, and information in the evaluation unit 240. It distributes to the elements of and transmits.

In step S130, the input device 300 confirms to the performance evaluation device 200 whether there is an execution command related to recording of the processing time distribution.

In step S140, it is confirmed whether or not the performance evaluation apparatus 200 has already stored the result of measuring the processing time of the execution unit 110 of the server 100. If the result has already been saved, step S150 is skipped.

In step S150, the distribution recording unit 230 of the performance evaluation apparatus 200 calculates and records the probability distribution of the time required for the execution unit 110 of the server 100 to execute the process.

Specifically, the distribution recording unit 230 receives the request 600 repeatedly and operates the server 100 that executes the process. The probability distribution of the processing time from the start of the process according to each request 600 in the server 100 Is calculated. Then, the distribution recording unit 230 records the calculated probability distribution of the processing time in the memory 202 as the processing time distribution data 233.

An arbitrary method may be used as a method for calculating the processing time probability distribution, but in the present embodiment, a method for calculating the processing time probability distribution from the processing time measurement result in the server 100 is used. .

As the data indicating the measurement result of the processing time in the server 100, processing time data 700 is used as follows.

The transmission unit 231 of the distribution recording unit 230 receives the execution command from the input unit 210 and transmits the request 600 to the execution unit 110. Each time the execution unit 110 receives a request 600, the execution unit 110 executes a process that is a target of a performance test. The second measurement unit 122 of the measurement unit 120 receives an execution command from the input unit 210, measures the time required for executing the process by the execution unit 110, and transmits the measurement result as processing time data 700. Alternatively, the second measurement unit 122 receives an execution instruction from the input unit 210, acquires information for calculating the time required to execute the process in the execution unit 110, such as a start time and an end time of the process, The acquired information is transmitted as processing time data 700. The calculation unit 232 of the distribution recording unit 230 receives the processing time data 700 from the second measurement unit 122, calculates the processing time distribution based on the received data, and stores the processing time distribution data 233 in the memory 202.

In step S160, the input device 300 confirms whether or not there is an execution command regarding the calculation of the response time distribution and the performance characteristic value 244 to the performance evaluation device 200.

In step S170, the evaluation unit 240 of the performance evaluation apparatus 200 calculates a response time obtained by adding a waiting time to the processing time from the arrival pattern of the request 600 stored in step S120 and the processing time distribution stored in step S150. The distribution and the performance characteristic value 244 representing the response time characteristic are calculated, and the calculation result is stored.

Specifically, the evaluation unit 240 determines the response from the arrival of each request 600 in the server 100 from the probability distribution of the processing time recorded by the distribution recording unit 230 and the second pattern recorded by the pattern recording unit 220. Calculate the probability distribution of time. Then, the evaluation unit 240 records the calculated probability distribution of response time in the memory 202 as response time distribution data 242.

Any method may be used as the method for calculating the response time probability distribution. In this embodiment, the server 100 calculates the probability distribution of the waiting time from the arrival of each request 600 and the processing time A method of calculating the probability distribution of the response time from the probability distribution and the probability distribution of the waiting time is used. The waiting time is calculated from the probability distribution of the processing time and the second pattern using a queuing model.

In the present embodiment, the evaluation unit 240 further calculates an index value of the performance of the server 100 from the calculated probability distribution of response time. Then, the evaluation unit 240 records the calculated index value in the memory 202 as the performance characteristic value 244.

More specifically, the first calculation unit 241 of the evaluation unit 240 receives an execution command from the input unit 210, reads the arrival pattern data 222 and the processing time distribution data 233, and executes the execution unit 110 based on the read data. Response time distribution is calculated and stored in the memory 202 as response time distribution data 242. The second calculation unit 243 of the evaluation unit 240 reads the response time distribution data 242, calculates a performance characteristic value 244 such as an average or maximum response time based on the read data, and stores it in the memory 202.

In step S180, it is confirmed from the input device 300 to the performance evaluation device 200 whether a performance test result output execution command for an arrival pattern other than the arrival pattern of the request 600 stored in step S120 is issued. If such an execution instruction has been issued, step S110 and subsequent steps are repeated.

In step S190, the output unit 250 of the performance evaluation apparatus 200 outputs necessary items among the performance test results determined based on the response time distribution and the performance characteristic value 244 stored in step S170. .

Thus, in this embodiment, the output unit 250 outputs the performance index value of the server 100 calculated from the probability distribution of response time.

Specifically, the output unit 250 performs a performance test result determined based on the response time distribution data 242, the performance characteristic value 244, the response time distribution data 242, the performance characteristic value 244, and the performance requirement data 260. Are converted into a format that can be output by the output device 400 and presented to the user via the output device 400.

The first measuring unit 121 can be omitted when the arrival pattern input through the input device 300 is stored in the memory 202 as the arrival pattern data 222.

The detailed operation of step S120 in FIG. 4 will be described with reference to FIG.

In step S121, the input unit 210 determines whether the arrival pattern of the request 600 received by the execution unit 110 during operation can be measured. If measurement is not possible, steps S122 to S125 are skipped.

In step S122, the first measurement unit 121 measures the arrival time of the request 600 received by the execution unit 110 during operation, and adds the measurement result to the arrival time data 500. The arrival time data 500 is data representing the arrival time of the request 600 received by the execution unit 110 during operation in a format as shown in FIG.

In step S123, the first measurement unit 121 confirms whether the arrival time data 500 is sufficient for calculating the probability distribution of the arrival time interval that is the arrival pattern. If not, step S122 is repeated.

In step S124, the first measuring unit 121 transmits the arrival time data 500 to the calculating unit 221.

In step S125, the calculation unit 221 reads the time described in the arrival time data 500, calculates the difference from the previous time, and uses the difference as the arrival time interval to discretely calculate the probability distribution of the arrival time interval. calculate. Then, the calculation unit 221 stores the calculation result as arrival pattern data 222 in the memory 202. This calculation result corresponds to the first pattern. The arrival pattern data 222 is data representing a discrete probability distribution of arrival time intervals in a format as shown in FIG. When there are a plurality of arrival patterns, data in the format shown in FIG. 7 is stored in the memory 202 for each arrival pattern.

In step S126, the input unit 210 determines from the input received from the input device 300 whether the arrival pattern needs to be corrected or input. When correction is necessary, the input unit 210 converts the input received from the input device 300 and transmits the converted input to the correction unit 223. If correction or input is necessary, that is, if an arrival pattern correction is instructed by the user or an arrival pattern is input by the user, step S127 is the next step. In other cases, the operation of step S120 ends.

In step S127, the correction unit 223 determines whether to correct the arrival pattern data 222 or to add data to the arrival pattern data 222 based on the information received from the input unit 210. In the case of correction, step S128 is the next step. In the case of addition, step S129 becomes the next step.

In step S128, the correction unit 223 corrects the arrival pattern data 222 according to the information received from the input unit 210.

As the correction, arbitrary correction may be performed, but in the present embodiment, correction for increasing the average of the probability distribution of arrival time intervals is performed. As a specific example, when the discrete width is 1 and the average is increased by 2, the following processing is executed.
Step 1: The probability that the arrival time interval corresponds to 0 or 1 is set to 0.
Step 2: Set n = 2.
Step 3: Let the probability that the arrival time interval corresponds to n be the probability that the original arrival time interval corresponds to n-2.
Step 4: Increase n by 1.
Step 5: Repeat Step 3 and Step 4 until n-2 reaches the maximum value of the original arrival time interval.

As another correction, a correction that increases dispersion may be performed. Regardless of what correction is performed, the value of the arrival time interval is limited to 0 or more, so correction that gives a negative value is avoided. The corrected probability distribution of arrival time intervals corresponds to the second pattern.

In step S129, the correction unit 223 adds the information received from the input unit 210 to the arrival pattern data 222.

Referring to FIG. 8, the detailed operation of step S150 in FIG. 4 will be described.

In step S151, the transmission unit 231 transmits the request 600 to the execution unit 110.

In step S152, the second measurement unit 122 measures the time required for the execution unit 110 to process one request 600 and adds it to the processing time data 700 as the processing time. The processing time data 700 is data representing the time required for the execution unit 110 to process one request 600 in a format as shown in FIG.

In step S153, the second measurement unit 122 confirms whether or not the processing time data 700 is sufficient for calculating the probability distribution of the processing time. If not, step S151 and subsequent steps are repeated.

In step S154, the second measurement unit 122 transmits the processing time data 700 to the calculation unit 232.

In step S155, the calculation unit 232 discretely calculates the processing time probability distribution from the processing time data 700. Then, the calculation unit 232 stores the calculation result as processing time distribution data 233 in the memory 202. The processing time distribution data 233 is data representing a discrete probability distribution of processing time required for the execution unit 110 to process one request 600 in a format as shown in FIG.

The detailed operation of step S170 in FIG. 4 will be described with reference to FIG.

In step S171, the first calculation unit 241 reads the arrival pattern data 222 and the processing time distribution data 233.

In step S172, the first calculation unit 241 calculates a response time distribution using the read data, and stores the calculation result in the memory 202 as response time distribution data 242. Although not shown, the response time distribution data 242 is a similar format to the processing time distribution data 233, showing a discrete probability distribution of response time required for the execution unit 110 to respond after receiving one request 600. It is data represented by.

The calculation of the response time distribution may be performed by an arbitrary method, but in the present embodiment, it is performed by the following method.
Step 1: Considering the behavior of the execution unit 110 as G / G / 1 in the Kendall notation, the distribution of the waiting time is calculated using the Wiener-Hopf method and the Lindley equation. As a method for calculating the distribution of waiting time, for example, there is a method described in Non-Patent Document 1.
Step 2: Utilizing the fact that the response time is the sum of the processing time and the waiting time, a response time distribution is calculated by a composite product of random variables.

In step S173, the second calculation unit 243 reads the response time distribution data 242.

In step S174, the second calculation unit 243 uses the read data to calculate a numerical value described in the performance requirement such as an average or maximum response time as the performance characteristic value 244. Then, the second calculation unit 243 stores the calculation result in the memory 202.

The detailed operation of step S190 in FIG. 4 will be described with reference to FIG.

In step S191, the output unit 250 reads the performance characteristic value 244 and the performance requirement data 260 stored in the memory 202 in advance. The performance requirement data 260 is data indicating performance requirements.

In step S 192, the output unit 250 determines the performance test result using the performance characteristic value 244 and the performance requirement data 260.

In step S193, the output unit 250 converts the performance characteristic value 244 and the performance test result into a format that can be output by the output device 400, and transmits the conversion result to the output device 400. Then, the output device 400 outputs the information received from the output unit 250.

*** Explanation of the effect of the embodiment ***
In this embodiment, the probability distribution of response time is calculated from the probability distribution of processing time when the server 100 is operated and the arrival pattern different from the arrival pattern of the request 600 when the server 100 is operated. The Therefore, it is possible to perform a performance test in consideration of the influence of the arrival pattern without reproducing the arrival pattern. Therefore, according to the present embodiment, both the accuracy and efficiency of the performance test are improved.

In the present embodiment, the request 600 is determined from the processing time information measured by transmitting the request 600 in an arbitrary arrival pattern to the execution unit 110 and the specific arrival pattern information recorded in the arrival pattern data 222. The performance characteristic value 244 in the specific arrival pattern is calculated without being generated in the specific arrival pattern. Therefore, it is possible to efficiently obtain a performance test result considering the influence of the specific arrival pattern.

If the arrival patterns listed in the performance requirements deviate from the range that can be generated with limited resources, or if the arrival patterns cannot be recorded in advance, the conventional method can perform performance tests. Although it is impossible, the performance test can be performed according to this embodiment.

When there are many performance requirements and the response time in a large number of arrival patterns is to be confirmed, the conventional method requires many measurements, but one measurement is sufficient according to the present invention.

According to this embodiment, not only the average value of the response time can be calculated, but also the distribution characteristic such as the maximum value of the response time or the percentile value required for the performance test can be calculated and output.

*** Other configurations ***
In the present embodiment, the functions of the input unit 210, the pattern recording unit 220, the distribution recording unit 230, the evaluation unit 240, and the output unit 250 of the performance evaluation apparatus 200 are realized by software. The functions of 210, pattern recording unit 220, distribution recording unit 230, evaluation unit 240, and output unit 250 may be realized by a combination of software and hardware. That is, some of the functions of the input unit 210, the pattern recording unit 220, the distribution recording unit 230, the evaluation unit 240, and the output unit 250 may be realized by dedicated hardware, and the rest may be realized by software.

The dedicated hardware is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a logic IC, a GA, an FPGA, an ASIC, or some or all of these. . “IC” is an abbreviation for Integrated Circuit. “GA” is an abbreviation for Gate Array. “FPGA” is an abbreviation for Field-Programmable Gate Array. “ASIC” is an abbreviation for Application Specific Integrated Circuit.

Both the processor 201 and the dedicated hardware are processing circuits. That is, regardless of whether the functions of the input unit 210, the pattern recording unit 220, the distribution recording unit 230, the evaluation unit 240, and the output unit 250 are realized by software or a combination of software and hardware. Operations of the unit 210, the pattern recording unit 220, the distribution recording unit 230, the evaluation unit 240, and the output unit 250 are performed by a processing circuit.

Embodiment 2. FIG.
In the present embodiment, differences from the first embodiment will be mainly described with reference to FIGS.

*** Explanation of configuration ***
Since the overall configuration of the system according to the present embodiment is the same as that of the first embodiment, description thereof is omitted.

Since the configuration of the server 100 is the same as that of the first embodiment, description thereof is omitted.

With reference to FIG. 13, the configuration of the performance evaluation apparatus 200 according to the present embodiment will be described.

The configuration of the performance evaluation device 200 is almost the same as that of the first embodiment, but in this embodiment, the performance evaluation device 200 is stored in the memory 202, the auxiliary storage device 203, or a register or cache memory in the processor 201. As data, there is processing time pattern data 234.

*** Explanation of operation ***
Operations of the server 100 and the performance evaluation apparatus 200 according to the present embodiment will be mainly described with reference to FIG. The operations of the server 100 and the performance evaluation apparatus 200 correspond to the performance evaluation method according to the present embodiment.

In Embodiment 1, the time required for the execution unit 110 to process the request 600, that is, the processing time is measured. In the present embodiment, instead of measuring the processing time, the resource consumption rate of the execution unit 110 is measured, and the processing time is calculated from the measurement result using the technique described in Patent Document 2.

Patent Document 2 discloses a method for simply predicting performance by regarding the arrival process of a request as a Poisson process. In the present embodiment, by changing the mathematical expression to be used, the technique described in Patent Document 2 can be handled when the arrival pattern of the request is an arbitrary pattern.

Since the operation from step S210 to step S240 is the same as the operation from step S110 to step S140 in the first embodiment, description thereof will be omitted.

In step S250, the distribution recording unit 230 of the performance evaluation apparatus 200 calculates an average value of the processing time from the resource usage rate and the request arrival rate of the execution unit 110 of the server 100, and the calculated average value of the processing time and the advance The processing time distribution is calculated from the processing time pattern data 234 stored in, and recorded.

Specifically, the distribution recording unit 230 calculates the average processing time from the resource usage rate of the server 100 and the arrival rate of the request 600 in the server 100, and the processing time probability distribution from the average processing time. Determine the parameters. Then, the distribution recording unit 230 calculates the probability distribution of the processing time using the determined parameter, and records the calculated probability distribution of the processing time in the memory 202 as the processing time distribution data 233.

As a method for calculating the average processing time, a queue model is used to calculate the average processing time for each resource of the server 100 from the usage rate of the plurality of resources of the server 100 and the arrival rate of the request 600 in the server 100. A method of calculating and summing the calculated averages is used.

Processing time pattern data 234 is used as follows when determining the parameter of the processing time probability distribution from the total of processing time averages.

The transmission unit 231 of the distribution recording unit 230 receives the execution command from the input unit 210 and transmits the request 600 to the execution unit 110. Each time the execution unit 110 receives a request 600, the execution unit 110 executes a process that is a target of a performance test. The second measurement unit 122 of the measurement unit 120 receives an execution command from the input unit 210, measures the resource usage rate and the request arrival rate of the execution unit 110, and transmits the measurement result as processing time data 700. In the present embodiment, as the resource usage rate, a CPU usage rate and an I / O usage rate per unit time are measured. As the request arrival rate, the number of request arrivals per unit time is measured. The calculation unit 232 of the distribution recording unit 230 receives the processing time data 700 from the second measurement unit 122, calculates the processing time distribution based on the received data and the processing time pattern data 234, and the processing time distribution data 233. Stored in the memory 202. In the present embodiment, the processing time probability distribution has one parameter, specifically, the processing time average, but there may be two or more processing time probability distribution parameters. In that case, the function of the probability distribution and the relational expression between the parameters are combined and defined by the processing time pattern data 234. For example, in the case of normal distribution, there are two parameters, average and variance. In this case, the processing time pattern data 234 is data defining a relational expression such as a normal distribution function and variance = average × a.

Since the operation from step S260 to step S290 is the same as the operation from step S160 to step S190 in the first embodiment, the description thereof will be omitted.

The detailed operation of step S250 in FIG. 14 will be described with reference to FIG.

In step S251, the transmission unit 231 transmits the request 600 to the execution unit 110.

In step S252, the second measurement unit 122 measures the resource usage rate and the request arrival rate of the execution unit 110, and adds the measurement result to the processing time data 700. The resource usage rate is a rate at which resources are used per unit time. The request arrival rate is the number of requests 600 that arrive per unit time. The processing time data 700 is data representing the resource usage rate and request arrival rate of the execution unit 110 in a format as shown in FIG.

In step S253, the second measurement unit 122 confirms whether or not the processing time data 700 is sufficient to calculate the average resource usage per process. If not, step S251 and subsequent steps are repeated.

In step S254, the second measurement unit 122 transmits the processing time data 700 to the calculation unit 232.

In step S <b> 255, the calculation unit 232 uses the processing time data 700 to calculate an average resource usage time, which is a time during which resources are used on average for each process, for each resource. If the arrival rate of the request 600 is λ, the number of CPU cores is n, the CPU consumption rate is Ucpu, the average CPU usage time is Tcpu, the I / O consumption rate is Uio, and the average I / O usage time is Tio, the average CPU usage The time Tcpu and the average I / O usage time Tio are obtained by the following equations.
Tcpu = n × Ucpu / λ
Tio = Uio / λ
The CPU core number n is assumed to be held as known information by the calculation unit 232.

In step S256, the calculation unit 232 calculates an average value of virtual response times in each resource using the average resource usage time calculated in step S255. Specifically, the calculation unit 232 regards the CPU processing as the M / M / c model in Kendall's notation and the I / O processing as the M / M / 1 model, and the virtual response time in each resource The average value of is calculated. The original process is executed by combining a number of fine CPU processes and I / O processes, but for convenience, a single CPU process and a fine I / O process are grouped together for each request. The response time when approximated to one I / O process is a virtual response time. The average value of the virtual response time in each resource corresponds to the average of the processing time for each resource. The virtual average response time TRcpu for CPU processing is obtained by the following equation.
TRcpu = ((Tcpu × (λ × Tcpu) ⁿ ) / ((n−1)! × (n−λ × Tcpu) ² )) × P + Tcpu

The virtual average response time TRio of the I / O processing is obtained by the following equation.
TRio = Tio / (1-λ × Tio)

In step S257, the calculation unit 232 calculates an average value T of processing times. The average value T of the processing time is obtained by the following equation using the virtual average response time TRcpu of the CPU processing obtained earlier and the virtual average response time TRio of the I / O processing.
T = TRcpu + TRIo

In step S258, the calculation unit 232 reads the processing time pattern data 234 stored in advance. The processing time pattern data 234 is data representing a distribution shape of processing time per processing, such as an exponential distribution.

In step S259, the calculation unit 232 discretely calculates the processing time distribution from the average value of the processing time and the processing time pattern defined in the read processing time pattern data 234. Then, the calculation unit 232 stores the calculation result as processing time distribution data 233 in the memory 202.

*** Explanation of the effect of the embodiment ***
In the present embodiment, the resource usage rate and the request arrival rate of the execution unit 110 are measured, and the processing time is estimated from the measurement result, thereby eliminating the need for the execution unit 110 to measure the time required for processing the request 600. .

100 server, 101 processor, 102 memory, 103 auxiliary storage device, 104 network interface, 110 execution unit, 120 measurement unit, 121 first measurement unit, 122 second measurement unit, 200 performance evaluation device, 201 processor, 202 memory, 203 Auxiliary storage device, 204 network interface, 205 input interface, 206 output interface, 210 input unit, 220 pattern recording unit, 221 calculation unit, 222 arrival pattern data, 223 correction unit, 230 distribution recording unit, 231 transmission unit, 232 calculation unit 233 processing time distribution data, 234 processing time pattern data, 240 evaluation unit, 241 first calculation unit, 242 response time distribution data, 243 second calculation unit, 244 performance characteristics Value, 250 output unit, 260 performance requirements data, 300 input device, 400 output unit, 500 the arrival time data, 600 requests 700 processing time data.

Claims

Calculates the probability distribution of processing time from the start of processing according to each request in the server when the server that executes processing by repeatedly receiving requests is operated, and records the probability distribution of processing time in the memory A distribution recording unit,
A pattern recording unit that records, in the memory, a second pattern that is an arrival pattern different from a first pattern that is an arrival pattern of a request in the server when the server is operated;
A performance evaluation comprising: an evaluation unit that calculates a probability distribution of response time from arrival of each request in the server from the probability distribution recorded by the distribution recording unit and the second pattern recorded by the pattern recording unit apparatus.
The performance evaluation apparatus according to claim 1, wherein the pattern recording unit calculates the first pattern and determines the second pattern by correcting the calculation result.
3. The performance evaluation device according to claim 2, wherein the pattern recording unit calculates a probability distribution of request arrival time intervals at the server as the first pattern from data indicating an arrival time of each request at the server.
4. The performance evaluation apparatus according to claim 3, wherein the pattern recording unit calculates a probability distribution having different characteristics as the second pattern from the probability distribution of the arrival time intervals.
The performance evaluation apparatus according to claim 1, wherein the pattern recording unit receives an input of the second pattern via an input interface.
The evaluation unit calculates a probability distribution of waiting time from the arrival of each request in the server from the probability distribution of the processing time and the second pattern using a queue model, and the probability of the processing time The performance evaluation apparatus according to claim 1, wherein the probability distribution of the response time is calculated from the distribution and the probability distribution of the waiting time.
The performance evaluation apparatus according to any one of claims 1 to 6, wherein the distribution recording unit calculates a probability distribution of the processing time from a measurement result of the processing time in the server.
The distribution recording unit calculates an average of the processing time from a resource usage rate of the server and an arrival rate of a request at the server, and calculates a probability distribution parameter of the processing time from the average of the processing time. The performance evaluation apparatus according to claim 1, wherein the performance evaluation apparatus is determined.
The distribution recording unit calculates an average of the processing time for each resource of the server from a use rate of a plurality of resources of the server and an arrival rate of requests in the server using a queuing model. The performance evaluation apparatus according to claim 8, wherein a parameter of the probability distribution of the processing time is determined from an average sum.
10. The performance evaluation device according to claim 1, further comprising an output unit that outputs an index value of the server performance calculated from the probability distribution of the response time.
The server receives the request repeatedly and executes the process.
The performance evaluation device calculates a probability distribution of processing time from the start of processing according to each request in the server when the server is operated, and records the probability distribution of processing time in a memory,
When the performance evaluation device operates the server, the second pattern that is different from the first pattern that is the arrival pattern of the request in the server is recorded in the memory;
The performance evaluation method, wherein the performance evaluation device calculates a probability distribution of response time from arrival of each request in the server from the probability distribution of processing time and the second pattern.
On the computer,
Calculates the probability distribution of processing time from the start of processing according to each request in the server when the server that executes processing by repeatedly receiving requests is operated, and records the probability distribution of processing time in the memory Distribution recording procedure to
A pattern recording procedure for recording, in the memory, a second pattern that is an arrival pattern different from the first pattern that is an arrival pattern of a request at the server when the server is operated;
Performance for executing an evaluation procedure for calculating a probability distribution of response time from arrival of each request in the server from the probability distribution recorded by the distribution recording procedure and the second pattern recorded by the pattern recording procedure Evaluation program.