JP5012850B2 - Exclusive control method for clustered computer system - Google Patents

Description

  The present invention relates to a fault-tolerant computer system for constructing an application system, and in particular, when a computer program or operating system that is executing an application has a fault, a system that allows the running application to be transferred to another computer. The present invention relates to a computer system having a program having a switching function.

  A computer system that requires high reliability has a configuration including an active computer that executes a process (application) and a standby computer that takes over the process when a failure occurs in the active system. A procedure for taking over the processing from the detection of a failure occurring in the active system to the standby system is provided by the cluster program. Further, when the application uses data on the disk, the disk is configured to be shared between the active system and the standby system. In order for the standby system to take over processing in the event of a failure in the active system, determine the computer that will be the standby system from the computers that make up the cluster, and simultaneously use the resources used by the application and operating system (OS). It is necessary to take over resources that cannot be used (shared resources), for example, shared disks and IP addresses. Furthermore, even if a failure (network split) occurs that interrupts the path where the standby system monitors the failure of the active system, it is guaranteed that the active system and the standby system do not use shared resources at the same time. It is necessary to achieve higher reliability.

  In a cluster configuration, a method for determining a standby system to take over processing by taking over a shared disk exclusively is performed by many cluster programs. Examples of such methods include Document 1 and Patent Document 2. In Patent Document 1, a shared resource is released by using a mechanism for stopping the active system from the standby system, so that the standby system resets the active system to release the shared resources owned by the active system. A technology for exclusive control of a shared resource by owning a resource is described.

  Next, Non-Patent Document 1 discloses a technique in which a cluster program uses the SCSI commands RESERVE and RESET to exclusively control access authority to a shared disk when a failure occurs in the active system and the system is switched. be written. Here, RESERVE is a command for reserving access authority to a disk, and a disk reserved for RESERVE by a computer is a command for denying access and RESERVE from other computers. On the other hand, RESET is a command for releasing the access authority of the disk, and a command for releasing the access authority of the reserved disk.

JP 10-207855 A Shared disk type multiplex system

Microsoft support technical information 309186

  In Patent Document 1, in a cluster-structured computer system, when the active system can no longer be monitored, the active system is stopped to realize exclusive control of the shared resource. However, in a two-cluster cluster where each other is the other's standby system, if a network split occurs, all the other systems may be reset because they try to reset each other's system. There is. Therefore, there is a problem that processing is interrupted at the time of network split, and high availability cannot be realized.

  The standby system resets the active system, but the active system does not reset the standby system. Therefore, when considering a cluster composed of a working system and two standby systems (standby system 1 and standby system 2) that take over it, two clusters consisting of the active system and the standby system 1 and In a case where the system 2 is separated from the system 2, the standby system 2 resets the active system and performs system switching. On the other hand, when the active system is reset by the standby system 2, the standby system 1 also detects a failure of the active system and executes system switching. Therefore, there is a problem that the standby system 1 and the standby system 2 become the active system at the same time, and double access of the shared resource occurs.

  Further, in Non-Patent Document 1, in a computer system having a cluster configuration, when the active system cannot be monitored, the standby system forcibly releases the control right of the active system for the shared disk by the SCSI command RESET. Processing, and after that, any standby system issues a SCSI RESERVE command to acquire the control right of the shared disk that has been released. By these two processes, the system that takes over the shared disk, that is, processing The system to take over is determined. Here, if the latter processing by RESERVE is invalidated by the former processing by RESET, an excessive system in which one standby system takes over the processing once taken over by RESERVE. In order to prevent this from happening, switching is sufficient to guarantee that all standby systems have finished issuing the RESET command between the former RESET processing and the latter RESERVE processing. Time is required. Therefore, there is a problem that the system switching time is delayed for a fixed time regardless of whether or not a network split actually occurs.

  In this method, even if a network split occurs, system switching can be performed, but further processing is required for taking over shared resources other than shared disks, for example, taking over IP addresses. There is a problem that the time required for completing the system switching increases and the system switching time is delayed.

  The present invention is a high availability computer system including a working / standby computer, wherein the working / standby system shares at least one resource, and each computer monitors each other's faults, and a mutual heartbeat path. In a computer system having a reset path for stopping the system, a function for realizing system switching is provided. For example, the shared resource is a disk device.

  The present invention has an index (reset priority) for determining the order in which each computer issues a reset command for resetting another system. The reset priority is a value uniquely determined in the computer system. For example, the priority is determined in the order of the IP address of each computer. Each computer in the cluster other than the active system sets a delay time until reset issuance in the timer based on the reset priority of its own computer when a failure occurs that makes it impossible for the active computer to monitor.

  When detecting the elapse of the reset delay time set by the timer, the system issues a reset to the active system (failed system) that has failed. The reset delay time set for each system has a time difference at which each system does not reset at the same time. For example, during the reset delay time of each system, it is preferable to perform a reset and use a certain time difference that occurs until it is determined that the reset has succeeded or failed.

  Next, when the faulty system is reset by a certain system, the operation of the faulty system is stopped and the use of the shared resource is terminated. This operation stop processing may be, for example, power off or OS shutdown. Furthermore, the system that issued the reset (reset system) notifies the other system of the reset of the faulty system. The system that has received the reset notification of the faulty system stops the reset timer for the faulty system. In this way, the same machine is prevented from being reset multiple times.

  After the faulty system is reset, the standby system that takes over the processing of the faulty system takes over the shared resource and takes over the processing performed in the faulty system. Here, the determination of the standby computer may be performed by a reset system that resets the fault system, or may be performed by the reset system notifying another computer.

  In this way, in a clustered computer system, if there is a normally operating system, the faulty system is always reset by the system having the highest reset priority among the systems. Since there is always one standby system that should take over the process, including the reset system, a high availability system is realized in which the process is taken over by switching over to the standby system.

In the Example of this invention, it is a high-level system block diagram of the computer system model of a cluster structure in case system switching is performed. In the Example of this invention, it is the figure which showed the structure of the cluster state management table which a cluster program manages. In the Example of this invention, it is a processing flowchart explaining the process which a cluster program performs at the time of monitoring of another system. In the Example of this invention, it is a processing flowchart explaining the process in which a cluster program performs system switching. In the Example of this invention, it is the figure which showed the structure of the priority definition which defines the priority which a cluster program resets. In the Example of this invention, it is a sequence diagram explaining the timing which a cluster program resets.

  The drawings and descriptions relating to the present invention have been simplified to show appropriate elements for a clear understanding of the present invention, and known elements have been omitted so far as they do not hinder the practice of the invention. I want you to understand. Some of the prior art in the art may have other elements desirable and / or required to implement the present invention. However, these elements in the art are known and will not be described here as they do not facilitate understanding of the present invention. The following is a detailed description with reference to the accompanying drawings.

  FIGS. 1 and 2 simply show the system blocks of the working / standby computer in the present invention. For ease of explanation, a 4-digit number is used as the label for each program. The numbers of the active computer and standby computer use the same last three digits for the same program, and the thousands are 1 for the active computer (system A) and the standby computer (system B, system). In (C), they are represented by 2, 3 respectively. Below, each program is demonstrated previously. In this description, the program of each computer is described by the program number on the active computer, but it also serves as the description of the corresponding program on the standby computer. Further, in FIG. 2, the same numbers are used in the following, but these are the same as those in FIG. 1 unless otherwise described.

  In FIG. 1, the system A has network adapters (NICs) 1103, 1104, 1105 and a reset unit 1106 as devices for transmitting and receiving communications with the outside. The software program includes an operating system (OS) 1105, an application 1102, and a cluster program 1110. The NIC 1103 is used for the application 1102 to communicate with the outside, the NIC 1104 is used for communication for the cluster program 1110 to monitor each other system, and the NIC 1105 is used for the cluster program 1110 to reset the other system. . Here, NICs 1103 and 1104 are separated for understanding, but may be the same.

  Next, the reset unit 1106 has a function of receiving a reset from the other cluster programs 1210 and 1310 and stopping the system A. This system stop function is realized by forcibly stopping the OS 1105, for example. The cluster program 1110 has three modules. Hereinafter, the cluster program will be described with reference to FIG.

(1) The monitoring unit 1111 has a function of monitoring whether the own-system application 1102 operates normally, and a function of monitoring the status of the other-system cluster programs 1210 and 1310 via the communication unit 1112.
(2) The communication unit 1102 communicates with the other system cluster programs 1210 and 1310 via the NIC 1104, and issues a reset instruction to the other system reset units 1206 and 1306 in response to a request from the system switching unit 1113. It has a communication function to perform.

  (3) Based on the status of each system obtained by the monitoring unit 1111, the system switching unit 1113 has a function for instructing the system in which a failure has occurred via the previous communication unit 1112 and a reset. When successful, the communication unit 1112 is used to notify the other cluster programs 1210 and 1310 of the successful reset of the faulty system. In addition, when the failure of resetting the faulty system is notified from the cluster program 1210 or 1310 of another system, if the local system is a standby system that needs to take over the faulty system processing, it can take over the faulty system processing. And have. More detailed processing of the cluster programs 1110, 1210, and 1310 will be described later with reference to FIG.

Further, the system switching unit 1113 has a cluster state management table 1114 for managing the state of the cluster. FIG. 2 shows a cluster state management table according to the present invention. In FIG. 2, the cluster state management table has five pieces of information.
(1) a system identifier 21 for uniquely identifying each system;
(2) a system state 22 representing a state monitored by the monitoring unit 1111;
(3) Reset priority 23 indicating the order in which each system issues resets,
(4) When resetting the other system, a reset destination identifier 24 for identifying the reset unit 1206, 1306 of the other system that is the communication destination of the reset instruction,
(5) A reset timer 25 that indicates at which timing the reset of each system is instructed.

  Here, the reset timer 25 stores a delay time (reset delay time) from when the fault system is detected until the own system issues a reset to the fault system. When a certain failure occurs and, for example, the heartbeat detection of the system C and the system becomes impossible, the reset delay time set for each timer in each system has a difference. That is, the individual reset delay time is stored for each system so that the issuing timing of the reset instruction triggered by each of the plurality of systems has a time interval. In the embodiment, specifically, the reset delay time for each system is stored so that the reset is performed in the order based on the reset priority 23.

  For example, if a system with one priority higher than its own system gives a reset instruction and the time difference is more than a certain level that guarantees that the reset will be performed, it can be guaranteed that the reset will be performed according to the reset priority 23 . Therefore, the reset delay time set in each system timer may be set to have a time difference for each system in accordance with the system hardware in accordance with the reset priority set in each system. The reset interval definitions 1116, 1216, and 1316 hold information on this time difference. In another method, the reset delay time of the system may be directly set by the user for each of the reset interval definitions 1116, 1216, and 1316.

  Further, the reset priority 23 may be a value that is uniquely determined and does not overlap in all systems constituting the cluster. For example, when the user gives statically, there is a method of giving the cluster program 1110 with the priority definition 1115. FIG. 5 shows the priority definition 1115. The priority definition 1115 includes a system identifier 51 and a reset priority 52. The system identifier 51 may be the same value as the system identifier 21, or may be a value uniquely corresponding to the total identifier 21.

  Also, the reset priority 52 may be the same value as the reset priority identifier 23, or any value that uniquely determines the reset priority 23 using the reset priority 52. . For example, when the IP address is used as the reset priority 52, the reset priority 23 is uniquely determined by using the magnitude relationship. Further, in addition to the static assignment given by the prior term priority definition 1115 by the user, a method dynamically determined by the cluster program may be used. In this case, the information included in the priority definition 1115 may be used.

  Furthermore, addition and deletion of systems in the state management table 1114 are performed as follows. First, when the cluster program 1110 starts monitoring the new cluster program by the monitoring unit 1111, a new system is added to the table. On the other hand, when the cluster program 1110 resets the failed system by the system switching unit 1113, or when another cluster program notifies the cluster program 1110 that the failed system has been reset, Is deleted.

  In the present embodiment, for ease of explanation, an example in which the system switching unit 1113 has one cluster state management table 1114 is shown, but there are some pieces of information 21 to 25 included in the cluster state management table. The table may be divided and managed, or may be in a cluster program other than the system switching unit. 3 and 4 show the processing flow of the cluster program according to the present invention. FIG. 3 shows a system fault monitoring operation centered on the monitoring unit 1111. FIG. 4 shows the system switching unit 1113. This represents a system switching operation. Hereinafter, it will be described in detail with reference to FIGS.

  First, when the cluster program 1110 is executed, it monitors each other's faults. First, the monitoring unit 1111 periodically communicates with other cluster programs 1210 and 1310 via the communication unit 1112 to monitor the other system. The state of each system obtained in step 31 is registered in the system state 22 corresponding to the system identifier 21 in the state management table 1114 (step 301).

  Next, by referring to the system state 22 of each system obtained in the step 301, it is determined whether or not there is a faulty system (failed system) (step 302). If there is no faulty system, the entire system is operating normally, so the process returns to step 301 and the fault monitoring is continued periodically. On the other hand, if there is a faulty system, the monitoring unit 1111 calls the system switching unit 1113 (dotted line in the figure), and step 401 of the system switching unit is executed (step 303). After executing Step 303, the monitoring unit 1111 returns to Step 301 again, and again monitors the status of other systems in the cluster.

On the other hand, the instruction to the system switching unit performed in step 303 is determined in step 401. In step 401, it is determined whether or not a new failure system occurs. If there is a new failure system, the system switching unit 1113 refers to the reset priority 23 in the management table 1114 (step 402). Subsequently, based on the reset priority, the time until the own system resets the failed system is set in the reset timer 25 of the state management table 1114 (step 403), and step 404 is performed.
On the other hand, if there is no new fault system in step 401, step 404 is performed as it is.

  In step 404, it is determined whether or not the reset timer time set in step 403 has elapsed. If it is time to issue a reset, the communication unit 1112 communicates with the other system reset units 1206 and 1306 indicated in the reset destination identifier 24 of the system identifier 21 to be reset from the state management table 1114. The reset is instructed via (step 405). Here, the reset units 1206 and 1306 that have received the reset stop the use of the shared resource by stopping the operation of the own system. The method of stopping the operation of the own system may be, for example, power off, software reset, OS shutdown, or processing that hangs up the OS.

  After successful resetting at step 405, the system switching unit 1113 notifies the other system cluster program that the faulty system has been reset via the communication unit 1112 (step 406). Further, the reset timer 25 of the faulty system reset from the state management table 1114 is cleared (step 407), and the process returns to step 401 again.

  On the other hand, if the reset timer time set in step 403 has not yet elapsed in step 404, it is subsequently determined whether or not reset completion has been performed from another system (step 408). This is because when there is another system with a higher reset priority than the own system, the cluster system of the other system executes steps 404 to 407 first, so the faulty system may already be reset. It is. Therefore, if there is a reset completion notification from the cluster program of another system, clear the reset fault system reset timer (step 407), and do not reset the reset fault system again. Return to step 401.

  On the other hand, if there is no reset completion notification from the other system cluster program in the above step 408, the faulty system has not been reset yet, so nothing is done and the process returns to step 401 again. FIG. 6 is a diagram showing the timing at which the reset realized by the present invention is performed. The vertical axis in FIG. 6 indicates that time has elapsed, and represents reset processing over time. 6 represents the shared resource owner, that is, which system is the active system, and the right of FIG. 6 represents reset processing in each system.

  For simplification of explanation, FIG. 6 shows a case where a network split occurs in a cluster consisting of three units of system A, system B, and system C in descending order of reset priority. When a network split occurs at time T0, time T1, T2, and T3 are set for the reset timers of system A, system B, and system C, respectively, based on the reset priority. If each system is normal, the system is reset at a set time for the system that is in trouble as viewed from the own system.

  For example, since the system B and the system C are faulty when viewed from the system A, the system A resets each at time T1 (arrows 601 and 602). If a failure that prevents system A from being reset has occurred at the same time, system B resets at time T1 (arrows 603 and 604). Similarly, if a failure has occurred so that the system B cannot be reset, the system C resets at time T2 (arrows 605 and 606). The system B owns the shared resource until T3, and the system C owns the shared resource from time T3 to T4, and system switching is performed. Since the time after time T4 is a case where the reset path is not normal, it is out of the scope of the present invention. According to the embodiment of the present invention described above, there are the following effects.

  First, in a high-availability computer system with a cluster configuration that has a reset path, if the heartbeat for system monitoring is lost, there is a system that has been reset by resetting the faulty system according to the reset priority. Since it is guaranteed that only the system in the cluster uses the shared resource, system switching can be realized even during a network split. Furthermore, if the system with the highest reset priority does not become a failure, the system can immediately reset the failed system, thereby realizing high-speed system switching.

  Furthermore, when the user defines the priority definition statically in the cluster program, it is possible to realize system switching in which the reset priority is freely set and the system switching destination is controlled. Furthermore, when the user defines the reset interval in the cluster program, the timing at which the reset is performed can be controlled.

  According to the present invention, when the active computer can no longer be monitored, system switching can be realized in a cluster in which a system having a high reset priority exists. It is expected to be widely implemented as a system.

1101,1201,1301 Computer 1102,1202,1302 Application 1103,1104,1105,1203,1204,1205,1303,1304,1305 Network adapter (NIC) 1106,1206,1306 Reset section 1107,1207,1307 Operating system (OS ) 1110, 1210, 1310 Cluster program 1111, 1211, 1311 Monitoring unit 1112, 1212, 1312 Communication unit 1113, 1213, 1313 System switching unit 1114, 1214, 1314 Cluster status management table 1115, 1215, 1315 Priority definition 1116, 1216 , 1316 Reset interval definition 21,51 System identifier 22 System status 23,52 Reset priority 24 Reset destination identifier 25 Reset timer.

Claims (6)

A plurality of computers have a shared resource and are connected to each other by a heartbeat path for monitoring operations of other computers and a reset path for stopping other computers, and one of the plurality of computers is When executing processing using the shared resource as an execution system, exclusive control of the shared resource is performed, and the other computer is an exclusive control method of a cluster configuration computer functioning as a backup system of the execution system computer,
Any of the plurality of computers
Procedure 1 for storing priority information indicating a unique priority indicating which computer resets other computers in the cluster, respectively .
When detecting another computer whose heartbeat detection by the heartbeat path has been interrupted as a faulty system, it issues a reset to the faulty system based on the priority by referring to the stored priority information Procedure 2 to set the reset delay time until
When the reset delay time set in the procedure 2 elapses, the failure system is reset, and the procedure 3 that takes over the processing of the failure system;
Have a common procedure that includes
In the procedure 2, the exclusive control method is characterized in that the reset delay time is set shorter for the computer having the higher priority.   2. The exclusive control method according to claim 1, wherein, when the procedure 3 is executed, the computer notifies the other computer of the completion of the reset of the faulty system.   2. The exclusive control method according to claim 1, wherein in the procedure 1, the priority is determined in response to designation of an order of systems to be reset. 3.   The exclusive control method according to claim 1, wherein the priority information is determined based on an IP address of the computer.   The exclusive control method according to claim 1, wherein the priority information is information set by a user.   The reset delay time is invalidated when the reset of the fault system is completed by another computer between the time when the computer detects the occurrence of the failure and the time when the reset delay time elapses. The exclusive control method according to claim 1, wherein:

Images