JP2000259502A - Method and device for transferring write cache data for data storage and data processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the overhead concerning the communication, etc., between a calculation node and the storage media and to improve both system speed and efficiency by using a specific protocol of an effective data write cache of a distributed architecture. SOLUTION: Each of calculation resources is defined by a calculation node 200 and has a processor 216 which executes an application 204 that is managed by an OS 202. Then each of storage resources 104 is defined by a clique and includes a 1st I/O node or an ION 212 and a 2nd I/O node or an ION 214 which are set opposite to an inter-connect fabric 106 respectively via a system inter-connect 228 in terms of operation. In the ION 212, a write request including the write data is received from the node 200. Then the write data are transferred to the ION 214 from the ION 212 and a confirmation message is transmitted to the node 200 from the ION 214.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates generally to computing systems, and more particularly, to a method and apparatus for transferring write cache data in a data storage and data processing system. This provides a single point of view of virtual storage allocation in operation, regardless of processor or memory cabinet boundaries.

[0002]

BACKGROUND OF THE INVENTION Technology evolution is often the result of a series of seemingly unrelated technological developments. Each of these unrelated developments may be significant, but when combined, they can form the basis for a major technology evolution. Historically, components of large and complex computer systems have experienced uneven technological growth. This includes, for example, (1) disk I /
It includes rapid CPU performance advances compared to O performance, (2) evolving internal CPU architecture, and (3) interconnect fabric.

Over the past decade, the rate at which disk I / O performance has improved has been much slower than that of nodes. While CPU performance has increased at a rate of 40% to 100% per year, only 7% per year has improved disk seek time. If this trend continues as expected, the number of disk drives a typical server node can run will be such that disk drives will be the most influential component in both the volume and value of most large systems. To rise. This phenomenon has already become a reality in the introduction of existing large-scale systems.

[0004] Non-uniform performance improvements also occur in CPUs. To improve CPU performance, CPU vendors have used a combination of increasing clock speeds and architectural changes. Many of these architectural changes are proven technologies influenced by the concurrency community. These changes may result in unbalanced performance and may not be as good as expected. As a simple example, the rate at which a CPU can handle an interrupt has not increased at the same rate as that of a basic instruction. Therefore, system functions (such as I / O) that depend on interrupt performance have not improved with computing power.

[0005] Interconnect fabrics have also been characterized by uneven technology growth. This has been at a performance level of about 10-20 MB / sec for many years,
100MB / s (or more) in bandwidth over the past year
A major leap in levels has also taken place.

[0006] This significant performance improvement has made it possible to economically deploy large scale parallel processing systems.

[0007] This uneven performance adversely affects application architecture and system configuration options. For example, in terms of application performance, attempts to increase the amount of work to take advantage of performance improvements in some parts of the system, such as increasing CPU performance, are due to the lack of comparable performance improvements in the disk subsystem. Often inhibited. CP
Even if U can double the number of data operations performed per second, the disk subsystem can handle only a part of the increase. The overall effect of uneven hardware performance growth is that application performance is increasingly dependent on specific job characteristics.

[0008] The uneven growth of platform hardware technology has also created another serious problem. A reduction in the number of options available for configuring a multi-node system. A good example of this is the change in storage interconnect technology due to TERADA
This is the effect on the software architecture of TE4 Node Creek. The TERADATE creek model expected uniform storage connections between nodes in a single creek. In this case, it is possible to access each disk drive from all nodes. Therefore, when one node stops, the storage for that node can be divided by the remaining nodes. Due to the uneven growth of storage and node technology,
In a storage sharing environment, the number of disks that can be connected to one node is limited. This limitation is caused by the number of drives that can be connected to one I / O channel and the physical number of buses that can be connected in a four-node shared I / O topology. As node performance continues to improve, we must increase the number of disk spindles connected to a node in order to achieve performance improvements.

The design of clusters and massively parallel processing (MPP) is an example of a multi-node system design that attempts to solve the above-mentioned problems. Clusters suffer from limited scalability, and MPP systems require additional software to provide a sufficiently simple application model (in commercial MPP systems, this software is usually DBMS). MPP systems also require a form of internal clustering (clique) to provide very high availability. Both solutions still face challenges in managing the potential for a proliferation of disk drives. Such disk drives are electromechanical devices and have a sufficiently predictable failure rate. In the MPP system, since the number of nodes is usually large, the problem of the interconnection of the nodes is further exacerbated. Both approaches also suffer from disk connectivity issues, again due to the large number of drives required to store very large databases.

[0010] The above problems are ameliorated in an architecture in which the storage device and the computing device perform calculations in a high performance connection fabric and act as architectural equivalents. This architecture allows for increased flexibility in managing storage and computing resources. However, this flexibility presents some unique problems. One of these issues is ensuring secure data storage while maintaining the speed and flexibility provided by this architecture.

In conventional architectures, write cache techniques allow for efficient data storage. Data normally written to disk by the CPU is first written to the write cache. This data is then written to disk during the CPU idle cycle. This technique improves performance because writing to the write cache is faster than writing to disk or RAM.

There are certain risks associated with using a write cache for a disk. This is because data will remain in the volatile memory of the disk device for a long time before being written to the disk medium. This typically takes at most a few seconds, but if the crash or system failure occurs before the data is written to non-volatile storage, the data can be lost.

[0013] The write cache can also be used in highly distributed architectures. However, introducing a write cache protocol in such an architecture requires a large amount of communication and processing overhead between the compute nodes and the storage media, reducing the speed and efficiency of the system.

[0014]

SUMMARY OF THE INVENTION It is an object of the present invention to provide an efficient data write cache protocol in a distributed architecture to remedy the above-mentioned disadvantages.

[0015]

According to a first aspect, the present invention is a method for transferring write cache data in a data storage and data processing system, comprising the steps of: Receiving a write request from a computing node, transferring write data from a first I / O node to a second I / O node, and transmitting a confirmation message to the second I / O node.
Transmitting from the node to the computing node.

After the data has been written to the non-volatile storage of the first I / O node, the data is written to the second I / O node to remove the write data from the volatile memory of the second I / O node. An exclusion request or command is sent. In some embodiments, the eviction request is not sent until the first I / O node receives the second write request, in which case the eviction request is sent in the same interrupt as the write data of the second write request. . The data processing system comprises first and second I / O nodes, each node having a means for receiving a write request from a compute node and transferring the write node to another I / O node. are doing. In addition, each I / O node is provided with an I / O
Instead of sending the confirmation message via the O-node, there is also a means for sending the confirmation message directly to the computing node. The result is an I / O protocol that improves storage speed and turnaround by introducing a write cache while reducing the number of interrupts required to store data.

According to a second aspect, the present invention is an apparatus for transferring write cache data in a data storage and data processing system, wherein the first I / O node calculates a write request including the write data. Means for receiving from the node, and writing data to the first I / O
An apparatus comprises: means for transferring an O node to a second I / O node; and means for transmitting a confirmation message from the second I / O node to a computation node.

According to yet another aspect, the present invention provides a method for performing a step of transferring write cache data in a data storage system readable by a computer and according to claims 1 to 7,
The present invention resides in a program storage device that reliably implements one or more instruction programs executable by a computer.

[0019]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below with reference to the accompanying drawings.

A. Overview FIG. 1 is an overview of the same-row data processing architecture of the present invention. The architecture 100 includes one or more compute resources 102 and one or more storage resources 10
4, the storage resource 104 includes one or more interconnect fabrics 106 and a communication path 1
08 and a communication pair with the computational resource 102. The fabric 106 provides a communication medium for all nodes and storage, thus allowing for uniform, equal access between the computing resources 102 and the storage resources 104.

In the architecture of FIG. 1, since the storage is in a modern node-centric architecture, it is not limited to a single set of nodes, but each node can communicate with all storage. This is because the physical system topology limits storage and node communication,
In contrast to today's multi-node systems, where different topologies are often required depending on the workload.
Because the architecture of FIG. 1 provides a single physical architecture that supports a wide range of system topologies, the communication patterns of the application software determine the topology of the system at any time, resulting in uneven technology growth. It is possible to accept. Due to the separation provided by fabric 106,
Detailed scaling for each major system component is possible.

FIG. 2 shows a detailed description of the parallel architecture of the present invention. Computational resources are defined by one or more compute nodes 200, each having one or more processors 216 executing one or more applications 204 under the control of an OS 202. Peripheral devices 208 such as tape drives, printers, and other networks are operationally paired with the computing node 200. Furthermore, a local storage device 210 such as a hard disk for storing information unique to the computing node 200, such as the OS 202, the application 204, and other commands constituting information, is operationally paired with the computing node 200. Application instructions are stored and executed on one or more compute nodes 200 in the form of distributed processing. In an embodiment, the processor 206 includes the INTEL P
6 and an off-the-shelf off-the-shelf general-purpose processor and associated memory and I / O elements.

Storage resources 104 are defined at cliques 226, each of which has a first I / O node or ION 212 and a second I / O node or ION2.
14, each of which is interconnected by the system interconnect 228 in the interconnect fabric 1.
05 in operation. The first ION 212 and the second ION 214 are operationally paired with one or more storage disks 224 (also known as a "group of disks" or JBOD), which is accompanied by an IBOD enclosure 222. .

FIG. 2 shows a medium-sized system,
The ON 212 and the computation nodes have a typical ratio of 2: 1. The clique 226 of the present invention has three or more IOs.
A single ION 212 may be installed if the N214 is installed or if the availability of the storage node is insufficient. The number in creek 226 is purely a software problem, since no hardware is shared within ION 212. The paired IONs 212 are sometimes called “dipoles”.

The components of the present invention include a computing node 20
0, ION212, interconnect fabric 1
06 is also included.

The connection between the ION 212 and the JBOD 222 is as follows.
Here, it is shown in a simple format. The actual connection uses Fiber Channel cables for each tier (row, here four rows) of storage disks 224 in the configuration shown. In practice, it is expected that the number of storage disks managed by each ION 212 will be 40-80 instead of 20, as in the illustrated embodiment.

B. ION (storage node) Internal Architecture a) Hardware Architecture FIG. 3 is a detailed diagram of the configuration of the ION 212 and the interface with the JBOD 222. Each ION 212
Is JBOD via JBOD Interconnect 216
I / O connection module 302 for making communication connection with storage disk 224 in the array 222, ION 212
CPU and memory 304, which perform the functions of the ION physical disk driver 500 described in this document.
The power supply module 306 supplies power to support the operation of the ION 212.

B) JBOD FIG. 4 is a diagram showing details of the JBOD enclosure 222. All components of the monitor or controllable JBOD enclosure 222 are element 4
02-424. All elements 402-424 of any JBOD enclosure are returned with the configuration page code via the Receive Diagnostic Result command. The ION 212 uses the ordered list of the elements to number the elements. The first element 402 in the figure is element 0, the second element 404 is element 1, and so on. These element numbers are used when the management service layer 706 described in this document creates a LUN_C used for the address of the component.

[0029]

[Table 1]

Within the enclosure, element locations are identified by rack, chassis, and element numbers, as shown in Table I above. The rack number is a number inside the dipole assigned to the rack belonging to the dipole. The chassis position represents the height reported by the cabinet management device. The element number is the index of the element list returned by the SES configuration page. These fields form the LUN_C format.

C) I / O interface driver
Architecture FIG. 5 shows an ION including an ION physical disk driver 500 that acts as a “SCSI driver” for the ION 212.
FIG. 2 illustrates an I / O architecture of an I / O module 212. ION
The physical disk driver 500 is an I / O from a RAID (redundant disk redundant array) software driver or a management utility of the system administrator 230.
Manages fetching of requests, and JBOD Interconnect 2
The request is executed on the device on the side of the 16 devices.

The physical disk driver 500 of the present invention includes three major components. These are a high-level driver (HDL) 502, a device-specific high-level driver 504, and a low-level driver 506. HL
D502 includes a common part 503, a device-specific high-level part 504, and a low-level driver 506.
Common and device specific high level drivers 502 and 50
4 is adapter independent and requires no modification for new adapter types. The Fiber Channel Interface (FCI) low level driver 506 supports Fiber Channel adapters and is therefore protocol-specific rather than adapter-specific.

The FCI low level driver 506 is a SCSI
Translates the request into an FCP frame, Login
And common services such as Process Login. FCI low level driver 5
Operationally paired with 06 is a hardware interface module (HIM) interface 508, which separates Fiber Channel protocol operations from adapter-specific routines. A detailed description of the above components is provided below.

(1) High Level Driver The High Level Driver (HLD) 502 is the entry point for all requests to the ION 212, regardless of the type of device being accessed. When a device is opened, HLD 502 binds a command page to the device. These vendor-specific command pages are:
Indicate how to form a SCSI command descriptor block for a particular SCSI function. The command page allows the driver to more easily support devices that handle a particular SCSI function differently than specified by the SCSI specification.

(A) Common (Non-Device Specific) Portion The common portion of the HLD 502 includes the following entry points.

Cs_init Initializes the driver structure and allocates resources.

Cs_open Prepare to use the device.

Completion of cs_close I / O,
Remove a device from service.

Cs_strategy Block device read / write entries (Buf_strategy)
t interface).

Cs_intr Supports hardware interrupts.

These routines perform the same function on any device type. For most of these routines, the type of device (disk, tape, WOR
M, CD-ROM, etc.) and calls device specific routines,
Handles device-specific requirements.

The cs_open function ensures that the device is present and ready for I / O work there. Unlike current system architectures, this intersection 503 is based on the operating system (O
Do not create a table of known devices during initialization in S). Instead, the driver common part 503 performs self-configuration. The driver common part 503 determines the state of the device while the device is first opened. This allows
The driver common part 503 allows to "verify" devices that will be online after the OS 202 initialization phase.

During the initial open, the SCSI device is associated with a command page by issuing a SCSI inquiry command to the target device. If the device responds positively, the response data (including information such as vendor ID, product ID, firmware revision level, etc.) is compared in the SCSI configuration module 516 to a table of known devices. If a match is found, the device is specifically tied to the command page specified by the table entry. If no match is found, the device will send a generic CCS (Common Command Set) or SCS based on the format of the response data.
Unclearly tied to the II command page. The driver common part 503 includes routines and command page functions used by the low-level driver 506 to allocate resources, create a DMA list for distributed collection work, and complete SCSI work.

All FCI low level driver 506 routines are called from the driver common part 503.
Driver common part 503 is the only layer that initiates SCSI work by calling the appropriate low-level driver (LLD) routines in the Hardware Interface Module (HIM) to set up the hardware and begin work. is there. The LLD routine is also accessed by the SCSI configuration module 516 via the switch table indicated by the driver ID assigned during configuration.

(B) Device-specific part The interface between the common part 502 and the device-specific routine 504 is the same as the interface with the common part, and is csxx_intt, csxx_open, csxx.
_close, csxx_strategy commands are included. The designated location of “xx” indicates the type of the storage device (“dk” for disk or “tp” for tape). These routines handle any device-specific requirements. For example, if the device is a disk, csdk_
open reads the information of the partition table from a specific area of the disk, and reads csdk_state.
gy will use the information in the partition table to determine whether the block is outside the boundary.
(The partition table defines the block mapping from logical disk to physical disk for each specific physical disk.) (C) High-level driver error / failover processing (i) Error processing (a) Retry HLD502 The most common rehabilitation measures include failed I
Utilize / O retry. The number of retries for a particular command type is specified by the command page. For example, read and write commands are considered very important, so the associated command page may set the retry count to three. The retry count may be set to zero because the query command is not critical and continual retries during the first task of the day can slow down the system.

When a request is issued for the first time, its retry count is set to zero. Each time the request fails and the recovery scheme is repeated, the number of retries increases. If the number of retries exceeds the maximum number of retries specified on the command page, the I / O will fail and a message will be returned to the requestor. If not, a request is reissued. The only exception to this rule is for unit attention, which is usually an event notification rather than an error. If a unit attention is received as a command and its maximum number of retries is set to 0 or 1, high-level driver 502 sets the maximum number of retries for this I / O to two. In this way, it is possible to prevent the I / O failure from being accelerated due to the influence of the unit attention state.

Delayed retries are treated similarly to the retry scheme described above, except that retries are not replaced on the queue for a fixed amount of time.

(B) Sc issued from the failed Scsi_op FCI low-level driver 506
si_op may fail under some circumstances. Table II shows that the FCI low level driver 506 is HLD
502 indicates the type of failure that can be returned.

[0049]

[Table 2]

(C) Resource Shortage Resource shortage errors occur when some desired resources are not available at the time of the request. Typically, such resources are system memory and driver structure memory.

The processing for the system memory shortage is performed by using a semaphore
Done by blocks. Threads that block memory resources prevent all new I / Os from being issued. This thread continues to block until the I / O has completed and the memory is free.

The driver structure resource is Scsi_op
And an I / O vector (IOV) list pool. The IOV list is a list of values of the start value and the length of the memory transmitted and received by the disk. This memory pool is initialized at the beginning of the day with adjustable parameters that specify the size of the pool. Scsi_o
If p or IOV pools are empty, new I / Os will increase these pools. One page (4096 bytes) of memory is allocated at a time to increase either pool. All Scsi on new page
This page will not be free until the _op or IOV pool is free. ION 212 is Scsi_o
If you are allocating and freeing pages from p, or if you are constantly allocating and freeing pages,
Adjustment of the relevant parameters is desirable.

All resource shortage actions are recorded through events.

(Ii) First Processing of the Day At the beginning of the day, the HLD 502 initializes the necessary structures and pools and makes calls to initialize adapter-specific drivers and hardware. The first processing of the day begins with a call to cs_init (), which consists of (1) Scsi_Op pool allocation, (2) I
FCIhw_int for allocating the OV pool and (3) initializing the Fiber Channel structure and hardware
(4) Interrupt service routine cs_
Intr () is linked with the corresponding interrupt vector.

(Iii) Failover Processing Both dipole halves of the ION 212 are connected to a common disk device set. Dipole 22
6 IONs 212 and 214 must always be able to access all devices. From the perspective of the HLD 502, there is no special failover process.

(2) Command Page The ION 212 of the present invention uses a command page method that abstracts a common part and a device-specific part from the actual assembly of SCSI commands. The command page is a list of pointers to functions, and each function is a SCSI command (SCSI
_2_Test_UnitReady). As described above, a device is associated with a specific command page upon first opening or upon accessing the device. Vendor-specific or non-compliant special SCSI devices are managed by functions referenced on the device specific command page. Normal systems are shipped with Command Control Set (CCS), SCSI
Contains I and SCSI II pages, and vendor-specific pages, allowing integration of non-compliant SCSI devices or vendor-specific SCSI commands.

The command page function is implemented from the device common part 503, the device specific part 504, and the FCI low level driver 506 (Request Sense) through an interface called a virtual device (VDEV) interface. At this level, the software only matters whether the device is performing its intended function, not what SCSI language the device uses.

Each command page function assembles SCSI commands and allocates memory for direct memory access (DMA) data transfers, if necessary. This function then returns control to the driver common part 503. Driver common part 503 queues the SCSI work (sort here if necessary) and invokes the start routine of FCI low level driver 506 to implement this command. After the command has been executed, if a "call on interrupt" (COI) routine is present in the command page function, the COI is called before the driver common part 503 of the driver checks the data / information of the completed command. It is. By manipulating the reply data / information, the COI does not match the SCSI data /
The information can be converted to standard SCSI data / information. For example, if the device query data includes a vendor ID that starts with byte 12 instead of byte 8, the query command page function includes a COI that changes the reply query data vendor ID to byte 8. Driver common part 503 is always the vendor ID starting with byte 8
Extracting information eliminates the need to know about unmatched devices.

(3) IBOD and SCSI Configuration Module An important function of a RAID controller is to protect data from loss. To perform this function, RAID
Software must physically know where the disk devices are and how their cables are connected. Therefore, an important requirement in implementing RAID controller techniques is the ability to control the configuration of the storage device.

JBOD and SCSI configuration module 51
6 JBOD part is ION212 static IB
There is a task of defining the OD configuration. JBOD and SCSI
The configuration information described by the configuration module 516 is shown in Table III.

[0061]

[Table 3]

Adapter, JBOD enclosure 22
2. In addition to the physical location information of the storage disk 224, other configuration information such as the entry point of the FCI low-level driver 506 and the driver device specific part 504 and the definition of the command page must be described. To provide this information, use space. c file is used and I
The ON 212 assembles the configuration information when compiling the ION physical disk driver 500. Supported ION
If the configuration of 212 changes, a new version of I
The ON physical disk driver 500 must be compiled.

(4) Fiber Channel Interface (FCI) Low Level Driver The FCI low level driver 506 is a high level driver 50
2 for managing the SCSI interface. The interface between the driver common part 503 and the FCI low-level driver 506 includes the following routines. In addition, the FCI low-level driver 5
06 contains an identifier unique to the hardware controlled (eg, FCIhw_init).

Xxhw_init Initializes the hardware.

Xxhw_open Determines the current status of the host adapter.

Xxhw_config Set up host adapter configuration information (SCSI ID, etc.).

Xxhw_start If possible,
Start SCSI work.

Xxhw_intr All SCSs
Handle the I interrupt.

The low-level driver does not recognize or make a problem with the specification of the device, and simply executes S from the upper level.
In that it is a path for CSI commands, pure SC
It is an SI driver. This layer includes interrupt service routines, hardware initialization, mapping, address translation, and error recovery routines. In addition, multiple types of low-level drivers can coexist in the same system. Such a driver hardware management layer and other divisions allow the same high-level driver to run on different machines.

The basic function of the FCI module is (1) SC
SCSI to translate SI Op into FCI work object structure (I / O block (IOB))
Interface with high level driver (SHLD)
(2) providing a common interface to facilitate support of new Fiber Channel adapters through different HIMs 508; (3) any FC-4 protocol layer (Fibre Channel Protocol (F
Provision of FC-3 common services used by CP)),
(4) Asynchronous command (FCP command, FC-
3 commands, LIP commands), (5) (a) I / O request block (IOB), (b) vector table, (c) HIM508 resource (host adapter memory, DMA channel, I
/ O port, scratch memory, etc.)
(6) Resource management of the entire channel driver (FCI and HIM)
Arbitrate (for channel fabric)
Optimization of Loop Use A list of important data structures of the FCI low level driver 506 is shown in Table IV below.

[0071]

[Table 4]

(A) Error Processing Errors handled by the FCI low-level driver 506 tend to be errors specific to the Fiber Channel or FCI itself.

(I) Multi-Step Error Processing The FCI low-level driver 506 handles a specific error in a multi-stage processing. This allows optimizing error handling techniques depending on the type of error. For example, if a non-destructive procedure is not effective, more radical error handling techniques may be used.

(Ii) Failed IOBs All I / O requests are passed through the I / O request block
Sent to HIM 508. The following are the errors that HIM 508 may return.

[0075]

[Table 5]

(Iii) Resource Shortage The FCI low-level driver 506 manages the IOB resource pool and vector table. Since the size of such a pool is tuned to the configuration of the ION 212, there should not be a possibility of running out of such resources, so a simple recovery procedure is performed.

If an IOB or vector table request is made and there are not enough resources to satisfy this request, the I / O
Back to the queue and set a timer to retry the I / O. The occurrence of resource shortage is recorded.

(B) First Processing of the Day At the beginning of the day, the high-level driver 502 makes a call to each of the low-level drivers it supports (including the FCI low-level driver 506). The first process of the day for the FCI low level driver 506 is the FC
Beginning with a call to the Ihw_int () routine, which does the following: First, for the specified PCI bus and device, HIM_FindControlle
The r () function is called. This is FindContr
Call the version of urller (). The JBOD and SCSI configuration module 516 uses the PCI
Specify bus and device. Next, the adapter (ADAP
If found (such as available from the TEC), allocate the HCB and initialize it for the adapter. Then, in order to obtain adapter-specific resources such as a scratch memory, a memory map I / O, and a DMA channel,
Call HIM_GetConfigurationION (). Next, resources are allocated and initialized, and ADAP
H to initialize TEC HIM and hardware
Call IM_Initialize (). Finally,
Initialize by allocating IOB and vector table.

(C) Failover Process Both dipole halves of the ION 212 are connected to a common disk device set. Both ION2
12 must always be able to access all devices. From the perspective of the FCI low-level driver 506, there is no special failover process.

(5) Hardware Interface Module (HIM) Hardware Interface Module (HIM)
M) 508 is designed to interface with ADAPTEC's SlimHIM 509. HIM5
The 08 module is primarily responsible for translating requests from the FCI low level driver 506 into requests that the SlimHIM 509 can understand and issue to hardware. This includes I / O blocks (IOB)
Includes request capture and translation to the corresponding Transfer Control Block (TCB) request that the SlimHIM 509 can understand.

The basic functions of the HIM 508 include (1) a low-level application program interface (AP) for hardware-specific functions for searching, configuring, initializing, and sending I / O to the adapter.
Definition of I), (2) I / O block is SlimHIM /
Hardware understandable TCB requests (FC primitive TCB, FC extended link service (ELS) TCB,
(3) Interface with FCI low-level driver 506 for translation to SCSI-FCP work TCB, etc., (3) Track delivery and completion of commands (TCB) issued to SlimHIM, (4) SlimH
Interpretation of interrupt and event information from IM509 and FC
This is to execute appropriate interrupt processing and error recovery for the low-level driver 506. The data structure of the TCB is shown in Table VI below.

[0082]

[Table 6]

(A) First Process of the Day HIM 508 defines three entry points to use during the first process of the day. The first entry point is HIM_Find
In Adapter, this is called by FCIhw_int () and uses the PCI BIOS routine to determine if there is an adapter on a given PCI bus and device. To determine if an adapter is present,
Use the adapter's PCI vendor and product ID.

The second entry point is HIM_GetConf
iguratION, which is called by FCIhw_init () if an adapter is present, adding resource requirements to the provided HCB. ADAPTEC
, This resource contains IRQ, scratch, TCB
Includes memory. This information can be found by making a call to SlimHIM509.

The third entry point is HIM_Initial
In ize, this is called by FCihw_int () after allocating and initializing resources and initializes the TCB memory pool that makes calls to SlimHIM to perform scratch memory, TCB, and hardware initialization. .

(B) Failover Processing Both dipole halves of the ION 212 are connected to a common disk device set. ION212,
Both 214 must always have access to all devices. From the viewpoint of the HIM 509, there is no special failover process.

(6) AIC-1160 SlimHI
The ultimate purpose of the M SlimHIM 509 module is to provide a hardware abstraction of the adapter (ADAPTEC AIC-1160 in the figure). SlimHI
The primary role of the M509 is to forward Fiber Channel requests to the AIC-1160 adapter, service interrupts, and return status reports to the HIM module via the SlimHIM509 interface.

Further, SlimHIM 509 has an AIC
AIC-1160 manages and initializes hardware, loads firmware, and performs runtime work.
In the event of a 160 error, the AIC-1160 hardware is controlled.

2. External Interfaces and Protocols All requests for the ION physical disk driver subsystem 500 are made through a common high level driver 502.

A) Initialization (cs_init) A single call to the subsystem causes the device to
All initialization necessary to prepare O is performed.
During initialization of the subsystem, all driver structures and device or adapter hardware allocation and initialization are performed.

B) Open / Close (cs_ope)
n / cs_close) The open / close interface 510 performs initialization and classification of a structure required to access the device. The interface 510 differs from a general open / close routine in that all “open” and “close” are clearly layered. Therefore, I /
O All “opens” received by the physical interface driver 500 include the associated “closed” received.
Must be accompanied, and the device-related structure is not free until all "opens" are "closed". The open / close interface 510
The response of "open" or "close" indicating the completion of the request is synchronous.

C) Buf_t (cs_strateg)
y) The Buf_t interface 512 enables issuing logical block read and write requests to the device. The requester transmits a Buf_t structure describing the I / O. Device ID, logical block address, data address, I / O type (read / write),
Attributes such as callback routines are described by Buf_t. When the request is completed, the function specified by the requester in the callback is called. The Buf_t interface 512 is an asynchronous interface. Returning the function to the requestor does not indicate that the request has been completed. When the function returns, the I / O may or may not be running on the device. The request may be waiting to be executed on the queue. The request is not completed until the callback function is called.

D) SCSILib SCSILib 514 provides an interface that allows the transmission of SCSI Command Descriptor Blocks (CDBs) to the device except for normal reads and writes. Through this interface, requests such as starting and stopping units are used to spin and spin down disks, and requests to send or receive diagnostics to monitor and manage enclosure devices. All SCSILib routines are synchronous. The return of the called function indicates the completion of the request.

E) Interruption (cs_intr) The ION physical disk driver 500 checks all SCS
It is the central dispatcher for I and Fiber Channel adapter interrupts. In the embodiment, a front-end / back-end interrupt scheme is used. In this case, when the interrupt is serviced, the front-end interrupt service routine is called. The front end is executed from the interrupt stack and is responsible for clearing the source of the interrupt, disabling further interrupt generation by the adapter, and scheduling the back-end interrupt service routine. The backend is executed as a high priority task that actually handles the interrupt (along with any other interrupts that occur between the interrupt disablement of the adapter and the start of the backend task).

3. Functions of ION The ION 212 mainly performs five functions. This function is as follows.

Storage naming and projection: storage resources and resources stored in the storage disk 224
Work with the compute node 200 to project the image of the object to the compute node 200 and provide consistent storage naming.

Disk management: Performs data distribution and data redundancy techniques on the storage disk drives 224 that are operationally paired with the ION 212.

Storage management: Handles storage setup, data movement including processing of I / O requests from the computing nodes 200, performance measurement, and event distribution.

Cache management: read and write of cache data including cache sufficiency operations such as application hint prefetch.

Interconnect management: control of the data flow of the compute node 200 for performance optimization and control of the routing of requests, and consequently control of the storage distribution between the two IONs 212 of the dipole 226.

A) Storage Naming and Projection The ION 212 stores the storage resource object stored in the storage disk 224 in the computing node 20.
Project to 0. An important part of this function is the ION212
Globally unique names, fabric unique IDs, and volume set identifiers (VS
The creation and assignment of any of I).

FIG. 6 is a diagram showing the structure and contents of the VSI 602 and related data. Because it is important that the VSI 602 be unique and contention free, each ION 212 is responsible for creating and assigning a globally unique name for storage resources that the ION 212 manages locally. Further, only the ION 212 that manages the storage resource storing the storage resource object has the VS 212
I602 can be assigned. Only the ION that manages the resident storage resource can create and assign the VSI 602, but other IONs 212 may also manage storage and retrieval of the storage resource thereafter. This is because even if the VSI 602 assigned by the ION is moved to a storage resource managed by another ION, the VSI 602
This is because there is no need to change.

The VSI 602 is introduced as a 64-bit number including two parts, an ION identifier 604 and a sequence number 506. The ION identifier is a globally unique ID number assigned to each ION 212. One technique for obtaining a globally unique ION identifier 604 uses an electronically readable motherboard serial number that is often stored on a real-time clock chip. The serial number is unique because it is assigned to only one motherboard. ION identifier 60
4 is a globally unique number, so each ION 21
2 is a sequence number 606 that is unique only locally
By assigning a globally unique VSI 602
Can be created.

After binding the VSI 602 to storage resources on the ION 212, the ION 212 exports the VSI 602 to all nodes of the fabric via broadcast messages to enable access to the storage resource 104. This process is discussed in detail in the ION Name Export section of this document.

Using the exported VSI 602, the software on the compute node 200 can use the local entry point for storage resources that are semantically transparent in that they are indistinguishable from all other locally connected storage devices. Create For example, if the OS 202 of the computing node is UNIX, like the locally connected devices such as the peripheral device 108 and the disk 210,
Entries for block devices and unused devices are created in the device directory. Similar semantic equivalence is maintained in other OS 202 as well. Root names are also kept consistent between compute nodes 200 running different OSs 202 to provide the best support in this complex computing environment. The local entry point of the compute node 200 is dynamically updated to track the current availability of the exported storage resource 104. The OS-dependent algorithm running on the computing node 200 is V
Using the SI 602, a device entry point name of the imported storage resource is created. In this way,
Name consistency is guaranteed between nodes sharing a common OS. This allows the system to dynamically (rather than statically) create local (not static) entry points for globally named storage resources on each compute node 200 to provide consistent root names to support complex computing environments. Can maintain sex.

As described above, the storage resource 104
For details of VSI 602 creation, see Storage Resource 10
4 is directly controlled by the ION 212 that is exporting 4. Each VSI 602 is associated with one or more description headers, taking into account the possibility of OS 104 differences within the
Stored together with SI 602. Each VSI 602 descriptor 608 includes an operating system (OS) dependent data section 610 that consistently creates device entry points (name and operational) on the compute node 200 for a particular VSI 602. This is to store sufficient OS 202 dependent data, which is necessary for performing the meaning (the same is applied to the entire computation node 200). Included in the OS dependent data 610 are, for example, a data description local access right 612 and owner information 614. After VSI 602 is established by ION 212 and imported by compute node 200, VSI
Before the entry point of the storage resource 104 linked to 602 is created, the ION 212 sends the corresponding OS-specific data 610 to the computing node 200. The multiple description headers of the VSI 602 allow for simultaneous support of multiple compute nodes 200 using different OSs (each OS has its own description header) and support for separate access rights within different groups of compute nodes 200. Compute nodes 200 sharing the same description header create device entry points in a common and consistent manner. Therefore, in all the computing nodes 200 sharing the common access right, consistency between the name and the operational meaning is maintained.

The VSI descriptor 608 includes an alias field 616, which is a human-readable VSI 60
Used to display two people on a compute node. For example,
If the alias for VSI 1984 is "soma", then compute node 200 will have both 1984 and "soma" directory entries. VSI descriptor 6
08 is stored on the ION together with the VSI 602, so that the same alias and local access rights
602 appears on each import node that imports.

As described above, the present invention uses a naming method suitable for the distributed allocation scheme. In this method, names are generated locally according to an algorithm that guarantees global uniqueness. A variation on this is a local centralization method, where a central name server exists on each system, but with greater availability and robustness requirements than a purely distributed method. According to the method described above, the present invention can create a locally executed algorithm that guarantees global uniqueness.

The creation of a globally consistent storage system requires more support than simply maintaining the consistency of names on compute node 200. Closely related to names is a security issue, and the present invention takes two forms. One is the security of the interface between the ION 212 and the computing node 200, and the second is the security of the storage from within the computing node 200.

B) Storage Authentication and Authorization VSI 602 resources are protected by two types of mechanisms: authentication and authorization. ION 212 is a compute node 20
If 0 is authenticated, the VSI name is exported to the compute node 200. The exported VSI 602 becomes a device name on the computing node 200. Application threads on compute nodes can attempt to work with this device name. The access rights of the device entry point and the OS meaning of the compute node 200 determine whether the application thread is authorized to perform a particular permission.

This permission method extends the permission by the compute node 200 of the storage resource 104 at a location accessible by the interconnect fabric 106. However, the present invention differs from other computer architectures in that the compute node 200 does not directly manage the storage resources 104 of the present invention. This difference makes it impractical to simply tie local permissions data to the file system.
Instead, the present invention ties the authorization policy data of the compute node 200 with the VSI 602 of the ION 212 and uses a two-stage approach where the compute node 200 and the ION 212 share a level of mutual trust. The ION 212 allows each compute node 200 to access a particular VSI 602, but the refinement of granting a particular application thread to data specified by the VSI is
00 responsibility. The calculation node 200 is an ION 212
To enforce the authorization policy on the storage 104 using the policy contained in the authorization metadata stored in the storage 104. Therefore, the compute node 200 needs to trust the maintenance of the metadata by the ION 212 and the ION 212 to trust the enforcement of the permission by the compute node 200. One advantage of this method is that the ION 212 does not need to have knowledge of how to interpret the metadata. As such, the ION 212 is decoupled from the enforcement of specific permission semantics that are imposed by the different permission semantics imposed by the different OSs 202 used by the compute nodes 200.

All data (including the access right) related to the VSI 602 is stored in the ION 212, but the responsibility for managing the contents of the access right data lies with the computing node 200. More specifically, when the ION 212 sends a list of exported VSIs 602 to the compute nodes 200, each VSI 602 is tied to all OS-specific data required by the compute nodes 200 to enforce local permissions. It is. For example, the computing node 200 that runs UNIX has a name, group name,
D, that is, enough data to create a device entry node in the file system. An alternative name of the VSI 602 unique to the computing node OS 202 (or unique to the computing node 200) is included in each VSI 602. The local OS specific command for changing the access right of the storage device is converted into a message captured by the software of the computing node 200 and sent to the ION 212. This message is the OS version specific VS
Update I access rights. When this change is completed, the ION 212 sends an update to all the compute nodes 200 that use the OS of the system.

When the computing node (CN) 200 goes online, it sends a message “I am here” to each ION. This message is sent to compute node 200
Includes a digital signature that identifies If the ION 212 recognizes the compute node 200 (if the ION 212 approves the compute node 200), the ION 212 exports all VSI names to which the compute node 200 has access. The calculation node 200 uses the VSI 60
Assemble the local access entry point for system storage using the two-person list. When the application 204 in the computing node 200 first refers to the local endpoint, the computing node 200 requests the ION 212 for the access right description data of the VSI 602 through the interconnect fabric 106. This request message includes the digital signature of the requesting compute node 200. The ION 212 receives the message, finds the appropriate set of VSI permissions to return using the digital signature, and
The data is transmitted to the requesting computing node 200 via 06. However, the ION 212 is
It does not interpret the access right to send to, but only sends data. The software of the computing node 200 uses this data to generate a corresponding set of local access rights,
Bind to the local entry point of this target storage object.

The set of compute nodes 200 can share the same set of access rights by using the same digital signature or by having the ION 212 tie multiple different signatures to the same set of access rights. In the present invention, authentication is used to both identify the compute node 200 and specify which local authorization data is used to create a local entry point. The computation node retrieves the permission data only when the application first references the VSI 602. This “pull on demand” model avoids the startup costs of moving large amounts of access rights metadata for very large systems.

When the computation node 200 fails in the authentication,
The ION 212 returns a message not including the VSI 602 name, and an authentication failure flag is set. Compute node 20
0 can be continued without the VSI device name from the ION 212, and a failed authentication can be reported according to the request of the system administrator. of course,
Even if the authentication is successful, the transmission of the VSI device name to the computing node is not performed.

C) Conflict Resolution at Startup At startup, the ION 212 attempts to export the VSI 602 to the interconnect fabric 106. In this case, the data integrity of the system must be protected from any interruption by the new ION 212.
To accomplish this, the new ION 212 is checked before the storage can be exported.
This is performed as follows. First, the ION 212 checks the local storage and exports the VSI
A list 602 is created. VSI 602 metadata includes VSI generation or mutation number. The VSI mutation number increases when there is a major change in the state of the VSI 602 (such as when the VSI is successfully exported to the network). All nodes involved in the VSI conflict detection, including the compute node 200 and the ION 212, maintain the VSI history and mutation number exported to memory. All nodes on the interconnect fabric have the VSI of the exported VSI 602
SI contention must be monitored constantly. Initially (when initially created for storage expansion), the VSI variant number is set to zero. VSI60 exported
2 has a smaller mutation number than when it was last exported, even if the IO associated with the real VSI 602
Even if N212 is out of service, the variant number provides a criterion for conflict resolution in that its VSI can be presumed to be spoofed VSI. False VSI 602 associated with ION 212 having a mutation number larger than the mutation number of actual VSI 602
Is considered a real VSI 512 unless the I / O has already been implemented on the real VSI. IONs 212 that are newly introduced into the interconnect fabric need to start with mutation number 0.

After notifying the request to join the system, the IO
N212 sends the VSI 602 list and the associated mutation number. All other IONs 212 and compute nodes 20
0 obtains this list, and the ION 212
Check the validity of exporting a list of.

The other IONs currently exporting the same VSI 602 are assumed to be valid, and
, A message that does not allow the export of the competing specific VSI is sent. If the new ION 512 has a larger generation or mutation number than what is currently used in the system (an event that does not occur in normal operation because VSI is globally unique), the system that takes the necessary action Notify and report this to the administrator. If there is no conflict, each ION 212 and compute node respond with a continue vote. All ION
Upon receiving a response from the compute node 200 and the VSI 60 of the new ION 212 that is not competing,
In 2, the generation number increases, and it becomes possible to export to the system.

When the computing node 200 has the application reference and access of the VSI 602, the computing node 2
00 tracks the generation number locally. When ION212 publishes VSI602 (when trying to export)
Always, the compute node 200 compares the generation number published by the VSI 602 with the generation number stored locally for the VSI 602. If the generation number is coded, the computation node 200 votes to continue. If the generation numbers conflict (such as when an older version of the VSI goes online), the compute node 200 sends a reject message. Compute nodes 200 that have a generation number older than the new ION has published for the VSI vote for continuation and update the generation number of the local version of that VSI 602. The computing node 200 does not save the generation number until the reboot. This is because, in the basic design, the system throughout the interconnect fabric 106 is stable and constantly checks all new entrants, including the compute nodes 200 and IONs 212.

At the first power-on, a situation may arise in which the stability of the namespace of the VSI 602 becomes a problem. To solve this problem, first turn on the ION 212 so that name conflicts can be continuously resolved,
We will deal with it by accepting the participation of 0. As a result, the expired version of the VSI 602 (due to old data in the disk drive or other deterioration conditions) can be eliminated by the generation number. As long as the compute node 200 is not using its VSI 602, a new entrant with a higher generation number can override what is currently exporting a particular VSI 602.

(1) Name Service (a) Export of ION Name The ION 212 exports a working set of the exclusively owned VSI 602 to allow access to associated storage. The working set of the VSI exported by the ION 212 is a buddy ION (Dipole 21
6 (represented by other IONs 212, 214)
Determined dynamically through I ownership negotiations, it becomes globally unique among all nodes communicating with the interconnect fabric 106. This set is usually IO
The default or primary set of the VSI 602 assigned to N212. The VSI 602 set to be exported may be changed to a primary set other than the primary set due to VSI migration for dynamic load balancing, or exceptional situations including abnormalities of the buddy ION 214 and abnormalities of the I / O path.

Whenever the working set of the VSI is changed, the ION 212 using the broadcast message is used.
And provides the latest VSI configuration to the compute node 200. Compute node 200 is also ION2
12 may be asked about the working set of the VSI 602. When the ION 212 subscribes or re-subscribes online for the exported VSI 602, I / O to the VSI 602 by the compute node 200
Access is started. As described above, if there is any conflict in the exported VSI 602, the ION 212
Is not allowed to join online. Although all VSIs 602 associated with a storage set should be unique, a race condition may occur where multiple sets of storage have the same VSI (e.g., VSI is an ION).
Unique ID and ION attached to 212 hardware
212 is formed from the sequence number managed by
ON 212 hardware is physically moved).

When the working set is exported, the exported ION 212 enters an online state,
Before enabling I / O access to the exported VSI 602, a contention check timer (2 seconds) is set. The conflict check timer attempts to give the importer enough time to perform the conflict check process and notify the exporter of the conflict, but this cannot be guaranteed unless the timer is set to a very large value. . Therefore, the ION 212 requires explicit approval from all nodes (computing node and ION 212) to formally authorize online subscription. All nodes respond simultaneously to the online broadcast message, and the results are compiled and returned. If the summarized result is ACK, IO
N212 officially subscribes online. ION212
If he is not authorized to join online, he cannot access the newly exported set of VSIs 602. The node that sent the NAK then sends a VSI conflict message to the exported one to resolve the conflict. When the conflict is resolved, the ION 212 exports the adjusted working set and attempts to join online again.

(B) Importing CN Name The compute node 200 is responsible for acting to import all VSIs 504 exported by the ION 212. During the first processing of the day, compute node 20
0 requests all IONs 212 on-line for the previously exported VSI 602 to get the new state of the namespace. Thereafter, the compute node 200 pays attention to the export of the VSI 602.

The control information on the VSI 602 is ION
212 is included in the vs node managed. The computation node 200 portion of the vs node includes information used for construction and management of names exemplified in the application 204. The vs node information includes a user access right and a name alias.

(I) Name Domain and Alias VSI 602 may be configured with an application-defined name alias, which provides an alternative name for accessing the associated storage. name·
Aliases can belong to a virtual storage domain to logically organize a set of names. Name aliases must be unique within the virtual storage domain.

(Ii) vs node The modification of the vs node by the computing node 200 is sent to the owning ION, and is immediately updated and processed. next,
The change of the node is transmitted to all the nodes by the ION 212 exporting the change and rejoining the online state.

D) Storage Disk Management The JBOD enclosure 222 is responsible for providing a physical environment for disk devices and for providing some services to disk device and enclosure management applications. These services include (1) component error notification (power supply, fan, etc.), (2) threshold notification (temperature and voltage), (3)
(4) enabling and disabling a sound warning; and (5) setting a device ID of a disk device.

Previously, management applications were generally
An out-of-band connection interfaced with the enclosure. Single Network Management Protocol (SN
A serial or Ethernet connection to a remote enclosure using a protocol such as MP) could receive status information about the state of the enclosure. In the present invention, the disk enclosure may be physically separated from the host system, and it is not practical to monitor the configuration or status of the enclosure by a direct connection such as a separate serial path. To avoid extra cabling, the present invention uses an in-band connection that can monitor the status of the enclosure and control the configuration of the enclosure over existing normal Fiber Channel loops.

The in-band connection uses a SCSI command set transmitted from the host to the SCSI device for inquiring and controlling configuration information, and a mechanism by which the device communicates this information to the enclosure itself.
The protocol part of the host and disk device is SC
It is detailed in the SI-3 Enclosure Services (SES) specification, which is included in the references of this document.

To introduce the SES interface, three SCSI commands of inquiry, transmission of a diagnosis, and reception of a diagnosis result are used. The query command specifies whether the particular device is an enclosure device or a device that can forward SES commands to the enclosure service process. Sending the diagnosis and receiving the diagnosis result respectively
Used to manage and receive status information from enclosure elements.

When using the diagnosis transmission or diagnosis result reception command, a page code must be specified. The page code specifies the status or type of information being requested. All sets of default SES pages that can be requested by the Send Diagnostic and Receive Diagnostic Results commands are listed in Table VII below. Items in bold are those required by the SES event monitor.

[0133]

[Table 7]

The application client may execute the read diagnostics command intermittently to poll the enclosure and request a minimum allocated length greater than one enclosure status page. Information summarizes the status of the enclosure5
Returned by one byte containing the bits. If one of these bits is set, the application client can reissue this command with a long assigned length to get the complete status.

E) ION Enclosure Management FIG. 7 shows the ION enclosure management module and the ION
The relationship of the physical disk driver architecture 500 is shown. This subsystem is composed of two components: SES event monitor 702 and SCC2 + to SES gasket 704. The SES event monitor 702 is responsible for monitoring the service processes of all connected enclosures and for reporting by the event recording subsystem when status changes. This report is forwarded to the management service layer 706 if necessary. The SCC2 + to SES gasket 704 provides S
Responsible for translating CC2 + commands and translating them to one or more SES commands corresponding to the enclosure service process. This eliminates the need for the application client to know the specifications of the JBOD configuration.

(1) SES Event Monitor The SES event monitor 702
To the management service layer 706. Status information is reported by the event recording subsystem. SE
The S event monitor 702 executes a diagnostic result read command requesting an enclosure status page to intermittently poll each enclosure process. The diagnosis result read command is transmitted via the SCSLib interface 514 provided by the ION physical device disk driver 500. The reported status includes the status items in Table VIII below.

[0137]

[Table 8]

When the SES event monitor 702 starts, it reads the status of each element 402-424 included in the enclosure. This status is the current status. Upon detecting a change in status, a report of each status that has changed from the current status is returned to the management service layer 706. This time, this new status becomes the current status. For example, if the current status of the fan element is OK and the status change causes the element to be in a fan failure state, this event is reported as indicating a fan failure. If another status change indicates that the element has not been installed, this event is reported as an indication that the fan has been removed from the enclosure. If yet another status change indicates that the fan element is OK, this event is generated as an indication that the fan has been hot-plugged and is operating normally.

(A) First Processing of the Day The SES event monitor 702 starts after the ION physical disk driver 500 has been successfully initialized. After startup, S
The ES event monitor 602 reads the JBOD and SCSI configuration module 516 to check the mutual relationship between the disk device and the enclosure service device and the address of the device. Next, the status of each enclosure status device is read. It then generates events for all error conditions and missing elements. When these steps are completed, the status becomes the current status and polling starts.

(2) SCC2 + to SES Gasket SCC2 + is a protocol used by the ION 212 for configuration and management of virtual and physical devices. The plus (+) of SCC2 + means the addition of SCC2 to allow complete management of ION 212 devices and components and the consistent mapping of SCC2 defined commands to SES.

The service layer 706 addresses the JBOD enclosure 222 elements through SCC2 maintenance in and maintenance out commands. The following sections describe service actions that provide mechanisms for configuring, controlling, and reporting the status of components. These commands are a series of SCs for transmitting a series of diagnostics and receiving diagnostic results, respectively.
It is introduced into the ION 212 as an SI command.

The components are composed of the following services.
Performed using actions.

Component / Device Addition--The component / device addition command is used for configuring a component / device in the system and defining a LUN address. The LUN address is assigned by the ION 212 based on the location of the component on the SES configuration page. Following this command, a component device reporting service action is performed to obtain the result of the LUN assignment.

Component Device Report-A component device status report service action is a vendor specific command to retrieve the complete status for a component device. SES provides the status of each element type in 4 bytes. This new command is needed because the status report and component device report service actions only allocate one byte for status information,
This is because the default status codes conflict with those defined in the SES standard.

Component Device Connection-A component device connection requires that one or more units be logically connected to a particular component device. This command may be used to form a logical relationship between a volume set and its dependent component devices such as fans and power supplies.

Component Device Replacement-A component device replacement service action requires the replacement of one component device with another.

Remove Component Device-Peripheral Device / Remove Component Device service action requests the removal of a peripheral or component device from the system configuration. When removing a component device connecting a logical unit,
This command ends with a status check. The sensing key is an unreasonable request, with the removal of the faulty logic unit of an additional sensing qualifier.

The status of a component and other information are obtained by the following service actions.

Component Status Report-The component device status report service action is a vendor specific command to retrieve complete status information about the component device. SES provides the status of each element type in 4 bytes. The status report and component device report service action include 1 in the status information.
Allocate only bytes, default status code is SE
Conflicts with those defined in the S standard. So this new command is needed.

Status Report-The status report service action requests status information for the selected logical unit. One or more statuses for each logical unit are returned.

Component Device Report-The component device report service action is a JBO
Request information about the component devices in D. An ordered list of LUN descriptors is returned, reporting the LUN address, component type, and overall status. This command is used as part of the initial configuration process to determine the LUN address assigned by the add component device service action.

Component Device Connection Report-The Component Device Connection Report service action requests information about the logical unit that is connected to a particular component device. component·
A list of device descriptors is returned, each containing a list of LUN descriptors. The LUN descriptor contains the type and L of each logical unit that connects to the corresponding component.
Specify the UN address.

Component Device Identifier Report-The Component Device Identifier Report service action requests the location of a specific component device. An ASCII value indicating the location of the component is returned. This value must have been previously set by the component device identifier setting service action.

The management of components is performed as follows.

Component Device Command-The component device command is used to send a control command such as power on / off to the component device.
The actions that apply to a particular device depend on the component type and are vendor specific.

Suspend Component Device-The Suspend Component Device service action places a particular component in a suspended (inactive) state.

C. Interconnect Fabric 1. Overview To enable mass data movement, the storage model attached to the fabric of the present invention must address I / O performance issues related to data copy and interrupt handling costs. In the present invention, data copy,
He works on interrupts and flow control issues by combining methods. Unlike the destination-based address model used in most networks, the present invention uses a sender-based address model, in which a sender sends a destination-based address model before sending data to the fabric. Select a local target buffer. In the sender-based model, the destination sends the sender a list of destination addresses to which the message can be sent before sending the message. To send a message, the sender first selects a destination buffer from a list. This is possible because the target application first gives the address of these buffers to the OS, making it available to the target network hardware, so that the network hardware can use DNA work without copying. This is because it has enough information to transfer the data to the correct target buffer.

[0158] Despite the benefits, there are also some problems with sender-based addresses. First, with sender-based addresses, the protected area of the entire fabric extends from the destination to include the sender, losing overall separation and raising data security and integrity issues. With pure sender-based addresses, memory addresses are released to the sender and the destination must trust the sender, which is a major problem in high availability systems. For example, assume that a destination node has provided a list of destination addresses to a sender. Before the sender uses all these addresses, the destination node crashes and performs a reboot. The sender then has a set of address buffers that are no longer valid. At the destination, this address may be used for another purpose. Sending a message to any of these can have serious consequences, as important data will be destroyed at the destination.

Second, the introduction of sender-based addresses requires that the network cooperate to extract the destination address from the message before initiating the DMA of the data, but most network interfaces So it's not designed to do this.

The required address model includes the advantages of the sender-based model and avoids problems. The present invention uses a proprietary “Put It There” (P) that uses an interconnect fabric based on BYNET.
This problem is solved by a hybrid address model using the (IT) protocol.

[0161] 2. BYNET and the BYNET Interface BYNET has three important attributes that help to implement the present invention.

First, BYNET is inherently scalable-it can easily implement additional connections and additional bandwidth, and can immediately take advantage of everything in the system. this is,
In contrast to other bus-oriented interconnect technologies, where additional connections do not add bandwidth. Compared to other interconnects, BYNET not only can increase fan out (the number of ports available in a single fabric), but also has a dichotomous bandwidth that increases with fan out.

Second, BYNET can be extended by software to an active message interconnect-under the direction of the user (computation resource 102 and storage resource 104), BYNET allows the You can move data between them. DMA is used to move data directly to predefined memory addresses, avoiding unnecessary interrupts and internal data copying. This basic technique can be extended to optimally move such data blocks by multiplexing smaller data blocks into larger interconnect messages. Each data block can be processed using a modified DMA-based technique to optimize the use of the interconnect without losing the operational effectiveness of the node.

Third, since BYNET can be configured to provide a plurality of fabrics, the traffic shaping can be used to further optimize the interconnect. This is basically a mechanism provided by BYNET software that assigns a specific interconnect channel (fabric) to a specific type of traffic, for example, the randomization of long and short messages on shared channels with a large number of users. To reduce interference caused by various combinations. Traffic shaping is enabled by BYNET and can also be selected by the user for expected traffic patterns.

FIG. 8 is a diagram of the BYNET and the host-side interface 802. The BYNET host-side interface 802 includes a processor 804 that executes a channel program whenever a line is created. The channel program is executed by the processor 804 at both the transmitting interface 806 and the destination interface 808 of each node. The sending interface 806 hardware executes the channel program created according to the down call that controls the creation of the line, sends the data, and then shuts down the line. The destination interface 808 hardware executes the channel program to transfer the data to the destination memory and completes the line.

[0166] BYNET is composed of a network that interconnects the ION 212 with the computing node 200 operating as a processor in the network. Also B
The YNET includes a plurality of switch nodes 810 having input / output ports 814. The switch node 810
g (logbN) or more switch node stage 812
It is arranged in. b is the total number of input / output ports of the switch node, N is the total number of network input / output ports 816, and g (x) is a round-up function for obtaining a minimum integer larger than the argument x. Therefore, switch node 8
10 provides multiple paths between any network input port 816 and network output port 816,
Extend error tolerance and reduce contention. In addition, BYNET
Is used to manage message transmission throughout the network.
It consists of a plurality of bounce points on a bounce surface 818 along the highest switch node stage of the network.
The bounce points are switch node 810, which loads the balance message through the network, and switch node 81, which directs the message to the receiving processor.
0 is logically distinguished.

Processors, such as compute node 200 and ION 212, can be partitioned into one or more superclusters that comprise a subset of predefined processors that are logically independent. Communication between the processors is performed by a point-to-point method or a multicast. In multicast mode communication, a single processor can broadcast a message to all other processors or superclusters. Multicast commands can occur simultaneously in different superclusters. The sending processor issues a multicast command that is communicated to all processors or groups of processors over the transport channel. The multicast message is directed to a specific bounce point on bounce surface 818 and is routed to the processors in the supercluster. This prevents network deadlocks and prevents multicast messages to different superclusters from interfering with each other, since only one multicast message passes through a particular bounce point at a time. The processor receiving the multicast message replies, for example, by transmitting the current status over a return channel. BYNET plays a role in combining responses in various ways.

BYNET currently supports two basic types of messages: in-band messages and out-of-band messages. The BYNET in-band message conveys the message to a kernel buffer (or buffer) in the memory of the destination host, completes the line, and signals an up-call interrupt. In a BYNET out-of-band message, the header data of the line message is
Causes the interrupt operator of the YNET driver to create a channel program to be used for processing the remaining line data being received. For either type of message, the success or failure of the channel program is returned to the sender by a small message on the BYNET reply channel. This return channel message is processed as part of the line shut down operation by the sender's channel program (the return channel is the constant bandwidth return path of the BYNET line). After the line is shut down, an up call interrupt is (optionally) signaled at the destination, signaling the arrival of a new message.

The use of the BYNET out-of-band message is as follows.
This is not the most desirable configuration because the sender waits for the channel program to be created first and then executed. B
For YNET in-band messages, a copy of the data is required because the sender cannot directly target the application buffer. In order to solve this problem, the present invention utilizes BYNET hardware in a unique way. Destination side interface 80
8, the transmission interface 806 creates channel programs on both the transmission side and the destination side instead of causing the transmission interface 806 to create channel programs necessary for data processing.
The transmitting channel program transmits, as part of the message, a very small channel program that can be executed by the destination. The channel program describes how the destination moves data to a specific destination buffer of the target application thread. Because the sender knows the destination thread to which this message should be delivered, this technique allows the sender to control both how and where the message is delivered, and the traditional up-call handling of the destination. Damage can be almost avoided. This format of the BYNET message is called a directed band message. Unlike the active message used in the active message-interprocess communication model (which includes data and small message processing routines used to process messages at the destination), the present invention uses a BYNET I / O processor to provide a simple channel. BYN to run the program
Use the ET directed band message. In an active message, the host CPU normally executes an active message operator.

By using the reply channel, the transmitting interface can suppress the conventional interrupt method for notifying the completion of message transmission. In both out-of-band and directed-band messages, the sender's indication of success only indicates that the message was successfully delivered to the destination memory.

This guarantees that the message has certainly moved to the memory space of the destination node, but does not guarantee the processing of the message by the destination application.
For example, even though the destination node has a functional memory system, the destination application thread may have an anomaly that can permanently prevent the processing of the message. The present invention employs several independent methods for detecting and correcting message processing anomalies to ensure message processing. Regarding the communication protocol of the present invention, time-out is used for detecting message loss on the transmission side. Retransmission is performed as necessary, and when an abnormality in software or hardware is detected, recovery work is started.

According to the present invention, it is possible to transmit a message to a specific target at a destination while using a directed band message, so that a correct target application
A mechanism must be provided to give the sender enough data to send the message to the thread buffer. In the present invention, this is achieved by a ticket-based authentication scheme. A ticket is a data structure that cannot be forged and grants rights to the owner. In essence,
A ticket is a one-time permission or right to use a particular resource. In the present invention, the ION 212 can control the distribution of services to the computing nodes 200 by the distribution of tickets. In addition, tickets provide for the specification of specific targets that are a prerequisite for implementing a sender-based flow control model.

D. "Put It There" (PI
T) Protocol Overview The PIT protocol is a ticket-based authentication scheme that sends tickets and data payloads through active messages using the BYNET directed band message protocol. The PIT protocol is a unique combination of ticket-based authentication, sender-based address, debit / credit flow control, zero memory copy, and active messages.

[0174] 2. PIT Message FIG. 9 shows a PIT header 902 followed by a payload
PIT message or packet 9 containing data 904
It is a figure showing the basic feature of No. 01. PIT header 90
2 is a PIT ID, which represents an abstracted target data buffer, and is a time-limited ticket indicating a right to access a buffer of a specified specific size. The element that owns the PIT ID is
An element that has the right to use a particular buffer, PI
T ID 906 must be discarded when using the PIT buffer. When the destination receives the PIT message, the PIT ID 906 of the PIT header indicates the DM
The target buffer is specified for the BYNET hardware to which the payload is carried by the operation A.

[0175] Flow control in the PIT protocol is based on debit / credit credit using the sender base address.
Model. The transmission of the PIT message implies the flow control debit of the sender and the flow control credit of the destination. In other words, the device has a PIT ID
If 906 is sent to the thread, the thread will be credited with the PIT buffer for that address space.
If the device returns a PIT ID 906 to the sender, the device has either relinquished the right or released the buffer specified by the PIT ID 906. Device is PIT
Upon sending a message to the destination buffer abstracted by ID 906, the device also relinquishes its rights to the PIT buffer. When the device receives the PIT ID 906, it becomes a credit in the PIT buffer in the sender's address space (unless that PIT ID 906 is the device's returning PIT ID 906).

At the top of the header 902 is a BYNET channel program 90 for processing the PIT packet 901.
8 (sender side and destination side). Below that is PIT
There are two fields for transmitting an ID ticket, a credit field 910 and a debit field 912. Debit field 912 contains the PIT where the payload data is transferred by the destination network interface via the channel program.
ID 906 is included. This is called a debit field because the PIT ID 906 is borrowed by the sender's application thread (lent to the destination thread). Credit field 910 is where the sender thread transfers or lends the PIT buffer to the destination thread. The credit field 910 typically has a PIT ID 906 that the sender thread expects to send a reply message. The usage of this credit PIT is SASE
(Self address stamp envelope) called PIT. Command field 914 describes the work that the target performs on the payload data (eg, a disk read or write command). The argument field 916 is data related to the command (for example, the disk and block number of the disk that performs the read or write operation). Sequence number 918
Is a monotonically increasing integer that is unique to each source and destination node pair (each pair of nodes has one sequence number for each direction). Length field 920 specifies the length of the PIT payload data in bytes.
Flags field 922 contains various flags that modify the processing of the PIT message. One example is a message duplication flag. This is used to retransmit potentially lost messages and prevent more than one processing of the event.

When the system is first started, no node has a PITID 906 for another node. The BYNET software driver prevents the transmission of all directed band messages until the PIT's first open protocol is completed. PIT ID9
06 is distributed to the application
The thread is started when a virtual disk device on the ION 212 is first opened. During the first opening, the ION 212 and the compute node 200 enter a negotiation phase, where the exchange of work parameters takes place. Part of the first open protocol is PIT ID906
Is an exchange. The PIT ID 906 can specify two or more buffers as an interface that supports both the sender's collection DMA and the destination's distributed DMA. This application provides a PIT ID 90 for any application on all other nodes.
6 can be freely distributed.

The size and number of the PIT buffers exchanged between the computation node 200 and the ION 212 are adjustable values. Debit and Credit PIT ID 906 (Debit field 912 and Credit field 9
10) form the basis of a flow control model of the system. The sender can send only as many messages to the destination as the number of credited PIT IDs 906. This limits the number of messages that a particular host can send. Each node has its own P
It has an IT ID 906 pool and each sender can consume at most the assigned PIT ID 906
Only in that it is fair.

The ION 212 manages a pool of PIT tickets issued to the computing node 202. The initial assignment of PIT ID 906 to compute node 202 occurs during a first open protocol. The number of distributed PIT IDs 906 is based on a prediction of the number of concurrently running compute nodes 200 that will use the ION 212 at the same time and the memory resources in the ION 212. Since this is only a prediction, the size of the PIT pool is also dynamically adjusted by the ION 212 during operation. This PIT
The redistribution of resources is necessary to ensure a fair response to requests from a plurality of computing nodes 200.

The PIT reassignment to the operation calculation node 200 is performed as follows. Since the computing node 200 is constantly making I / O requests, by controlling the flow of PIT credits for completed I / O messages,
T resources are redistributed. Until you reach the right level
No PIT credit is sent due to the completion of the ION 212 (the PIT pool of the compute node 200 is reduced). The situation is more complicated if you already have a PIT assignment but the compute node 200 is not running (and the resources are not moving). In this case, the ION 212 assigns a PIT (or PIT) to each of the inactive compute nodes 200.
ID list) can be sent. If an inactive compute node 200 does not respond, the ION 212 can invalidate all of its PIT IDs before the other compute nodes 200
PIT ID can be redistributed. If a non-working compute node 200 attempts to use the reassigned PIT, that compute node 200 is forced back to the first open protocol.

The increase in PIT allocation to the computing node 200 is performed as follows. The PIT Assignment message can be used to send the newly assigned PIT ID to any compute node. An alternative technique sends more than one PIT credit per I / O completion message.

[0182] 3. How the PIT Protocol Works-Read and Write Disks To illustrate the PIT protocol, the compute node 200
The case where a read operation from ON 212 is requested will be discussed. Here, it is assumed that the first open has already been performed and that there is a sufficient number of free PIT buffers in both compute node 200 and ION 212. The application thread makes a read system call and passes the address of the buffer for transferring disk data to the high-level SCSI driver (CN system driver) of the compute node. The CN system driver creates a PIT packet containing this request (including the virtual disk name, block number, and data length). The upper half of the CN system driver provides information in the debit and credit PIT ID fields 910, 912. This debit PIT field 912 is the PIT ID 906 of the destination ION 212 to which this read request is sent. Since this is a read request, the ION 212
When creating the completion packet, you need a way to specify the application's buffer (provided as part of the read system call). Because PIT packets use the sender base address, PIT
If it has the ID 906, only the ION 212 can address this application buffer. Since the application buffer is not part of the normal PIT pool, the buffer is fixed in memory and a PIT ID 906 is created for the buffer. Since a read request also requires a reply status from the disk operation, a PIT distributed buffer is created to include the reply status. As part of the read PIT packet,
SASE PIT is sent to the credit field. PIT packets are placed in an output queue. PIT
When sending a packet, the BYNET interface 8
02 moves this from the sender side by DMA work,
Transfer through interconnect fabric 106. When the PIT packet arrives at the BYNET interface 808 on the destination side, the execution of the PIT channel program by the BYNET channel processor 804 is started. Host side interface 802
The BYNET channel processor 804 extracts the debit PIT ID 906 and places the endpoint on the ION 212. The channel program extracts the buffer address and the interface DMA engine
Program moving data directly to the PIT buffer-thus enabling the PIT protocol to provide zero data copy semantics. The BYNET interface 802 notifies the receiving application on the ION 212 of the interruption. No interruption occurs in the computing node 200. If the reply channel message indicates a transfer failure, the I / O will be retried depending on the reason for the failure. After several attempts, an ION error status is entered (see ION 212 Recovery and Failover Work Details in this document) and compute node 200 enters the dipole buddy IO.
Attempt to have N214 handle the request. When the message is reliably delivered to the destination node's memory, the host sets a retransmission deadline (longer than the worst case I / O service time) to ensure that the ION 212 has successfully processed the message. When this timer expires, compute node 200 retransmits the PIT message to ION 212. If the I / O is still in progress, the duplicated request is simply rejected; otherwise, the retransmitted request is processed normally. Optionally, the protocol may reset the expiration timer and require explicit notification of the retransmitted request to avoid application damage due to I / O failure.

FIG. 10 is a block diagram of the ION 212 function module. The inputs to the IONs 212 and 214 are data lines 1002 and 1004 and control line 1
006. Each module of the ION 212 comprises a control module 1008 that communicates with a control line 1006. The control module 1008 includes the data line 100
2 and provides a module control function. System function module 1010 implements the ION functions described in this document. ION 212 and 214
Is composed of a fabric module 1020, a cache module 1014, a data restoration module 1016, and a storage module 1018. Each of these modules comprises a control module, a work injector 1020 for inserting and retrieving data on the data lines 1002 and 1004, and a data fence 1022 for inhibiting the passage of data.

After the PIT read request is sent to the ION 212, it is transferred to the work injector of the ION cache module 1014. The work injector inserts the request into the ION cache module 1014 and
If the data is cached or a buffer of data is allocated, the cache module returns the data directly and passes the request to the ION storage module 1018. The ION storage module 1018 translates this request into one (or more) physical disk requests and sends the requests to the appropriate disk driver 224.
When the disk reading operation is completed, the disk controller notifies the interruption and notifies the completion of the disk reading. The ION work injector creates an I / O completion PIT packet. Debit PIT ID (Debit
Field 912) is the read request SASE
Credit from PIT (where the application wants to put the disk data) PIT ID (credit
(Stored in field 910). Credit PIT
ID is the PITI from which the compute node 200 sent this request.
Same as D, or replaced PIT ID if the buffer is not free. The credit PIT passes the credit to the compute node to send future requests (the queue for compute node 200 to ION 212 is increased by one since the current PIT request has just been completed). After processing a PIT, the ION 212 does not return PIT credits for three reasons. One is ION2
12 wants to reduce the number of outstanding queues from compute node 200. The second reason is that the ION 212 wants to redistribute the PIT credit to another compute node 200. The third reason is that a single PIT packet contains multiple requests (see the description of the super PIT packet in this document). The command field 914 is a read completion message, and the argument is a return code from the disk drive read operation. This PIT packet is queued to the BYNET interface 702 and sent back to the compute node 200. At this time, the BYNET hardware moves the PIT packet to the computing node 200 by DMA. Thereby, the calculation node 200B
Debit PIT by YNET channel program
Extraction of ID 912 begins and confirms this before DMA to the target PIT buffer (here, the application's fixed buffer) starts. When the DMA is complete, the compute node BYNET hardware initiates an interrupt to notify the application that the disk read is complete. In the ION 212, the BYNET driver returns the buffer to the cache system.

The operation of the write request is performed in the same manner as the read operation. The application calls the compute node's high-level driver and retrieves data, virtual disk names,
Pass the address including the disk block number and data length. The high-level driver of the compute node is the destination ION
The PIT ID 906 of 212 is selected, and a PIT write request is created using this data. SASE PIT
Contains only the return status of the write operation from the ION 212. The ION 212 is notified of the interruption when the PIT packet arrives. This request is
It is processed in the same manner as the T read operation. The write request is passed to a cache routine, which then writes the data to disk. Upon completion of the disk write (or when the data is securely stored in the write caches of ION nodes 212 and 214), an I / O completion message is sent back to compute node 200. ION212
Is running a working write cache,
The other ION 214 of the dipole, not the ION 212 to which the request was sent, returns an I / O completion message. This is further explained in this document at the location of the Bermuda Triangle Protocol.

[0186] 4. Revoked PIT ID and Recovery Issues The first open PIT ID exchange is the revoked PIT caused by hardware or software above.
This is a mechanism for invalidating the ID 906. ION21
2 and compute node 200 exchange PIT IDs,
Consider a situation in which ON 212 has crashed. PIT I
D906 means a target buffer fixed in the memory, and unless invalidated, ION2 that has been rebooted
12 or the unprocessed PIT ID 906 of the compute node 200
Can cause major problems with software integrity due to invalid or expired PIT IDs. BYNET hardware and directed band messages support providing a basic mechanism for revoking the expired PIT ID 906.

At the end of the first open protocol, both have added the PIT to the compute node high-level SCSI driver.
A list of hosts to which ID 906 is to be distributed must be passed. Stated another way, the host has passed to the compute node high level SCSI driver a list of hosts that will receive the PIT packet. The compute node high-level driver uses this list to create a table that manages the distribution of directed band messages. This table specifies ION 212 pairs that can send directed band messages to each other (this table also specifies one-way PIT message flows). The compute node high level driver stores this table inside the host (as driver specific data) as part of the BYNET configuration process. Adding or removing hosts from this list is always done by the PIT protocol with a simple notification message to the compute node high level driver. If a node fails, shuts down, or becomes unresponsive, BYNET hardware detects this and notifies all other nodes on the fabric.
The BYNET host driver on the cache node responds to this notification and removes all references to this host from the directed band host table. This action invalidates all PIT IDs 906 that the host has distributed to other hosts. This is the key that protects the node from previously distributed PIT packets. BY until the compute node high-level driver on that host is reconfigured.
NET does not deliver all messages to the host.
After the first reconfiguration has taken place, BYNET will not allow any directed band messages to be sent to this newly restarted or reconfigured host until the local PIT protocol signals. As a result, the distribution of the invalid PIT packet can be protected until the appropriate PIT protocol is initialized by the first open protocol.

If the host attempts to send a directed band message to an invalid host (using invalidated PIT ID 906), the sending compute node high level driver will reject this message with an error condition to the sender. . This rejection initiates a first open handshake between the two nodes. After the first open handshake is completed, any outstanding ION 212 I / O work (as viewed from compute node 200) will be retransmitted. However, unless this is a warm restart, the ION 212 is likely to be down for a long time, and outstanding I / O work will be restarted as part of the failover process and other I / Os in the dipole will be restarted.
It will be sent to ON 212 (see ION failure handling section for details). When the crashed node is the computation node 200, when the first open request of the computation node 200 that has already performed the first open arrives at the ION 212 unexpectedly, the PIT ID recovery operation is started. The ION 212 invalidates all PIT IDs 906 credited to the computing node 200 (or actually only reissues old ones). All outstanding I / O work for compute node 200 can be completed (unless the time to restart the node is extremely short, so this event is unlikely to occur). SA used
The SE PIT expires and the completion message will be rejected (and no application thread will have issued the I / O request).

[0189] 5. Super PIT (SPIT)-Small I
Improving I / O performance The PIT protocol can use normal SCSI commands. Since the core of the present invention is the communication network, not the storage network, the system uses a network protocol to
Improve the performance that the model allows. The overhead processing when handling up calls represents a performance barrier for workloads dominated by small I / O requests. There are several ways to improve small I / O performance. One is to improve the path length of the interrupt handling code. The second is to use a technique similar to that employed in device drivers to reduce the vector of multiple interrupts into a single call to an interrupt manipulator. Third, reduce the number of individual I / O operations and cluster (convoy) into a single request. Due to the difference in MTU size of the source and destination physical links, nodes that need to repackage incoming and outgoing data flows tend to collect data. This problem is further exacerbated by a mismatch between the speeds of the transmitting side and the destination side network (particularly the destination network becomes slower). These nodes are constantly subject to flow control from the destination. As a result, data flowing out of the router due to the burst is generated. This is called a data convoy.

In the present invention, the data convoy is stored in the ION
This is used as a technique for reducing the number of interrupts generated by the up call in the 212 and the compute node 200. For the sake of explanation, the calculation node 200
Consider the data flow to In the flow control debit / credit model used in the present invention, I / O requests are queued at both the compute node 200 and the ION 212. The queue starts with a PIT packet stored in the ION 212 and when all of this is used, the compute node 20
Return to 0 and continue. This is called an overflow condition. Normally, an overflow indicates that a node has its own PIT.
Occurs when you have more requests than buffer credits. Each time the I / O is completed, the ION 212 returns a completion message to the compute node 200. Typically, this completion message includes credits for the released PIT buffer resources. This is the basis for debit / credit flow control. When the system is flooded with I / O requests, the Ion 212 receives a new I / O each time the I / O is completed.
Replaced by O request. Therefore, during periods of heavy load, I / O requests flow into the ION 212 one at a time and are added to the ION 212 queue for an unspecified period. Each of these requests causes an up-call interrupt and increases the load on the ION 212.

This double queuing model has a number of advantages. A close offset transaction is performed on the PIT buffer allocated to the computing node 200. Sufficient work should be queued locally at the ION 212 so that new work can be dispatched as soon as the request is completed. However, by allocating to the cache system, the memory resources consumed by requests in the queue on the ION 212 can be used more efficiently. When the ION 212's PIT queue is in a state of insufficient memory to maintain memory, performance degrades if the ION 212 runs out of work and has to wait for work to be sent from the compute node 200.

The super PIT has a debit / high
A PI designed to reduce the number of up-call interrupts using the flow control of the credit system
This is a feature of the T protocol. Super PIT is OLT
Improves the performance of P and similar tasks dominated by large amounts of relatively small I / O. Rather than sending one request at a time, a super PIT packet is a collection of I / O requests all carried by a single large super PIT request. Each super PIT packet is transferred in the same way as a normal PIT buffer. Super PI
Each I / O request included in the T packet is transmitted to the ION 21
When two resources become available, they are extracted by the PIT work injector and inserted into the regular Onion 212 queuing mechanism. This individual I / O request is
Either read or write request may be used.

The PIT work injector is an ION212
(I) of the application request forwarded to
Acts as a proxy (on ON 212). PIT work injectors are also used as the RT-PIT and FRAG-PIT protocols described in later sections. As each super PIT request becomes empty, this resource is released to the compute node so that another super PIT packet can be sent and replaced. The number of super PIT packets allowed for one host is determined in the first open negotiation. Of course, the amount of work in the ION 212 queue must be sufficient to keep the ION 212 working until another super PIT packet is carried.

The ION 212 is calculated by the calculation node 200.
Consider the situation where enough work has been added to the queue to deplete its PIT credits and the addition of requests to the queue has started locally. The number of requests that can be enqueued in the super PIT request queue is limited only by the size of the buffer to which the super PIT is transferred. A super PIT packet works differently from a normal PIT packet. With the control model of the present invention, a device can send a request (debit) only if it has credit to the destination. It does not matter what PIT packet the device uses, as the device is not targeting a particular application thread within the ION 212. P to ION212
IT packets only limit the use of buffers (and have flow control as a side effect). In contrast,
The SASE PIT in the PIT request is different. SA
The SEPIT ID means an address space of a specific thread in the computing node 212. Super PIT
Includes a SASE PIT, but when the I / O for this request is completed, the created I / O completion message does not include the credit PIT. Super PI
Only when all requests in T are exhausted will a credit PIT be issued to that address space.

The creation of the super PIT in the computing node 200 is performed as follows. A super PIT can always be created if there are at least two I / O requests to a single ION 212 in the queue of the compute node 200. This ION
Regarding 212, if the super PIT packet of the compute node 200 has already reached the limit,
0 keeps adding requests to the queue until the super PIT returns. This compute node is then replaced by another super PIT
Issue a message. Within this system driver,
When a queue occurs, a queue for each ION is required to create a super PIT packet.

As described above, the super PIT message is an ION by a task dominated by a large number of small I / O requests.
212 can be reduced. Super PI
T messages improve the performance of the destination node and the availability of the interconnect fabric by increasing the average size of the messages. However, the concept of a super PIT message is also applicable when the ION 212 reduces the load on the compute node 200 caused by small I / O work. Creation of a super PIT message in the ION 212 has a completely different problem from creation in the compute node 200. In the computing node 200, the application thread that creates the I / O request is ION2
12 is subject to flow control for preventing overload. The service speed of the disk subsystem is
Because it is much lower than the ON 212, it is always the final limit on the performance of the ION 212. Request is for ION21
2 is blocked from entering the system until it has added it to the queue and has enough resources to perform the service. Importantly, requests are queued at the compute nodes (or applications are blocked) until resources are available on the ION 212. Insufficient resources are not a problem at the compute node 200. When an application of the computing node 200 issues an I / O request to the system, a part of the request includes a memory resource (application thread buffer) of the computing node 200 required to complete the I / O.
Is included. The I / O completion message that the ION 212 needs to send to the compute node 200 includes the PIT I
D (SASE PIT ID) is assigned.
From the perspective of the ION 212, the I / O completion message has already been allocated a target buffer and can be executed as soon as the data is ready. The I / O completion message is successful if delivered (ION 212 need not have the service time of the disk storage system of the compute node). Therefore, the ION 212 cannot block by the pressure of the flow control from the calculation node. To create a super PIT message, the compute node uses a flow control queue, but the ION 21
2 does not have this option. ION212,
Except for access to BYNET, there is much less opportunity to create a super PIT message since it has no resources to wait for.

Several methods are available for creating a super PIT message at the ION 212. One is to slightly delay the I / O completion request and
This is to increase the chances of creating a T packet. After a slight delay, if no new completion message has been prepared for the same node, it is sent as a normal PIT message. The problem with this technique is
Taking the time to delay the request in anticipation of creating a super PIT (to reduce the up-call overhead of the compute node) is to increase the total required service time correspondingly. Eventually compute node 200
But it also slows down the application. An adaptive delay time is valid (determined by the average service time for the compute node 200 and the total service time for a particular request). The second method is a slight modification of the first. In this case, each computing node 200 needs to provide each ION 212 with a delay time that increases as the ratio of small I / O at the computing node increases. This will increase the opportunity for a particular ION 212 to create a super PIT message when needed. A third approach is to delay certain types of traffic, such as small reads or writes, where the cache services directly and does not have to wait for storage 224 disk activity. The cache avoids disk traffic and reduces average I / O latency for some requests, but the distribution of latency varies with cache hits. Short queuing delays in cache hit requests do not result in a significant increase in service time compared to those involving disk work. In applications that are sensitive to service time allocation (uniform response time is critical for performance), small delays for super-PIT packet creation in ION 212 may improve overall system performance. . 6. Large Block Support and Fragmented PIT Packets Database application performance requirements are often independent of database size. As the size of the database grows, the speed at which disk storage is probed must be increased proportionally to prevent erosion of application performance. In other words,
In a growing customer database,
The response time for certain inquiries must be kept constant. The challenge in meeting this requirement is that it directly conflicts with current trends in disk drive technology. As disk drive capacity increases,
The random I / O performance has not changed. One way to alleviate this tendency is to increase the average size of disk I / O work as the capacity of disk devices increases. Given the current trends in storage capacity and performance requirements, the average I / O size of 24 KB may increase to 128 KB in the very near future. More aggressive cache usage and lazy write techniques may be useful for many jobs. The only non-uniform technology growth in disk drives is
It does not create an increase in O request size. B
With the spread of databases using LOBS (binary large objects), objects reaching the size of 1 MB or more are becoming common. Regardless of the particular cause, the system will need to support large I / O objects, the size of which will follow the economics of disk storage.

ION 212 Using PIT Protocol
And the transmission of large data objects by the compute node 200 have several problems. As mentioned in this document, the advantage of the PIT protocol is that it allocates the destination buffer first and addresses the issues of flow control and endpoint determination. However, up call semantics also need to ensure (or allocate) enough buffer space to store the message. In the PIT protocol, a target PIT ID 906 for storing each message on the transmission side.
This problem is addressed by having the sender select. Large I / O writes certainly complicate the protocol because the size of the message is a criterion in selecting a particular PIT ID 906 from the available pool. During periods of heavy load, PIT ID9 available to the sender
There may be situations where despite having 06 credits, not all of them meet the buffer size requirements for large I / O requests. In the PIT protocol, when the data size to be transmitted is wide, the transmitting side must cooperate with the receiving side to manage both the number and the size of the PIT buffer. This raises the problem of PIT buffer allocation size. That is, when creating a pool of PIT buffers, what is the proper allocation of buffer size for a pool of PIT buffers for a particular job?
BYNET software is not only for writing
Additional maximum transfer unit (MTU) that also complicates / O read
To force. I / O requests (both read and write) that exceed the BYNET MTU need to be fragmented on the transmitting side and reassembled on the receiving side by a software protocol (in this case, the PIT protocol). This creates a problem of memory fragmentation. Simply put, internal fragmentation is wasted space inside the allocated buffer, and external fragmentation is wasted space outside the allocated buffer that is too small to satisfy any demand. One solution is to use only a portion of the large PIT buffer, but using a large PIT buffer will result in unnecessary internal fragmentation. Large PIT buffers waste memory and degrade cost / performance.

In the present invention, BYNET MTU and PI
The problem of T-buffer allocation has been solved by adding two PIT message types. This is RT-P
IT (Round Trip PIT) and FRAG-PIT
(Fragment PIT). FRAG-PIT and RT-PI
T both use a data pull model instead of a PIT data push model (push
For data, the sender pushes the data to the destination.
In pull data, the destination pulls the data from the source). The FRAG-PIT message is designed to support reading large data, and the RT-PIT message supports writing large data. FR
AG-PIT and RT-PIT are both super PIT
Like, manage data flow using ION PIT work injectors.

A) RT-PIT message The computing node 200 requests the ION 212 to execute a large disk write operation, and the I / O write
YNET MTU or any available ION212P
When either of the IT buffers is larger, computation node 2
00 creates an RT-PIT creation message. RT-
PIT messages work in two stages. The boost phase and the round trip phase. In the boost phase, a list of source buffers for the data to be written is assigned to the compute node 200 series of PITIDs.
The fragmentation size of the source buffer is BYNET MTU
And the size constraint specified in the ION initial open protocol. This list of PIT IDs (and buffer sizes to be formed) is a single RT-PIT
Placed in the payload of the request message, the destination ION2
12 PIT credits. Additional PIT buffers are allocated from the compute node pool for direct use by the RT-PIT protocol. The PITID of this additional buffer is placed in the credit field of the PIT header. The remaining RT-PIT requests are
Same as the IT write message. Next, the computing node 200 transmits this RT-PIT request message to ION2.
Send to 12 (boost).

In the ION 212, the PIT work injector processes the RT-PIT request message in two steps. For each source-side PIT ID 906, the work injector must request a matching PIT buffer from the ION cache (I
This is all done together or one at a time, depending on the space available in the ON buffer cache). By matching PIT buffers, ION21
2 dynamically allocates resources and combines them with write requests. As a result, I / O can be advanced using the sequence obtained by modifying the normal PIT transfer. Here, the processing of the RT-PIT message is a round
Entering the trip phase, the work injector creates an RT-PIT start message for one (or more) matched pairs of source and destination PIT IDs (ION2).
12 also has the option to send one or a portion of the matching PIT ID). Single RT-PIT
The number of PIT IDs 906 included in the start message is
It controls the fineness of data transfer within the ION 212 (described below).

This RT-PIT start message is sent back to the compute node 200, and the boost phase of the RT-PIT message ends. Upon receiving the RT-PIT start message, the compute node 200 uses the normal PIT write message, one PIT pair at a time,
The data is transferred to the ION 212. Since both compute node 200 and ION 212 have enough data to handle the lost fragment, compute node 200 need not send the fragments in order (the matched PIT pair specifies the reassembly order). Do). When the ION 212 receives the PIT write message, the work injector is notified and
This write request is a larger RT-PIT I / O
Recognize that it is part of the task. The work injector has two options for the PIT write process: pass the fragment to the cache routine to start the write operation, or wait for the transmission of the last fragment before starting the write. Starting I / O earlier allows the cache to send data flow to the disk drive in a pipelined fashion (depending on write cache policy), but there is a risk of performance degradation due to small I / O size. However, if I / O is held until all fragments arrive,
The cache system may be overloaded. Since the total size and quantity of the fragments are known from the start, all data needed to optimize large I / O requests in current operating conditions is created by the cache system. On the computing node 200 side, whenever a single RT-PIT start message includes a plurality of fragments each time the transmission of a PIT write operation is successful, the start of writing the next fragment occurs. When the last fragment in a single RT-PIT start command is received, the request injector will:
The data is passed to the cache system for the same processing as a normal write request. If the data is secure, the cache system creates an I / O completion message and sends it back to compute node 200 to indicate that this stage of the process (of the RT-PIT start operation) has been completed. If fragments remain, an RT-PIT start command is created and sent to the compute node, and the above cycle is repeated until all fragments have been processed. When the work injector and cache have completed processing the last fragment, a final I / O completion message is returned to the compute node with status, synchronizing the end of processing of the RT-PIT request.

The RT-PIT message can be optimized for BYNET with some changes. ION212 is RT-P
Consider the situation immediately after receiving an IT request. ION212
Work injectors have compute node buffers and IONs
212, trying to translate a large I / O request into many small regular write requests.
Synchronization is performed by an intermediate RT-PIT start command. However, if BYNET makes it possible to extract the data of the received channel program, RT-P
Intermediate steps of sending the IT start command can be eliminated.
For the purpose of explanation, this mode of BYNET operation is looped.
Let's call it a band message. A loop band message is actually two directed band messages, one embedded in the other. For example, RT-P
The work injector that has received the IT request will process each fragment by creating an RT-PIT start message that contains the data needed to create the second PIT write message at the compute node. RT
The PIT start message transfers the fragmentation template of the PIT write operation to the compute node 200. A channel program (RT-
Sent together with the PIT start message) stores the payload on the transmit queue of the compute node BYNET driver. This payload is similar to a request queue from the application thread making the first RT-PIT request. This payload contains the source and destination PIT
Using the ID pair, create a PIT write request for this fragment sent by the work injector. PI
The T write stores the fragment in the ION 212 and signals the work injector of arrival. The work injector repeats this cycle for each fragment and processes everything. The performance improvement due to the loop band message comes from the elimination of the interrupt and compute node processing required for each RT-PIT start message.

[0204] The FRAG-PIT message is designed to support the work of large I / O read requests from compute nodes. Application is large I / O
When a read is requested, the compute node fixes the target buffer and sets P to mean the target buffer for each fragment.
Create a list of IT IDs. Each PITID
Describes a scatter list consisting of a fragment target buffer and an associated status buffer. The status buffer is updated upon data transmission so that the compute nodes can determine when each fragment has been processed. The size of each fragment is determined by the same algorithm as the RT-PIT message (see RT-PIT section above). By combining these fields, FRAG-P
Create IT.

The FRAG-PIT request is sent to the computation node 200
To the ION 212 where it is processed by the work injector. This request includes the virtual disk name, the starting block number, and the data length of the data source on the ION 212. Work injector is RT-
Work on FRAG-PIT requests in the same way as PIT requests. Each fragment in the FRAG-PIT request is processed as a separate PIT read request in cooperation with the cache system. The cache system can choose to treat each fragment individually or as a single read request and supply disk data to the work injector when available. As the cache supplies the data fragments (individually or as part of a single I / O operation), the data of the large read request begins to flow back to the compute nodes. For each fragment for which the cache made the data available, the work injector fetched the data fragment
Send it back to the compute node in a G-PIT partial completion message. Each FRAG-PIT partial completion message carries data in the same manner as a normal PIT read request completion, but the FRAG-PIT partial completion message does not generate an interrupt at the compute node when delivered. The last completed fragment is returned to the compute node along with the FRAG-PIT complete completion message. The difference between the FRAG-PIT complete completion message and the partial completion message is that an interrupt signals the completion of the entire FRAG-PIT read request (full up call).

7. PI to other network devices
Introduction of the T Protocol Most of the performance of the above described approach to network attached storage depends on the interconnect fabric 106's ability to support the PIT protocol. In the case of BYNET, a low-level interface is created, which is closely aligned with the PIT protocol. Other networks such as Fiber Channel
The interface has the ability to support the PIT protocol as well.

E. Bermuda Triangle Protocol The present invention uses ION cliques 226 and write caches to provide data and I / O redundancy. I
The ON clique 226 comprises a plurality of IONs (usually installed as pairs or dipoles), for example, an ION 2 comprising a primary ION 212 and a buddy ION 214.
12 and 214.

The buddy ION 214 is a primary ION
212 serves as a temporary storage location for a copy of the modified cache page, providing data and I / O redundancy. IO
N Creek 226 (pair or dipole of ION in the figure)
Each ION 212 functions as a primary ION 212 for one group of volume sets and a buddy ION 214 for the other.

In order to provide high availability and a write cache, it is necessary to securely store data in at least two places until an application can confirm a write. This may be performed using a backup copy of a cache memory or a log for a high-speed sequential disk. If the storage controller fails after the write is confirmed and before the data is recorded to permanent storage, this failure of the redundant copy supply can lead to data loss.

However, since the ION 212 and the ION 214 are constituted by physically separated computers, it is necessary to maintain such a backup copy for communication on the interconnect fabric 106. For best system performance, it is necessary to minimize interrupts associated with BYNET transmission and write protocols while utilizing the write cache.

FIG. 11 shows one possible protocol for writing data to the disk 224 in the dipole 226. In steps 1 and 3, the compute node 200 sends the request to the primary ION 212 and the buddy I
Send to ON214. In steps 2 and 4, the ION responds to the write request. Compute node 200 is ION 21
Upon receipt of a response from both 2 and 214, the write is considered complete. When data is subsequently written to the disk, the primary ION 212 receives the buddy ION2
Send a rejection request to 14 indicating that it is not necessary to keep a copy of the page of write data. Assuming that the "transmission complete" interrupt is suppressed on the transmitting side, this protocol requires a minimum of five interrupts because each transmitted message generates an interrupt at the compute node or ION 212 and ION 214. Another drawback of this protocol is that when the write starts, one of the IONs goes down forever to avoid waiting forever for the second response if one computes the primary ION 212 and the buddy ION 214 It is necessary to know the situation.

FIG. 12 illustrates another possible protocol. In this protocol, the primary ION 21
2 sends a write request to the buddy ION 214, waits for a response, and instructs the compute node 200 to send back a confirmation. This protocol also requires at least five interrupts. The first interrupt is as shown in step 1 where the compute node 200 is the primary ION 212
Occurs when sending a write request to The second interrupt occurs in step 2 when primary ION 212 sends data to buddy ION 214. The third interrupt occurs at step 3 when buddy ION 214 signals receipt of data. The fourth interruption is step 4, in which the primary ION 212
Occurs when responding to, and the last interrupt is
Occurs when the primary ION 212 sends an exclusion request to the buddy ION 214 after the data has been safely transferred to the disk.

FIG. 13 shows a protocol used in the present invention for minimizing the number of interrupts required for processing a write request. This protocol is called the Bermuda Triangle Protocol. First, the calculation node 20
0 is a write request along with write data to the primary I
Issued to ON212. This write request is transmitted to the primary ION 212 via the interconnect fabric 106. This is represented in step 1. The primary ION 212 stores the write data in a write cache located in the memory 304 and sends the write data to the buddy ION 214. This is represented in step 2. Next, the buddy ION 214 sends a confirmation message to the compute node 200 to confirm the write request. Finally, when the data is securely stored on disk, the primary ION 212 issues an eviction request to ION2.
Send to 14. This exclusion step is shown as step 3 in FIG. This reduces the communication requirements of the data processing architecture 100 and increases the processing power, since the methods shown in FIGS. 11 and 12 require five steps, whereas the above protocol requires four processes. .

FIG. 14 is a diagram showing the above operation in the form of a flowchart. First, the primary ION 212 receives a write request from the compute node 200 140
2. Next, from the primary ION 212 to the buddy ION2
The write data of the write request is transferred to 14
04. A confirmation message is transmitted from the buddy ION 214 to the computing node 200 1406, executing exclusion logic and
408, the write data stored in the buddy ION 214 is excluded.

FIG. 15 shows an embodiment of the exclusion logic. In this embodiment, when the write data is stored in the non-volatile storage of the primary ION 212,
An exclusion command is sent 1502 from the primary ION 212 to the buddy ION. Typically, this occurs when data is written to media. FIG. 16 shows another embodiment of the exclusion logic. In this embodiment, the write data is stored in the nonvolatile memory until the write data is stored.
2. The point that the exclusion command is not transmitted is the same as the embodiment of FIG. However, in this embodiment, it also waits 1604 for primary ION 212 to receive the second write request before sending the exclusion command. Therefore, interrupts are further reduced by delaying the exclusion request and combining it with the next write data transmission that generates three interrupt protocols. Another advantage of this protocol is that the buddy I
Even if ON 214 goes down, primary ION2
12 can process the request in write-back mode and signal the write after the data has been securely stored on disk. Compute node 200 is a buddy IO
There is no need to know the status of N214. In embodiments, a software timer or other device may be introduced to ensure that the last eviction command is sent even if there are no more write requests to be received by the primary ION 212.

In the Bermuda Triangle protocol, the write cache can be used with fewer interrupts than the conventional protocol, and at the same time, data availability can be maintained. This is possible because Buddy ION 214
Is to confirm the write request sent to the primary ION 212. Given the high cost of interrupt handling in modern pipelined processors,
The protocol can be used in a wide variety of distributed storage system architectures, resulting in lower overall system overhead and increased performance.

F. Compute node Overview The computing node 200 runs a user application 204. In conventional systems, the dedicated common SCS used
The number of I-buses was the same as the number of storage accessible to nodes in the cluster or clique. In the present invention,
The storage is connected to the compute nodes 200 through one or more communication fabrics 106. This network-attached storage is used for inter-process communication (IP) in the user application 204 distributed between the computing nodes 200.
C) The communication fabric 106 is shared with traffic. Storage requests from the user application 204 are put by the fabric / storage interface into IPC messages for the storage management application located at the ION 212. The dedicated application on such a storage node translates this IPC message into a local cache or I / O work and sends the result back to the compute node 200 as needed. For the user application 204, network-connected storage and local-connected storage are indistinguishable.

The read and write requests for the virtual disk block arrive at the ION 212 via the interconnect fabric 106. The route of the request to a particular ION 212 can be determined by the choices made by the source at the compute node 200. All compute nodes 20
0 knows which ION 212 will receive requests for each fabric virtual disk in the system. Fabric virtual disks reflect a virtual disk model in which a unique storage extension is described, but the storage extension does not imply or symbolize the physical location of the physical disk by its name.

Each compute node 200 has a list that maps fabric virtual disk names to ION dipoles 226. This list is for compute node 2
It is created dynamically by adjusting 00 and the ION 212. At power-on and failure recovery operations, the IONs 212 in the dipole 226 partition virtual (and physical) disks and create a list of which IONs 212 own which virtual disks. Dipole 226
The other ION 214 (which does not own the virtual disk or storage resources) provides an alternate path to the virtual disk in case of anomalies.

This list is passed through the interconnect fabric 106 to all other dipoles 226.
Is periodically exported or broadcast to the computing node 200. Compute node 200 uses this data to create a master table of the first and second paths to each virtual disk in the system. Then, the calculation node 20
The interconnect fabric driver in 0 is
Coordinate with the dipole 226 to determine the path of the I / O request. The dipole 226 uses this "self-discovery" technique to detect and correct virtual disk name mismatches that can occur when dipoles 226 are added or removed in a running system.

An application running on a compute node 200 views the block interface model as a local disk for each fabric virtual disk exported to the compute node 200. As mentioned earlier in this document, compute node 200 creates an entry point for each fabric virtual disk at boot time and compute node 20
These entry points are dynamically updated using the naming protocol established between ION 212 and ION 212.

This document has described a method and apparatus for transferring write cache data in a data storage and data processing system. In this method, a write request including write data from a computing node is received by a first I / O node, and the write data is received by a first I / O node.
/ O node to the second I / O node,
Sending a confirmation message from the second I / O node to the compute node after the / O node receives the write data. After writing data to the non-volatile storage of the first I / O node, an exclusion request or command is sent to the second I / O node to eliminate the write data from the volatile memory of the second I / O node. . In some embodiments, the eviction request is not sent until the first I / O node receives the second write request, in which case the eviction request is sent in the same interrupt as the write data of the second write request. . The data processing system comprises a first and a second I / O node, each node receiving a write request from a compute node and providing another I / O node.
There is means for transferring the write node to the O node. Also, each I / O node has means for sending a confirmation message directly to the computation node instead of sending a confirmation message via the I / O node that sent the write data. The result is an I / O that introduces a write cache to improve storage speed and turnaround while reducing the number of interrupts required for data storage.
Protocol. The present invention may be described in terms of a program storage device such as a hard disk, a floppy disk, and a CD that reliably implements instructions stored in order to perform an instruction to implement the present invention.

[Brief description of the drawings]

FIG. 1 is a top block diagram showing the main architectural elements of the present invention.

FIG. 2 is a system block diagram of the present invention.

FIG. 3 is a block diagram showing an ION and an interconnect structure of the system.

FIG. 4 is a block diagram of elements of a JBOD enclosure.

FIG. 5 is a functional block diagram of an ION physical disk driver.

FIG. 6 is a diagram showing a structure of a fabric unique ID.

FIG. 7: ION enclosure management module and ION
FIG. 3 is a functional block diagram showing a relationship between physical disk drivers.

FIG. 8 is a diagram of an interface on the BYNET host side.

FIG. 9 is a diagram of a PIT header.

FIG. 10 is a block diagram of an ION212 function module.

FIG. 11 is a diagram showing a protocol for writing data to a dipole disk.

FIG. 12 is a diagram showing a second protocol for writing data to a dipole disk.

FIG. 13 is a diagram showing an efficient protocol for writing data to an ION dipole disk.

FIG. 14 is a flowchart illustrating operations utilized in executing an embodiment of the write cache protocol of the present invention.

FIG. 15 is a flowchart showing an operation used when data is written to the nonvolatile storage of the first ION and then the memory of the other ION is excluded.

FIG. 16 is a flowchart showing an alternative operation used when data is written to the nonvolatile storage of the first ION and then the memory of the other ION is excluded.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Kit Em Chou United States of America 92009 Carlsbad Corvidae's Treat 1336 (72) Inventor Peeks Muller United States of America 92103 San Diego Marilouis Way 2440 (72) Inventor Michael W. Meyer United States 92024 California Encinitas Summerhill Driving 2323 (72) Inventor Gary El Boggs United States 92064 Poway Sycamore Tree Lane 13743

Claims (14)

[Claims] 1. A method for transferring write cache data in a data storage and data processing system, comprising: at a first I / O node, receiving a write request including write data from a computing node; From the first I / O node to the second I / O
Forwarding to the node; and transmitting a confirmation message from the second I / O node to the computing node. 2. The method of claim 1, further comprising the step of transmitting an eviction request from the first I / O node to the second I / O node. 3. The method according to claim 2, wherein the exclusion request is transmitted when the write data is stored in the nonvolatile storage of the first I / O node. 4. The method of claim 2, wherein the eviction request is sent after a first I / O node receives a second write request following a first write request. . 5. The method of claim 2, wherein the second write request includes second write data, and the method further includes transmitting the second write data and an eviction request to the second I / O node with a single data interrupt. 4. The method according to claim 3, comprising the step of: 6. The method of claim 1, wherein the computing node, the first I / O node, and the second I / O node are communicatively coupled via an interconnect fabric. Method. 7. The computing node is coupled to a first I / O node and a second I / O node via an interconnect fabric, wherein the interconnect fabric comprises the computing node and an I / O node. Is a network connected via a plurality of network input ports and a plurality of network output ports, where b is the total number of input / output ports of the switch node, N is the total number of network input / output ports, and g (x) is an argument Given a round-up function to obtain the smallest integer greater than x, in a network composed of a plurality of switch nodes arranged in switch node stages equal to or more than g (log _b N), the switch node stage can be any arbitrary Provide multiple paths between network input port and network output port The switch node stage provides a plurality of bounce points to the highest switch node stage of the network, and the bounce points logically distinguish switch nodes that load balanced messages through the network from those that direct messages in the network. And a network.
The described method. 8. An apparatus for transferring write cache data in a data storage and data processing system, comprising: a first I / O node receiving a write request including write data from a calculation node; From the first I / O node to the second I / O
An apparatus comprising: means for forwarding to a node; and means for transmitting a confirmation message from a second I / O node to a computing node. 9. The apparatus according to claim 8, further comprising means for transmitting an exclusion request from the first I / O node to the second I / O node. 10. A means for judging when to store write data in the nonvolatile storage of the first I / O node, and a means for transmitting an exclusion request when the write data is stored in the nonvolatile storage. 9. The apparatus according to claim 8, comprising: 11. A device for determining when to receive a second write request, and a unit for transmitting an exclusion request to a second I / O node together with the second write request. The device according to claim 9, characterized in that: 12. The method of claim 8, wherein the computing node, the first I / O node, and the second I / O node are communicatively coupled via an interconnect fabric. apparatus. 13. The computing node, a first I / O node, and a second I / O node are connected via an interconnect fabric, wherein the interconnect fabric comprises an I / O and the computing node. A network connecting nodes via a plurality of network input ports and a plurality of network output ports, where b is the total number of input / output ports of the switch node, N is the total number of network input / output ports, and g (x) is When a round-up function for obtaining a minimum integer larger than the argument x is used, in a network including a plurality of switch nodes arranged in switch node stages equal to or more than g (log _b N), the switch node stage can be arbitrarily set. Multiple paths between network input ports and network output ports And the switch node stage provides a plurality of bounce points to the highest switch node stage of the network, wherein the bounce point logically connects the switch node loading the balanced messages through the network and the switch node directing the messages in the network. 9. The network according to claim 8, further comprising a network for distinguishing.
The described device. 14. One or more computer-readable instruction programs readable by a computer and executable by a computer to perform the steps of transferring write cache data in a data storage system according to claims 1 to 7. A program storage device that reliably implements