AT505630B1 - DEVICE FOR COORDINATED TESTING AND TROUBLESHOOTING IN DISTRIBUTED EMBEDDED MICROPROCESSOR SYSTEMS - Google Patents

Description

2 AT 505 630 B1

The invention relates to a device for coordinated testing and debugging in distributed embedded microprocessor systems having a plurality of embedded system nodes connected via a communication network, the system nodes each having a computation unit, a clock module, a test unit, a debug unit and a communication interface for communicating with have the respective other system nodes via the communication network.

Overall states and processes can often not be reconstructed retrospectively in a distributed system. A diagnosis in case of error is made more difficult. Distributed systems do not share common memory and therefore must realize their entire communication by sending and receiving messages. Such communication is very susceptible to error, which can lead to problems due to falsification of messages, duplication of messages and the loss of messages. In addition, the message time is unpredictable, so you can never predict with certainty whether a system has failed or whether it has only a long response time.

In distributed systems it is very difficult or even impossible in practice to define an exact program sequence. Some development systems therefore refrain from using runtime debuggers, but usually allow the definition of breakpoints where the state of all variables after the program stop can be analyzed.

Real-time debugging usually provides the ability to process the code line by line while monitoring the states of various variables. Often, additional breakpoints can be set and other options can be used, which greatly simplify the work with the code. The difficulty with distributed systems is that the code is not just executed on one computer as usual, but on a variety of system nodes. Thus, a corresponding tool must support remote debugging and request debugging information from the other node.

In offline debugging, on the other hand, troubleshooting consists of evaluating recorded information. This information is usually obtained with the help of recording units (trace units) or generated by the applications themselves.

Various proposals have already become known to improve troubleshooting, as well as testing, monitoring, and rendering program instructions. In this context, the topic of clock synchronization was explored.

US 5,812,830 describes a simple circuit for troubleshooting in microprocessors. The presented system also includes various triggering mechanisms for storing internal processor states.

DE 102004052861 A1 describes a diagnostic system for detecting faulty conditions, which is realized as a separate system. Message paths are mapped in the diagnostic system. Synchronized clocks, support for a variety of nodes, and the ability to play recorded data are not used.

WO 2004/023302 A2 relates to a method for detecting errors in a distributed real-time computer system. The periodic dispatch of a status bit, e.g. via a turn-based communication network, it allows to detect the drop of a node and to react accordingly. No means or the like are cited to identify the cause of an error (e.g., by recording or reproducing data, or by special troubleshooting functions).

EP 943995 A2 describes a processor with an external "real-time instruction insert-on". The processor includes other instruction sources, such as a fetch loop, a subroutine register, pipeline protection, and allows external data access. The chip does not include a test unit or playback unit. There is also a performance penalty due to the inclusion of NoOps in the execution process to allow data access.

The document WO 2005/073855 A1 relates to a data processing system with a debug module which delivers real-time debug messages to a DMA controller.

US Patent No. 6,769,076 relates to a "real-time processor debug system" which enables the recording of programs and data of a virtual bus of an embedded system. The proposal includes the reduction of power consumption and the reduction of the impact on the performance of the overall system through the debug interface. An integrated logic is used which stores the virtual bus according to the address and operating mode. This patent is thus directed to recording the internal virtual bus of an embedded system. However, this patent does not address distributed debugging of embedded systems.

U.S. Patent No. 7,080,283 is directed to a method that enables real time recording and debugging of multiple processor cores in a system on a chip. It is based on a dedicated debug output of each single processor core. The debug information is stored by one or more recording storage nodes for each processor core. The recording storage cores, in turn, are interconnected to form a daisy chain. The daisy chain ends at a record control module which stores the desired debug information. The recording control module transmits the information to an external host agent via a JTAG interface. Furthermore, the debug data may be compressed via compression nodes located between the processor cores and the at least one recording storage node located between the cores. All this is in a SoC.

US 2007/0168731 A1 describes a debug method for microcontrollers that is aimed at low impact and low cost. It is proposed to use hardware support for 2-cable high-speed communication in future microcontroller applications. This 2-cable communication interface is intended to be used to transfer debug data between the target microcontroller and a link microcontroller. The link microcontroller is on the same chip as the target microcontroller, but is configured by an EEPROM fuse as a link microcontroller. The link microcontroller executes an interface translation program to establish communication between the target microcontroller and the host computer.

European Patent No. 764903 relates to an on-chip debug port. for VLSI processors. It includes a debug circuit in the same integrated circuit as the processor to be addressed. The debug circuit is controlled by an external debug workstation connected to the circuit via an external debug bus and an external interface module. The debug circuit includes a debug bus unit that connects to the external debug bus, a plurality of units, and two internal buses that connect the units to each other and to the debug bus unit. Each unit of the plurality of units in turn is connected to specific parts of the VLSI processor. With this configuration, it is possible to either send debug commands to at least one of the plurality of units or obtain debug information from the units. Options are possible, such as Sending / monitoring instructions from the processor, " real-time insertion / extraction " Task data related to the execution of tasks on the processor and the monitoring of the program flow.

U.S. Patent No. 6,324,684 describes a processor having a debug circuit that can communicate with a debug host. The host can send " debug_halt, 4 AT 505 630 B1 debug_step, debug_run- and debug_release " commands. The processor may also process interrupts during debug mode so that the system can be searched for errors in operation without the use of a debug monitor which would use more RAM, ROM and system overhead because of contextual storage and the like ,

US Pat. No. 6,470,388 describes a system that deals with debugging and logging services for a distributed computing environment consisting of workstations, servers and other network devices. The logging services include conventional formatting and conventional storage of information for the purposes of debugging. The logging services can send trace and debug information to a central logging facility where the data is time-stamped and stored in a database. The system described involves logging debug information in a distributed computer system, but has no coordinated single-step, timing, or hardware-level functions.

US 2006/0080575 A1 discloses a clock synchronization with a main computer, calculation of the drift value, calculation of the transmission delay, hardware timestamping in an optional sniffer unit and optional optical connections between nodes. It does not disclose the support of hardware tests with test, trace or playback functions, BIST, JTAG or coordinated playback and coordinated single-step.

US 2007/0061628 A1 describes a method for examining remote systems and collecting in real time debug data in a multi-node system. In this method, devices are monitored with probe machines that detect triggering conditions that cause the nodes to stop automatically and the available state data to be collected. However, this trigger condition is set statically and the operator is unable to perform debugging operators such as coordinated single-step.

US Patent No. 5,642,478 describes a method to acquire trace data in a distributed system. In this method, a node controller that performs the task of communicating with other nodes in the system includes detection logic that stores trace data in internal memory buffers. Such trace data are then routed regularly to an external memory. The beginning and the end of the trace data acquisition are triggered by other subcomponents of the system.

The present invention now aims to enable distributed debugging of embedded systems and in particular to ensure coordinated access to test structures. The invention further aims at not increasing the processor load of the microprocessors as much as possible by the debugging activities. Furthermore, the testing and debugging should be possible without a separate interface to the system node.

To solve these tasks, the device of the type mentioned for coordinated testing and debugging in distributed embedded microprocessor systems is characterized essentially by the fact that means for synchronizing the clock modules of the system nodes are provided, that the system nodes each with the test unit and the De Have bug unit connected discharge unit, so that via the communication interface incoming or outgoing, the testing and troubleshooting data packets sent via the discharge unit or receive and possibly processed, and that the clock modules with the respective test unit and the debug unit for Inserting timestamps into data packets and / or triggering test and / or debug operations in response to clock signals interacting. By providing means for the synchronization of the clock modules of the system nodes, a multitude of nodes can be achieved system-wide via coordinated recording or playback of complex processes or coordinated access to test structures. The provision of a relief unit provides the ability to extract control data pertaining to the test and debug unit and to insert data into the communication stream for analysis or transmission, thus eliminating the need for a separate interface to the test and debug units , Special packets are thus filtered out of the communication stream and processed by the unloading unit, so that the loading of the remaining components of the individual system nodes and in particular the processor load of the microprocessors is not increased. The debugging can be further controlled by the invention without any processor intervention. So you can, for example, a processor that has crashed, remotely re-boot and put in a working state.

In order to realize additional functionalities, the embodiment is preferably developed in such a way that at least one system node has a recording unit for recording system processes, such as e.g. of arithmetic operations of the arithmetic unit or system bus activities, which is connected to the discharge unit, so that via the communication interface incoming or outgoing, the recording unit concerned data packets can be sent or received via the discharge unit and possibly processed. In this case, it can preferably be provided that the clock module interacts with the recording unit for triggering and / or terminating a recording as a function of clock signals, so that system processes can be recorded coordinated in a plurality of system accounts and subsequently used for troubleshooting.

When troubleshooting, it can often be advantageous if recorded or predetermined program instructions or program operations are specifically provided to individual system components or " played " be able to analyze the behavior of the respective components. In this context, the training is preferably developed in such a way that at least one system node has a playback unit for reproducing recorded system processes or program instructions which is connected to the discharge unit, so that data packets arriving or departing from the communication interface are sent via the discharge unit or can be received and possibly processed. For coordinated playback in a plurality of system nodes, the design can advantageously be such that the clock module interacts with the playback unit to begin and / or end playback in response to clock signals.

In order to simplify the communication between the individual components, it is preferably provided that the arithmetic unit, the test unit, the debug unit and optionally the recording and the playback unit are connected to a common system bus, which advantageously also the clock module the system bus is connected.

Furthermore, it can be provided that the test unit is formed in integrated circuits as a self-test unit for this. Alternatively, it may be provided that the test unit is designed in integrated circuits as access to test structures in the entire embedded system node, whereby the test unit can also be used as a JTAG controller (so to speak "JTAG over Ethernet") for components lying outside the IC. Practical application would be e.g. the configuration of an FPGA or the updating of software via the unloading unit, but also testing of other chips.

According to a preferred embodiment, it is provided that a control and control unit is provided in the communication network, which is designed for sending test, debug, Aufzeich-, and / or playback commands to individual system nodes.

The invention will now be described with reference to exemplary embodiments illustrated in the drawings. It shows: 6 AT 505 630 B1

Fig. 1 is a functional block diagram of a digital embedded system incorporating the apparatus and method of the present invention;

Fig. 2 is a functional block diagram illustrating the relieving unit of the embedded system of Fig. 1 according to the present invention;

Fig. 3 is a functional block diagram illustrating the clock unit of the embedded system of Fig. 1 according to the present invention;

FIG. 4 is a functional block diagram illustrating the embedded system test unit of FIG. 1 in accordance with the present invention; FIG.

Fig. 5 is a functional block diagram illustrating the playback system of the embedded system of Fig. 1 according to the present invention;

Fig. 6 is a functional block diagram illustrating the trace system of the embedded system of Fig. 1 in accordance with the present invention; and Figure 7 is a functional block diagram illustrating the debug unit of the embedded system of Figure 1 in accordance with the present invention.

Fig. 1 shows a digital embedded system 12 which is improved according to the present invention. The node system 12 includes a system bus 28, a system memory 18, one or more processing units or microprocessors 14, a communication interface 16, and other interfaces 40, such as sensors or actuators. During normal operation, the system communicates via the communication controller 26 of the communication interface 16 with other system nodes of a distributed multi-node system connected by a network 20. However, some traffic, and in particular network packets with a certain type of content, are not forwarded by the communication controller 26 to the microprocessor but are transmitted to a dedicated unloading unit 24. This relief unit 24, together with the clock unit 30, a test unit 32, a display unit 34, a trace unit 36 and a debug unit 38, form a functional unit 42. All components of the unit 42 are exchanged between the units 30 via a bus 22 , 32, 34, 36 and 38 are connected to each other and between the units 30, 32, 34, 36 and 38 and the discharge unit 24. Via the unloading unit 24 it is possible to configure all units 30, 32, 34, 36 and 38 which are connected via the bus 22. It is a further object of the present invention to use the watch unit 30 together with the unloading unit 24 to precisely synchronize the nodes of a multi-node system. The system-wide synchronized clock can then be used to simultaneously initiate test play, trace, and debug functions on a plurality of nodes 12 of the multi-node system comprising apparatus 42 of the present invention.

Fig. 2 shows details of the relief unit according to the present invention. A frame filter 72 allows the traffic 64 to be picked out. The packets are stored in a receive buffer 70, and a special microcontroller 68 in the unload unit extracts the data and addresses of the packet and outputs that data via the bus interface 66 to the bus which provides the unload unit with the other units 30, 32, 34, 36 and 38 connects. When data needs to be sent, a corresponding packet is placed in the transmit buffer 74 by the microprocessor 68 and inserted into the traffic 64 in sequence.

Fig. 3 shows the clock unit according to the present invention. A clock unit 82, in particular an addition-based clock unit, is controlled by an algorithm in the control unit 88 and a bus interface 86 which must be synchronized with other clocks in the multi-node system. An event trigger unit 84 allows events to be triggered at given times or timestamped to activities.

Fig. 4 shows the test unit according to the present invention. On-chip BIST structures

Claims (10)

7 AT 505 630 B1 110, a JTAG controller 106 and a memory BIST provide the highest possible functionality for a node. This unit is addressed via a bus interface 104. Fig. 5 shows the reproducing unit according to the present invention. A playback buffer 126 may store the previous messages and a playback controller 128 may insert these messages into the network traffic for debug purposes in the node's microprocessor. The unit is configured via a bus interface 124. Fig. 6 shows the recording or tracing unit according to the present invention. A recording memory 146 is used to record bus traffic or microprocessor activities of interest. This function is controlled by the trace controller 148, which is configured via a bus interface 144. Fig. 7 shows the debugging unit according to the present invention. A debug interface 170 has access to debug control lines 168 of the node's microprocessor to obtain full debug access via debug control 166. The unit is configured via a bus interface 164. It is further an advantage of the present invention that the units for the clock 30, the test structures 32, the rendering functions 34, the trace functions 36, and the debug functions are close together and as a single unit for coordinated testing, coordinated playback, and coordinated debugging. This includes complex triggers for tracing, debugging, testing, and rendering, for example, such that tracing is started after some break condition has occurred in the debug unit 38. In summary, it should be noted that the inventive device for coordinated testing, debugging, data recording and data reproduction in distributed embedded microprocessor systems does not require its own interface. Debug capabilities are integrated into embedded system nodes of a distributed multi-node system to provide access to test infrastructure (eg Boundary Scan or Built-In Seif Test), debugger processor facilities (eg, Background Debugging Module of the processor), record system events (eg, store system bus activity into a local cache), and to obtain data rendering (eg, system bus or network traffic) of each node. Access is via a communication interface equipped with a Relief Unit, which is part of the device integrated in the node. The unloading unit extracts control data concerning the units for testing, debugging, data recording and data reproduction, and inserts data into the communication data stream for analysis or transmission. A separate interface to the units of the institution is not necessary. The exact coordination of all nodes of the multi-node system is carried out by synchronization of the local clocks, through which the units for test, troubleshooting, data recording and data playback are triggered. As microprocessor systems are to be understood within the scope of the invention, systems of microcontrollers. Claims 1. A device for coordinated testing and debugging in distributed embedded microprocessor systems having a plurality of embedded system nodes connected via a communications network, the system nodes each comprising a computing unit (14), a clock module, a test unit, a debug unit, and a communication interface for communication with the respective other system nodes via the communication network, characterized in that the clock modules (30) of the system nodes (12) are synchronized, that the system nodes (12) each with the test unit (32) and the debug unit (38 ) Relief unit (24), 8 AT 505 630 B1 so that via the communication interface (16) incoming or outgoing, serving for testing and troubleshooting data packets via the discharge unit (24) can be sent and received and possibly processed, and that the clock modules (30) with the respective test unit (32) and the debug unit (38) for inserting timestamps into data packets and / or triggering test and / or debug operations in response to clock signals interaction. A device according to claim 1, characterized in that at least one system node (12) comprises a recording unit (36) for recording system operations, e.g. of arithmetic operations of the arithmetic unit (14) or of system bus activities, which is connected to the unloading unit (24), so that via the communication interface (26) incoming or outgoing data packets concerning the recording unit (36) are sent or received via the unloading unit (24) and possibly can be processed. 3. Device according to claim 2, characterized in that the clock module (30) cooperates with the recording unit (36) for triggering and / or stopping a recording in response to clock signals. 4. Device according to claim 1, 2 or 3, characterized in that at least one system node (12) has a playback unit (34) for reproducing recorded system operations or program instructions, which is connected to the discharge unit (24), so via the communication interface (16) incoming or outgoing, the playback unit (34) pertaining data packets via the discharge unit (24) can be sent or received and possibly processed. 5. Device according to claim 4, characterized in that the clock module cooperates with the playback unit (34) for starting and / or stopping a playback in dependence on clock signals. 6. Device according to one of claims 1 to 5, characterized in that the arithmetic unit (14), - the test unit (32), the debug unit (38) and possibly the recording (36) - and the playback unit (34 ) are connected to a common system bus (22). 7. Device according to claim 6, characterized in that the clock module (30) is connected to the system bus (22). 8. Device according to one of claims 1 to 7, characterized in that the test unit (32) is formed in integrated circuits as a self-test unit for this. 9. Device according to one of claims 1 to 7, characterized in that the test unit (32) is formed in integrated circuits as access to test structures in the whole embedded system node (12). 10. Device according to one of claims 1 to 9, characterized in that a control and control unit in the communication network (20) is provided, which for sending test, debug, record, and / or playback commands to individual system nodes (12) is formed. Including 7 sheets of drawings