JP2001350555A - Method and device for displaying program operation state - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem such that it is difficult to understand the opera tion state of a program. SOLUTION: A 1st icon and a 2nd icon which shows the operation states of the program are displayed on a display device and in response to an operator' s indication corresponding to the 1st icon, a specific processing procedure is followed. According to the progress state of the processing procedure, the display of the 1st icon and 2nd icon is changed by following the set procedure.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a use form of a computer, a method of executing a program, and a method of creating a program.

[0002]

2. Description of the Related Art In recent years, a calculation model called object-oriented has been widely used. A calculation model is a basic idea and way of understanding when performing calculations.When implementing this model on a computer, etc., the data structure, data transfer, and data conversion procedures must be defined when and how. This defines whether or not to perform at the timing.

[0003] Terms used in the object-oriented calculation model will be described below. (A) Object This is a set of data structure and procedure. It has a slot of (b) as a data structure and a method (b) as a procedure. (B) The name of the data structure of the slot object. For example, an integer object has an integer slot, and a document object has a document slot. (C) Slot value Data actually held in the slot of the object. (D) Method The procedure defined in the object. By allowing only operations on slots, the modularity of the program is guaranteed. (E) Object instance The object is treated conceptually, and is not actually placed on the storage device. An object that actually has a slot value is called an object instance. (F) Message A command for executing a method. It is composed of arguments and method names. By limiting the arguments to object instances, the modularity of the program is guaranteed.

[0004] The object-oriented computation model is a computation model that allows computation only by exchanging messages between object instances. The result of the calculation appears as a change in the slot value of the object instance. For example, an integer object has an integer value slot as a slot and an addition method as a method. To design an integer object instance A having a specific value 3, the slot value is set to 3. When it is desired to add 4 to the integer object instance A, another integer object instance B is prepared, and the value of the integer value slot of B is set to 4, and a message "B addition" is sent to A. As a result,
The value of the integer slot of A is 7, which means that the desired result has been obtained. However, since it is cumbersome to generate all object instances for simple state quantities such as integers, simple state quantities are called atoms, and when they are used as arguments, their values are Allow it to be given to other object instants. For example, in the above example, an informal message of "4 addition" is sent to A.

[0005] By using this calculation model, the data and the procedure are stored in the same place, so that an easy-to-understand program with high modularity can be created.

For example, an icon representing an object instance is displayed on a display device, and an operator sends a message to the icon using a mouse or the like to convert the state quantity of the object instance, and further moves or deforms the icon. It is widely practiced to do so. For example, by pointing a certain icon with a mouse to indicate a corresponding object instance, interpreting a subsequently input “(movement amount) move” as a message for this object instance, executing it, and executing this icon Is to move. As a result of this movement, the value of the slot representing the position in the object instance changes.

[0007] In addition to receiving and executing an input of an operator as a message, programming is performed using an icon. That is,
The object instance is displayed on the display as an icon, and the user operates the object instance to generate a program. This is a programming by example (Programming by Example)
e: PBE), which displays various computational resources, such as files, application programs, and other object instances, as icons on a display device, and allows the operator to operate these icons using a mouse or the like. The system interprets the operation, performs the operation on the object instance indicated by the operation, and at the same time generates a command representing the operation. Thus, a desired program can be obtained by repeating the operation, execution, and command generation. Although an object instance has a specific value (example), it is characterized in that a system for interpreting an operation on the object instance generates a program for a more general object. The generated program can be applied to an example different from an example operated by a user at the time of creating the program. This is because, for an object instance, the operation interpretation for this example has been converted to a form applicable to all that belong to the object. In addition, since programming is performed using a specific example, the programmer does not need to generalize the problem as in ordinary programming, and programming effort is reduced.

[0008] As an example of PBE, a person called an agent is defined as an object instance corresponding to a program for managing the activation of the program, and this is displayed as an icon on a display device. Nikkei Computer is a system with an interface to request program execution
User Interface in the Era of Software Multimedia, pp. 165-178 (1989.
28 issuance) (hereinafter, this technique is referred to as a first known technique). This system displays an icon consisting of a two-dimensional figure with a human face on a display device, which represents an object instance of a management program called an agent, and uses a two-dimensional figure representing an object instance representing a file, a cabinet, etc. Is displayed, and the operator gives an instruction to the displayed agent icon and other displayed icons using a mouse or a keyboard. The agent interprets the input from the user, selects an appropriate program, and executes it. The function of the PBE of this system is to store the result of interpreting the input from the user, attached to the requested agent icon, and to generate an input example as a program.

Although the concept of an object or a method is not used, as a technique for facilitating program creation using icons, a report of the 30th Programming Symposium of the Information Processing Society of Japan, p. 9-17 (19
There is a program generation system (hereinafter referred to as a second known technology) described by the inventors in January, 1989). Here, on the display device, examples of data such as numerical data and array data are displayed as icons, and in order to specify a processing target of a command to be input, an icon corresponding to the command is selected on the screen. I have. Further, a command input at the time of creating a program is executed on the example data, and the result is notified to the operator. Depending on the result, the displayed icon is divided into a plurality of areas. By displaying the execution state of the program as described above, the operator can be notified of the current execution state of the program being created accurately and easily.

Further, in this system, if the execution of the program being created is unsuccessful, it is possible to create a program consisting of another command sequence.

[0011] Also, IEE Workshop on Visual Language (IEEE Wo)
rkshop on Visual Languag
e), pp. 80-85 1989 (hereinafter referred to as a third conventional technique) includes the following technique.

[0012] As an example of the processing object, a disk constituting the tower of Hanoi is displayed, and using this as an icon, a technique of creating a program in the same manner as the second known technique is displayed. Here, a command to move the disk is input as a part of the command when the program is created, and the command is executed. In addition, this movement is performed as a moving image. This makes it easier to understand the execution state of the program.

[0013]

In the second or third known technique, the operator is notified of the execution stage of the program by changing the display of the icon corresponding to the processing target according to the execution stage. However, the second and third known technologies do not have the concept of an object, a method, or a message as in the first known technology. That is, only a program consisting of commands for directly specifying a processing target is created. On the other hand, in the first known technique, as a method (program) to be created for a certain object, it is possible to create a program for executing a process including a process by a method assigned to another object. In that sense, it can be said to be a very general-purpose or advanced program creation method. However, in the case of the first known technique, the execution state of the program being created cannot be known.

Accordingly, it is an object of the present invention to provide a program creating method which enables an operator to know the execution state of a program for executing a process including a process already assigned to various processing targets. .

Another object of the present invention is to provide a method for executing a program having the same characteristics. Still another object of the present invention is to provide a method for displaying icons which makes computer input easier.

[0016]

In order to solve the above-mentioned problems, a program being created corresponds to the program being created near the point in time at which it requests a process already assigned to another processing target. The display of the icon is changed in relation to the icon corresponding to the other processing target. For example, the former icon is moved to the latter icon. In the execution stage of the created program,
At this point, the display is changed. In addition, as an icon for computer input, an icon obtained by projecting an object in a three-dimensional space onto a two-dimensional plane is used.

According to the execution stage of a program being created or already created, the icon corresponding to the program is displayed.
Since the requested processing is changed in relation to the icon for the processing target to which the processing is assigned, the user can know the execution state of the program.

Further, by using the three-dimensional icons, it is possible to input a computer using a screen with a very close feeling.

[0018]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Overview FIG. 1 is a diagram showing a configuration of a program generation system according to an embodiment of the present invention. As shown in this figure, the system comprises a display device 101, a mouse 109, a keyboard 1
And a main storage device 1 storing a program 10 for creating a program.
12, an auxiliary storage device 113, a CPU 114, a network device 115 for connecting to a large general-purpose computer 116 and a telephone line 117, and an audio output device 118.

The display screen of the display device 101 comprises screen areas 102 to 107 as shown in the figure. Various objects in the office space, such as a telephone 3104, a desk 3105, and a terminal 310 are displayed on the object display screen 102.
6 and a three-dimensional figure 3102 representing a secretary standing there when viewed from a virtual TV camera (not shown) at a certain point are displayed in advance. In this embodiment, an object referred to in the object-oriented calculation model is defined for each of these figures.
These figures are used as icons of the corresponding objects. A desired program is created as a method for the person 3102 using these icons. In this embodiment, in order to make the operator aware of the execution state of the created program after execution, the person 3102 is caused to perform an operation corresponding to the execution state of the program when the program is created. A command instructing this operation is input as a part of a program to be created. As will be described later, programming is performed while moving the person 3102. Motion control screen 103
Sliders 217-22 for instructing this movement
2 is displayed. On the view controller screen 104,
Sliders 230 to 235 for controlling the position or posture of the virtual VT camera are displayed in order to change the field of view of the office space. The method programming screen 105 displays commands and the like input to configure a program to be created. On the program control screen 106, a menu or the like for controlling execution or storage of a command input by the operator is displayed. An input device 109 is displayed on the terminal screen 107.
To 111, control information for notifying the operator of the state of the terminal including the display device 101 is displayed. The object modeling screen 108 displays information on an object represented by an icon selected by the operator among various icons displayed on the object display screen. Reference numeral 201 denotes a mouse cursor shown on the display screen.

In FIG. 2, 119, 120, 126
Reference numerals 136 and 140 413 denote various modules of the program 10 prepared in advance. The program creation program 10 interprets user input to create commands and executes these commands.

The following three databases are stored in the main memory 112 in advance for the program creating program 10.

(1) Information of each object, that is, an object instance library 124 for storing object instances. Each object instance includes the structure data of each object, an object slot, and an object method. One example is 3A
3C to 3C.

(2) A three-dimensional shape database 123 which is a set of graphic data of three-dimensional icons representing each object. An example of the graphic data is shown in FIG.

(3) A candidate speech database 125 used for speech recognition from the speech input device 111. An example of this data is shown in FIG.

Although these data are originally stored in the auxiliary storage 113, they are appropriately stored in the main storage 112 when required by the program creating program.

The program 10 for creating the program will now be described.
An outline of the processing will be described. (1) A command is created by interpreting user input information from the three input devices of the mouse 109, the keyboard 110, and the voice input device 111 by processes 119, 126, 127, 140 to 143, and stored in the command buffer 120. Is added to the command queue 121 at the top of the command queue stack 122. The above processing is performed by processing 119, 126, 127, 140-143 in FIG. The command is a person icon 310
A command to perform a desired operation is also input to the second or the like. At this time, various sliders in the motion control screen 103 are operated.

(2) Object method execution processing 13
In 5, the command is taken out from the command queue 121 and executed with reference to the three-dimensional shape database 123 and the object instance library 124. That is, if the extracted command has a method defined for the object to which the specified object instance belongs, the appropriate program library specified by the method is executed. When the command is, for example, a command for moving the person icon 3102, the display of the three-dimensional icon corresponding to the object is appropriately changed to a moving image. This allows the user to know which command was executed when the program was created. At the time of displaying the moving image, the execution process 135 updates the three-dimensional shape database relating to the object according to the program library, and the object display process 125 changes the display of the object display screen 102 immediately in response to the update result. . In this way, a three-dimensional image is displayed as a moving image by executing the command, and a sound for notifying the operator of the execution status is output by the sound output device 118 in accordance with the execution of the command.

(3) If the method required by the command is not defined in the methods of the object specified by the command, a new method is generated by using the object method execution processing 135.

(4) If there is any trouble as a result of executing the command, a command failure is reported.
In this case, the method specified by the executed command is not reprogrammed. This is also performed using the object method execution processing 135. In this way, the object method execution processing 135 performs various functions depending on the situation, and the command queue stack 122 functions to control it.

(5) The position and display state of the three-dimensional shape icon corresponding to the object instance can be changed at any time. This is performed by moving the valuators (bold vertical lines) of the sliders 217 to 222 on the motion control screen 103 left and right with a mouse. The result of the change is not stored as an instruction in the program being created during the programming. When a “store” button 216 in the program control screen 106 is selected with a mouse, a command sequence that reproduces an operation or a change in position until the change is generated is generated and stored in the program being created.

(6) As in (5), the state of projecting an icon having a three-dimensional shape on the object display screen 102 can be changed at any time. By selecting the storage button 216 with the mouse as in (5), a command for reproducing the state is generated and stored in the program.

As described above, in the present embodiment, the person icon 31 to which this program is assigned during program creation
There is one feature in that a command for causing a desired motion in 02 is input as a part of a command belonging to a program to be created. Further, when the program is created, this is executed, and the operator is notified programmatically of the execution state of the inputted command. Further, in this embodiment, the three-dimensional figure is used as it is, including the person icon. Other features of the present embodiment will be described in the following detailed description.

B. Components of System Hereinafter, before describing the details of the processing of the system of the present embodiment, the details of each component will be described.

B. 1 Database 1. Three-dimensional Shape Database 123 (FIG. 4) In this embodiment, a three-dimensional figure representing a person, a telephone, a desk, and the like arranged in an office space is viewed from a camera arranged in the space. It is displayed on a display screen, and the displayed figure is used as an icon for representing a corresponding object. The three-dimensional shape database 123 stores such three-dimensional figure data. As shown in FIG. 4, the data of each three-dimensional figure is composed of respective types and positions of a plurality of polyhedrons constituting each three-dimensional figure, attributes of each of the polyhedrons such as color and gloss, and a conversion matrix. Is done. FIG. 4 shows a receiver object instance 275 which is a subelement of the telephone object instance 2750 shown in FIG. 3B.
4 shows a part of the three-dimensional graphic data 2799 of FIG. Here, an octahedron and its vertex coordinates, color, gloss, its position, and direction are shown.

2. The object instance library 124 (FIGS. 3A to 3C) holds information (object instances) of various objects, for example, object structure data, object slots, and object methods. 3A to 3C show specific examples of object instances corresponding to a person, a telephone, and a camera, respectively. Here, the person object 2701 in FIG. 3A registered in the object instance library 124 will be described as an example.

The slot 2711 is the name of the attribute of the person object and has a slot value. Slot 27
Reference numeral 11 designates a name of each attribute and a data type relating to the attribute. In the example of FIG. 3A, the position 27 is set in the slot 2711 of the person object instance 2701.
There are three attribute names: 20, direction 2721, and name 2722, and the data type for each is a vector,
This indicates that data of a type called vector or character string is stored.

An object method is a procedure unique to an object. Here, the object method is a set of a name, a plurality of command strings corresponding to the name, and an argument object. Method calls at runtime are performed in the order of definition. In the example of FIG. 3A, a call method 2712, a goto method 2713, a tel_take method, 2714, c
connect (1) method 2715, connect
A t (2) method 2716 is defined. Here, the connect method has two definitions, and ordering is performed to indicate the order of the definitions.

In this embodiment, the object has an element composed of a tree structure. Each element is also an object. The same applies to object instances. The object structure data 270 in FIG.
As you can see from Figure 2, person object instance 2
703 has a body object instance 2704 as a lower element. The body object instance 2704 is the upper right arm (rum).
Object instance 2708, left upper arm (luar
m) It has an object instance 2705, etc. as lower elements. Among them, the ruarm object instance 2708 has an rlarm object instance 2709 as a lower element, and the rarm object instance 2709 has a rhand object instance 2710 as a lower element. Specific examples of the object instance of the element are omitted from the drawings and detailed description for simplification.

In this embodiment, each element has a three-dimensional shape. The three-dimensional shape is stored in the three-dimensional shape database 123 (FIG. 2). In addition, the object instance (not shown) of each element includes, as an attribute of the element,
The movable attribute and the rotatable attribute are held. In the example of FIG. 3A, (+) is written for an element having a movable attribute, and (*) is written for an element having a rotatable attribute. This is the same in other figures, for example, FIGS. 3B and 3C.

When the movable attribute and the rotatable attribute of the attribute of the element are immovable or non-rotatable, the element cannot be moved or rotated, respectively. In the example of FIG. 3A,
All the lower elements of the person object instance 2703 cannot have the movable attribute. This realizes a natural state in which each human joint can only rotate and cannot move.

As an instance for a special object, a camera object instance 2780 (third object)
C). Camera object instance 27
Reference numeral 80 denotes a slot 2790 as a camera position 2791, a horizontal direction 2792 indicating the horizontal direction of the camera's field of view, and an upward direction 279 indicating the vertical direction of the camera's field of view.
The vector type data is stored in each of the three attributes “3”. This object instance has a method 2795 as a built-in method. This method displays various three-dimensional figure data in the three-dimensional shape database 123 on the object display screen 102 (first
The image is converted into data in the coordinate system on the upper side, shading and rendering are performed, and the obtained figure is displayed on the object display screen 102.

The object display processing 136 (second
FIG. 7) is executed when the three-dimensional shape database 123 or the object instance library 124 is updated, or when there is another screen display request.

Object instance library 12
Reference numeral 4 is modified when an object instance is created or deleted.

3. Candidate voice database 125 (fifth
FIG.) The candidate voice database 125 stores the voice input device 11.
Fifth, used when recognizing the voice input from 1
It is a database that stores a plurality of candidate voice data 501 shown in the figure. Here, the candidate voice data 501 is composed of voice data 502, a data type 503, and a character string 504. The voice data 502 directly stores the voice input, the data type 503 indicates the type of the candidate voice data, and the character string 504 is a character string output as a result of voice recognition. Here, the data type 503 represents one of an object instance, an object method, an embedded instruction code, and a program controller. The following are the candidate voice data registered from the beginning. (1) Candidate audio data related to a built-in instruction (FIG. 3) described later. This data type is an embedded instruction code. (2) Candidate audio data related to an object instance defined from the beginning as an embedded object instance, such as a camera object instance. These data types are object instances. (3) Candidate voice data corresponding to the operator's "execute" voice input. The data type of the candidate audio data is program control, and the character string is “execute”. (4) Candidate voice data corresponding to a voice input of “memory” by the operator. The data type of the candidate audio data is program control, and the character string is “record”.

B. 2. Processing Hereinafter, processing by various programs in the present system shown in FIG. 2 will be described. 1. Input processing Input to the system is keyboard 109, mouse 1
10. This is performed using the voice input device 111. 1.1 Event Data (FIG. 6) As will be described later, a voice event generation process 143 and a mouse event generation process 14 respond to an input to the system.
0 and the keyboard event generation processing 141 generate event data. Event data 401
Is composed of an event type area 402 and an additional information area 403 as shown in FIG. Here, the event type 402 is any of motion control, view control, object selection, program control, command input, object modeling, or keyboard. The additional information 403 differs depending on the event type. When the event type is one of an object selection, a command input, and a keyboard, the event data is used to configure a command constituting a program to be created or a part thereof. Other types of event data are not directly related to command generation.

1.2 Speech Processing 1.2.1 Speech Input From the speech input device 111, the operator inputs speech as if instructing the process to be executed by the program to be created to the person icon 3102. . For example, "take a call" is entered. In the voice input processing, the input voice is recognized, and one or a plurality of necessary event data is generated. Not only a voice for instructing a process from the voice input device 111 but also an input related to the execution of the created command, for example, “execution” can be input. However, any type of voice needs to be an expression registered in advance.

1.2.2 Voice Recognition Processing 142 (Seventh
(FIG. 5A) In the voice recognition processing 142, as shown in FIG. 6, each of the different portions of the input voice sent from the voice input device 111 and all the candidate voice data 501 in the candidate voice database 125 (FIG. 5). ) Is compared with the audio data 503,
The candidate voice data most similar to each voice portion is extracted, and step 601 starts voice event generation processing 143 (FIG. 2) for each voice portion (step 603). If there is no similar candidate voice (step 602), nothing is output.

1.2.3 Audio Event Generation Processing 143
(FIG. 7B) As shown in FIG. 7 (B), the data type area in the candidate voice data extracted in the voice recognition processing 142 is checked (steps 701, 703, 705, 707) and depending on the type. Event data of a specific event type is generated (steps 702, 704, 706, 708), and the input event processing 126 (FIG. 2) is started (step 709).
The generated event data is as follows. In addition,
When the generated event data is generated, the additional information 40
For 3, the character string area 504 of the extracted candidate voice data 501 is used as it is. For example, the data type of the candidate voice data extracted for the voice portion representing the processing target is an object instance, such as “phone” of the input voice “take a phone call”. In that case (Step 7
01), event data whose event type is object selection is generated (step 702). For example, the data type of the candidate voice data extracted for the voice portion representing the processing content is the object method, such as "Take" of the input voice "Take the phone". In this case (step 703), event data whose event type is command input is generated (step 704). The data type of the candidate voice data extracted for the portion representing the embedded instruction (described later) in the input voice is an embedded instruction code. In this case (step 705), event data whose event type is command input is generated (step 706).

The above three event data are generated when an operator inputs an object instance or an object method as a processing target or an argument of a command to be generated or an instruction code of the command by voice. For example, the data type of the candidate voice data extracted for the input voice “store” or “execute” is program control. In this case (step 707), event data whose event type is program control is generated (step 708).
The event data corresponding to the input voice “execution” instructs execution of the command generated immediately before, but is not used for generation of the command. "Remember" is used to generate a command.

1.3 Mouse Input and Mouse Event Generation Process 140 (FIG. 8A) The display screens 102 to 102 in the display device 101 shown in FIG.
Mouse cursor 2 on any of 104, 106, 108
By setting 01, input using the screen becomes possible.
The mouse event generation process is a generic term for the mouse event generation process, and includes an object selection event generation process 304 and a motion event generation process 30 described below.
5, a view event generation process 306, a program control event generation process 307, and an object modeling event generation process 308. These processes 30
4 to 308 are activated when the mouse cursor 201 is on the display screens 102, 103, 104, 106, and 108, respectively, as shown in FIG. 8A. The event data generated in the object selection event data generation processing 304 is used for generating a command, but the event data generated in the other processing 305, 306, 308 is not used for generating a command. Process 307
Some event data are used for command generation and others are not.

1.3.1 Object Selection Event Processing 304 (FIG. 8B) This processing is executed when the operator inputs an object instance used as a processing target or an argument of a command to be generated with a mouse. That is, the operator places the mouse cursor 20 on an icon in the object display screen 102 representing the object instance desired to be input.
Move 1 and press the mouse button (not shown). In response to this mouse operation, an object selection event, processing 304, is activated. First, as shown in FIG.
The object instance selected by the mouse operation is recognized with reference to the dimensional shape database 123 (step 801), and the event data in which a character string representing the object instance selected as the event type and the selected object instance is stored as additional information is stored. Generate (Step 802). If the selected object instance is a component of another object instance, as this additional information, a character string connecting the name of the higher-level object instance to the name of the selected object instance separated by a period (.) Is stored. Then, the input event processing 126 (second
) (Step 803).

1.3.2 Motion event generation processing 305 (FIG. 8C) This processing is started when the operator instructs movement (or rotation) of the icons in the object display screen 102 using the motion control screen 103. Is done. That is, as shown in FIG. 1, six valuators 217 to 222 are displayed on the motion control screen 103. When one of the valuators is operated by the mouse 109, as shown in FIG. 8C, the event type is motion control, the name of the operated valuator is the additional information, and the scale on the support guide of the valuator slider. (That is, the movement amount or the rotation amount) is generated (step 901). Then, the input event processing 126 (second
) Is started (step 902). Before the variator operation, an object must be selected for the object display image 102.

1.3.3 View event generation processing 30
6 (FIG. 8D) In this process, the view control screen 10
In addition to using the six variators 230 to 235 in 4 to generate event data in which the view control is stored in the event type, the motion event generation processing 30
Perform the same processing as in FIG. 5 (FIG. 8D, steps 1001, 1
002).

1.3.4 Program Control Event Generation Process 307 (FIG. 8E) When the mouse cursor 201 is positioned on the "execute" button 215 (FIG. 1) in the program control screen 106, a mouse button (not shown) ) Is pressed, the eighth
As shown in Fig. E, event data in which program control is stored in the event type and "execute" is stored in the additional information is generated. This is used to direct the execution of the command just completed. This is the same as the case of the voice input “execute”. Similarly, when the “store” button 216 is pressed, event data in which program control is stored as the event type and “record” is stored as the additional information is generated (step 1101). In any case, the input event processing 126 (FIG. 2) is started (step 11).
02). The “store” button 216 is displayed on the object screen 1
Icon or camera displayed on 02 (not shown)
Pressed to generate a command to execute the movement or rotation after moving or rotating. The voice input "memory" is used as well.

1.3.5 Object Modeling Event Generation Processing 308 (FIG. 8 (F)) As shown in FIG. 8 (F), the mouse input on the object modeling screen 108 changes the event type into object modeling and additional information. Then, event data in which the input information is stored is generated (step 1201). Then, the input event processing 126 (FIG. 2) is started (1202).
The event data shown here is used for creating or modifying an object instance and is not directly related to creating a command.

1.4 Keyboard Event Generation Process 30
9 (FIG. 9) An input from the keyboard 110 (FIG. 1) is sent to the keyboard event generation processing 141 for each key operation. As shown in FIG. 9, when the mouse cursor 201 is on the object modeling screen 108 (step 1301), event data in which an object type is stored as an event type and a key code is stored as additional information is generated (step 1302). Mouse cursor 201
Is not in the object modeling screen 108 and the key code does not include a line feed code (1303), the event type storing the keyboard as the event type and the key code or the key code as the additional information is generated (step 1304). A key input in a state where a line feed code is not input indicates that information necessary for command creation is input but is not the last input for command creation. Therefore, the additional information of the event data is used as part of data of a command to be generated. The mouse cursor 201 is not on the object modeling screen 108,
If the key code does not include a line feed code at the end (step 1303), a command is input for the event type, and empty event data is generated for the additional information (step 1303).
5). The line feed code is input by the operator to indicate the last information necessary for generating the command, that is, the completion of the input of the instruction code. Therefore, this event data indicates that the command should be completed with the input data up to that time. Then, input event processing is started (step 1306).

1.5 Input Event Processing 126 The input event processing 126 includes a voice event generation processing 143 and a mouse event generation processing 140 as shown in FIG.
Is activated by the respective event generation processes of the keyboard event generation process 141, the event data 401 generated there is added to the input event queue 119.

The input event queue 119 is implemented as a first-in, first-out queue.
6 is an event queue 11 for storing event data.
Start from the end of 9. On the other hand, an event dispatching process 127 for extracting event data described below extracts event data from the first stored event data in the input event queue 119. This allows the input data to be processed in chronological order without loss.

2. Interpretation and Processing of Input Event Each of the event data stored in the input event queue 119 as described above is interpreted and processed one by one by the event dispatch processing 127.

2.1 Command (FIG. 10) The format of the command 1701 constituting the program to be created is as shown in FIG. That is, each part is composed of three parts composed of character strings, and each part is separated by a blank character.

(1) The first part 1702 is processing symmetric. This corresponds to a pointing object in the object-oriented model. Here, the name of the object instance to be processed is stored. This object instance contains the name of the object instance of the object itself, the name of a component of the object instance, the name of the slot of the object instance, or a character string indicating whether there is no processing target Is done. If there is no processing target, it is limited to special built-in commands.
Here, the component name of the object instance is expressed by putting a period (.) Between the object instance name and the component name, and the slot name of the object instance is defined between the object instance name and the slot name. It is expressed by a sharp (#). In this way, a unique name is given in the system.

(2) The next part 1703 is an argument. The argument 1703 is made up of an arbitrary number of argument objects. Argument objects are separated by white space. The number of arguments depends on the method. The argument object is either an object instance as described above or an atom. Atoms are either integers, floating point numbers, or strings.

(3) The third part 1704 is an instruction code. The instruction code 1704 is either a method name of an object instance or a built-in instruction code.

The command buffer 120 stores the command 170
One temporary storage location, in which one or a plurality of event data or a part of each of them is stored left justified as described later, and one command is completed.

However, the event data used to create the command is only for the event type of object selection, command input or keyboard.

2.2 Command Queue 121 (Eleventh
As shown in FIG. 11, the command queue 121 holds a plurality of commands completed using the command buffer 120 as a first-in first-out queue.
When a command is added to the command queue 1801, it is added next to the last command added, and when a command is taken out from the command queue 1801,
Take out the first command added.
The command queue 1801 is created or deleted depending on the execution status of the program.

2.3 Command Queue Stack 122
(FIG. 12) As shown in FIG.
Reference numeral 2 denotes a stack for storing a plurality of command queues 121, and only one is prepared in the system. Operations on the stack include push and pop, where push adds a new command after the most recently pushed command queue, and pop removes the most recently pushed command queue. An initial command queue 1903 is placed at the bottom of the stack.

2.4 Event Dispatch Process 127
(FIG. 13) As shown in FIG. 13, the event dispatching process 127 fetches event data one by one from the head of the input event queue 119 (FIG. 2) (step 140).
1) The processing of the event data is selected as follows according to the event type. The event queue 11
If 9 is empty, do nothing. If the event type is a keyboard (step 1402), this event data is data for command creation but not the last data for creation as described for the key bond event generation processing 309. That is, the key code of the additional information of the event data indicates a processing measure or an argument. Therefore, the key code of this additional information is stored in the command buffer 120 (step 1403). The storage location is a free space as much as possible in the command buffer 120. On the other hand, if the event type is object selection, command input motion control, view control,
If it is program control or object modeling (steps 1410 to 1415), respectively, an object selection process 130, a command input process 131, a motion control process 128, and a view control process 1
29, the program control process 132 and the object modeling process 133 are started (step 140)
4-1409). In the object selection process 130 and the command input process 131, event data is stored in the command buffer 120 for creating a command as described later. Step 1403 or step 14
Upon completion of 10-1415, the process returns to the processing of the first event in the input event queue 119 (FIG. 2).

2.5 Object Selection Process 130 (FIG. 14) When the event type of the event data interpreted in the event dispatch process 127 is object selection,
This processing 130 is started. As described above, such event data is generated when the operator uses the mouse to select a command processing object or an object instance to be used as an argument. In the object selection process 130, as shown in FIG. 14, additional information in the event data sent from the event dispatch process 127 (this indicates a process target or an argument)
Then, a blank character string is added, and the data is stored in the command buffer 120 with the MSB justified (step 2001).

2.6 Command Input Process 131 (FIG. 15) When the event type of the event data interpreted in the event dispatch process 127 is a command input, this process 131 is started. As described above, such event data is generated by inputting the last information (that is, an instruction code) of information used for a command to be created at the same time or after inputting the same. Generated in response to an operator input from the input device 111 or the keyboard. In the instruction input process 131, the fifteenth
As shown in the figure, the command is completed by adding the additional information (the end of which is an instruction code) of the event data sent from the event dispatching process 127 in the command buffer 120 (step 2101). The completed command is further transferred to the command queue stack 122.
Is added to the end of the command queue at the top (step 2102), and the command buffer 120 is cleared (step 2103).

2.7 Motion Control Processing 12
8 (FIG. 16) This process is started when the type of event data interpreted by the event dispatch process 127 is motion control. As described above, this type of event data is transmitted to the motion control screen 10 by the operator.
Operate the valuator in 3 and display the object surface screen 1
It is activated when moving or rotating the icon in 02. As described above, before this value operation, an object selection for designating an icon to be moved or deformed must be performed. That is, as a result of this object selection, the name of the object instance to be subjected to motion control must already be stored in the processing target portion in the command buffer 3120. This motion control processing 1
At 28, the icon corresponding to the object instance specified by the processing symmetric part is moved or rotated according to the valuator name and the movement amount or the rotation angle stored in the additional information of the event data given from the event dispatch processing 127. The movement or rotation is realized by modifying the graphic data in the three-dimensional shape database 123. If no processing target is stored in the command buffer 120, the following processing is not performed. More specifically, as shown in FIG. 16, when the valuator name is the x-direction movement 217 (step 1502), the object instance attribute to be processed is checked from the object instance library 124, and "movable" is determined. Confirm that there is (Step 15
05) The icon corresponding to the object instance indicated as the processing target is moved in the X direction according to the value of the valuator (step 1506). Even if the attribute to be processed is not movable in step 1505, if the object to be processed is a component of some object, the attribute of an element higher than the component is checked from the object instance library 124, and if it is movable,
The upper element is determined to be movable and is moved.
If there is nothing that can be moved even when following the upper component of the object, no processing is performed. When the valuator is moved in the y direction 218 and moved in the z direction 219,
The same processing is performed (steps 1503 to 1506). If the valuator name is rotation 220 around the x-axis, the attribute of the processing target is checked (step 1507), and it is confirmed that it is “rotatable” (step 1510), and the processing target is changed to the X-axis according to the value of the valuator. (Step 1511). Note that, even if the attribute of the processing target is not “rotatable” in the determination in step 1510, if the instruction target is a component of the object, the attribute of a higher element of the component is checked, and the attribute of “rotatable” is determined. if there is,
It is determined that the pointing target is “rotatable”, and in step 1511, the pointing target is rotated by rotating this higher-order element. If there is nothing that is “rotatable” even when following the upper component of the object, no processing is performed. Rotation processing on this is object library 1
24 is performed by referring to the contents of the description. The same processing is performed when the valuator is rotated around the y-axis 221 and rotated around the z-axis 222 (steps 1508 to 1511).

2.8 View Control Processing 129
(FIG. 17) The view control process 129 is performed according to the position of the camera (not shown), the type of the valuator whose horizontal and vertical directions are operated, and the slider scale of the valuator as shown in FIG. Control (step 1601-
1604, 1650, 1651). The six valuators 230-235 (FIG. 1) mean a forward command, a lift command, a horizontal movement command, a head-up command, a swing command, and a rotation command, respectively. For example, when the forward valuator 230 is operated (step 1601), the position slot value 2791 of the object instance camera 2780 (FIG. 3C) is changed in the direction of the direction slot value 2792 to the value of the additional information of the event data. The value is changed according to the value of the slider scale (step 1602). By moving and rotating the object instance camera 2780, a desired view can be selected on the object display screen 102. This movement or rotation is performed with reference to the object library 124 and the three-dimensional shape database 123. After the movement or rotation, the three-dimensional graphic data for the object is updated in the same manner as in the case of the motion control processing 128. .

2.9 Program Control Process 13
2 (FIG. 18) This process is started when the data type in the event data interpreted by the event taste patch process 127 is program control. As described above, this type of event is generated when "store" or "execute" is instructed by voice input, mouse input, or key input. As shown in FIG. 18, this program control processing 132 is executed if the additional information “record” in the event data is present (step 2201).
The state storage process 134 (FIG. 2) is started (step 22).
04), if the additional information is “execute” (step 2
202) Object method execution processing 135 (second
) Is started (step 2203).

2.10 Object Modeling Processing 1
33 (FIG. 19) This processing is started when the mother data type in the event data interpreted by the event dispatch processing is object modeling. This type of event data is generated using mouse input when newly defining or modifying an object as described above. The object modeling process 133 is one independent application program. Here, first, as shown in FIG. 19, a set of polyhedrons representing each object and respective attributes are newly defined (step 2601). Next, this object is hierarchically composed of a plurality of elements as illustrated in FIGS. 3A to 3C, and a new object is defined for each element. At this time, it is specified whether the component is movable or rotatable (step 2).
602). Next, an object name, an object slot name, and a data type included in the object slot are specified (step 2603). The information on the graphic representing the object and the information on the object itself are stored in the three-dimensional shape database 123 and the object library 124, respectively (step 26).
04). At any time, object modeling process 1
The object already defined in step 33 can be edited. For example, the person object 2703 in FIG. 3A
If a hierarchy is created so that the handset 2805, which is a lower element of the telephone 2803, is registered as a lower element of the right hand 2710, which is a lower element of, the receiver operates according to the movement of a human hand, It is possible to realize that a human picked up the receiver. Embedded instruction li described later
nk are thus realized. The names, shapes, and attributes of the objects defined in the object modeling process are displayed in areas 236, 237, and 238 of the object modeling screen 108 (FIG. 1). When modifying a created object, if there is a method defined for the object, the name of the method is displayed in area 239.

3. State storage processing 134 (FIG. 20) As described above, this processing is started by the program control processing 132 when storage of a created command is instructed by the operator. In the state storage processing 134, first, it is checked whether or not there is data indicating a processing target in the command buffer 120 (step 240).
1). If data indicating the processing target exists, the processing target state storage processing 2402 (FIG. 21) is started (step 2402). This data does not exist when the current state storage instruction is given by the operator after changing the field of view of the camera (not shown) using the view control screen 104, and the state of the object instance camera is stored. I do. That is, the current state (position, direction, and upward direction) of the camera (not shown) operated by the view control process 129 is stored in the instance 27 corresponding to the current state in the object library 124.
80 (FIG. 3c) to check (step 240).
3). Then, the command buffer 120 stores the camera object instance 2780 as a processing target, the current position, direction, and upward direction of the camera as an argument, and camera_set as an instruction code, and forms a command for a camera view set. It is added to the end of the command queue at the head of the command queue stack 122 (step 2404). The instruction code camera_set is
This is an instruction code for a built-in instruction for arranging a camera (not shown) corresponding to the camera object instance 2780 in the direction specified by the command argument. On the other hand, as a result of the determination 2401, it is determined that the data indicating the processing target is in the command buffer 120 because the operator uses the motion control screen 103 (FIG. 1) prior to the current state storage instruction. This is when an instruction to move or rotate an icon in the object display screen 102 is given by a mouse input. Therefore, when the processing target state storage process 240 is activated,
As shown in the figure, whether the processing target indicated by the data in the command buffer 120 is movable or rotatable is checked from the attribute data relating to the processing target object instance (steps 2501 and 2504). In determining the possibility of movement or rotation, refer to “2.7
In the same manner as described in "Motion control processing 129", an object instance that can be moved or rotated is selected from the object instance to be processed or the object instance that is a lower-order component as the object to be moved and the object to be rotated. If the object can be moved or rotated, the current position of the selected moving object instance and the current rotation angle of the rotating object instance are obtained (step 2502). Then, for each of the movement and the rotation, the object object to be moved or rotated, the obtained current position or the rotation angle, and the special instruction code tr related to the movement or the rotation are stored in the command buffer 120.
Stores anslate or rotate and generates commands for movement or rotation. Each command is added to the end of the command queue at the head of the command queue stack 122 (step 2502). By this processing, a command for executing movement and rotation of the icon designated by the operator can be embedded in the program.

4. Execution of Object Method In this system, execution of an object method and programming of an object method can be performed simultaneously. The system elements that achieve this are described below.

4.1 Object Method Execution Processing 1
35 (FIG. 22) As shown in FIG. 22, when the object method execution processing 135 is activated, the following processing is performed in accordance with a call command (a command targeted for this processing).

(1) If execution of a series of commands generated in advance of the command is successful, if the command is a command success_end for instructing to end the program execution (step 2801), method execution end processing 240
(FIG. 23) is started (step 2802), a success end is reported (step 2803), and control is returned.

(2) If the command is not success_end, it is checked whether or not the instruction code of the command is an embedded instruction code (step 2804). Then, the program is executed using the reed program library (step 2806). If the execution is successful (step 2807), the success is reported (step 2809), and control is returned. If the execution is not successful, the failure is reported (step 2808), and the control is returned.

If the call command is success_en
If the instruction code is not d and the instruction code is not a built-in instruction code, the command is defined by the user or is to be defined, and the following processing is performed.

(1) It is checked whether an unexecuted method exists among the methods already defined for the object instance specified by the processing of the command (step 2811). If it does not exist, the object method generation processing 250 is started (step 2812), and successful completion is reported (step 2813).
Return control. If it does, it means that the method has been defined by the user and proceeds to perform this.

(2) The first one of the unexecuted methods defined by the user for the object is selected (step 2815). For this selection, when a plurality of methods are defined for each object, they are arranged in the order of definition and arranged in the object instance library. Then, the object instance specified as the argument of the selected method is replaced with the object instance of the call command (step 2816).

(3) From among the unexecuted commands in the command string defining the method, the first one is selected (step 2817), and the object method is executed using this command as a call command (step 2817). 2818).

(4) It is checked whether an unexecuted command exists in the method (step 2819). If it does not exist, it means that the execution of the method has been completed normally, and reports success and returns control (step 2821). If it still exists, the method is not yet complete, so proceed.

(5) The result executed in (3) is checked (step 2820). If the execution is successful, the execution of the method is proceeding smoothly and returns to (3). Otherwise, if it has failed, execution of the method has been hindered, and the procedure returns to (1) to recover this.

4.2 Method Execution End Processing 2301
(FIG. 23) When the method execution end process 2301 is started by the method execution process 135 because the execution target command is “success_end”, the following processes are performed according to the called command as shown in FIG. If the called command corresponds to the command queue at the top of the command queue stack 122 (step 230)
2) Perform the following processing. Otherwise, return control. A command string in the command queue at the top of the command queue stack 122 is added to the command code method of the command, which is the target of the command (step 2303). And the command queue stack 1
Pop 22 to clear object instances created while the popped command queue is at the top, and clear the method programming screen corresponding to the command. (Step 2304) Then, control is returned.

4.3 Object Method Generation Processing 2
812 (FIG. 24) As shown in FIG. 24, the following processing is performed. A new command queue corresponding to the call command is created and pushed onto the command queue stack 121 (FIG. 2) (step 2910). In addition, a method programming screen corresponding to the method of the called command is generated (step 2921). Then, the method name, the instruction target and the object name to which it belongs, the argument and the object name to which it belongs are displayed on the screen (step 2).
930), control is returned.

4.4 Embedded Instruction Execution Process 2806 FIG. 25 shows an embedded instruction list 3001. In this figure, an object 3002 as a target of an embedded instruction,
The required argument 3003, the built-in instruction 3004, and the operation 3004 when the instruction is executed have been described. The actual target or argument is an instance of an object or an atom.

C. Description of Operation Hereinafter, the creation of a program according to the present embodiment will be described using a specific example. The first example creates a program to make phone calls. In the second example, a program is created that communicates with a large-scale general-purpose computer and uses an existing application program on the computer. In any case, a command instructing an operation to be performed by the person icon 3102 is input as a program command as a program command, and the command is executed to cause the person icon 3102 to perform the operation. This command sequence is stored in the person object as one of the methods executed by the person object.

1. Creating a program to make a phone call Here, create a program (method) for making a phone call. The method name assigned to the person object as a method is call.

A. Command input In the initial state, the object display screen 102
Suppose that it is as shown in the figure. Person icon 3102
(FIG. 1) represents the person object 2701 of FIG. 3A, and the telephone icon 3104 (FIG. 2) represents the telephone object 2750 of FIG. 3B. The telephone number object has not been described yet, but in the following it is assumed that it is already defined initially. Reference numeral 3103 denotes an icon representing this object, which has a rectangular shape and has a slot value telephone number at the center thereof. The instance name of the person object is rie, the instance name of the telephone object is tel, and the instance name of the telephone number object is tn. Hereinafter, these names are used as the names of the icons representing the corresponding objects.
The person icon 3102 is an arbitrary phone icon 3
104 from the phone icon 3104
The handset remains on. The number 012-3456 is displayed inside the telephone number icon 3103. In addition, as shown in FIG.
The command queue stack 122 within 12 has only an initial queue stack 1903. Here, in order to make the person icon 3102 make a call, the user inputs a command (0) rie tel tn call.
Here, other than “call” indicates a processing target, and “call” is a method name. (A-1) Command Processing Target and Argument Input In the following, the "processing target" refers to a command processing target portion unless otherwise specified or unless it is unnatural in context. In order to input this command, the operator uses the voice input device 111, the mouse 109, or the keyboard 111 to process a character string "rie" and a character string "tel".
And "tn" are input in this order. (A-1-1) In the case of voice input For example, to input a character string by voice, "Rie" is uttered toward the voice input device 111 (FIG. 1). The voice is transmitted from the voice input device 111 to the voice recognition process 142 (second
Here, as shown in FIG. 7A, the closest candidate voice data in the voice data area of the candidate voice database 125 is extracted (step 601), and the voice event utterance process 143 is started (step 601). 603).
The extracted candidate voice data is received, it is confirmed that the data type of the candidate voice data is the object instance name (step 701), the event type is object selection, and the additional information is the character string area of the candidate voice data. rie "is generated (step 702), and the input event processing 126 is started (step 709). Input event processing 126
Then, the received event data is added to the input event queue 119 (FIG. 2). Note that the candidate voice data is created as follows when the person icon 3102 is previously defined as an instance of the person object 2701. That is, at the time of this definition, at the same time as the operator inputs the name of the instance by keyboard, the system requests voice input of the name. When the operator voice-inputs this name, it is stored as the content of the candidate voice data for rie in the candidate voice database 125. Event dispatch processing 127 (FIG. 2)
Can operate in parallel with input by the operator and process event data in the mechanical event queue 119. In the present example, since the event type is "object selection" for the event data for "rie", the object selection processing 130 (FIG. 2) is started, and the additional information "rie" and a blank are stored in the command buffer 120 (step). 2001 (FIG. 14)).

(A-1-2) Mouse Input In order to input “rie” using the mouse 109 (FIG. 1), the mouse is placed on the body of the person icon 3102 displayed on the object display screen 102. Bring the cursor 201 (FIG. 1) and press the mouse button (not shown). Thereby, the object selection event generation processing 30
4 is activated (FIG. 8A). In this processing 304, as shown in FIG.
With reference to the dimensional shape database 123 (FIG. 2), it is recognized that the mouse button has been pressed on the (person icon 3102) (step 801), and the object type is selected for the event type, and additional information is input from the keyboard. Event data storing the character string "rie" is generated (step 802), and the input event process 126 is started (step 803). In the input event processing 126, the sent event data is stored in the input event queue 119.
Add to When this event data is later dispatched as in the case of voice input, the command buffer 120
"Rie" and a blank are stored in the "."

(A-1-3) Keyboard Input When "rie" is input using the keyboard 110 (FIG. 1), the mouse cursor 201 is placed at a place other than the object modeling screen 108 (FIG. 1). "R"
Type "i" and "e" as they are. The keyboard event processing 309 (FIG. 2) is started in response to the input of the burnt key. In this process 309, as shown in FIG. 9, after the judgments 1301 and 1303, the event type storing the keyboard as the event type and the key code given from the keyboard as the additional information is generated (step 1).
304), and activates the input event processing 126 (step 1306). Input event processing 126 adds this event to input event queue 119. When event data corresponding to such an input is dispatched later, the key code in the event data is used as it is in the command buffer 1.
20 empty areas are sequentially input from the front.
In this example, the processing target name “r” is stored in the command buffer 120 from the event data for each of “r”, “i”, and “e”.
ie ”is stored. As described above, in order to enable command input using any of voice input, mouse input, and keyboard input, each input from each is converted into event data in a common format. By the operation up to this point, the processing target “rie” to be included in the command to be input is stored. Subsequently, the processing target "te
l "and" tn "to the command. This operation is performed in the same manner as the previous input. The object selection is the same for the subsequent command input. Is
“Rie tel tu” is stored.

(A-2) Input of Method Name Subsequently, the name call of a newly defined method is input from the keyboard 110. Since this is the first method used in the system,
Only from 10 can be input. In response to this input, event data is generated as described above and stored in the input event queue 119. Subsequently, in order to input the end of the input of one command, when a line feed code is key-inputted as a command input instruction, the keyboard event processing 141 proceeds to step 1303 in FIG.
Event data whose event type is command input and additional information is empty are generated and added to the event queue 119. Since the event type of the event data is a command input, the event dispatching process 127 activates a command input process 131 (step 1405 (FIG. 13)). In this process 131, the contents of the command buffer 120 are added to the initial command queue 1903 at the top of the command queue stack 122, and the command buffer 120 is cleared (steps 2102 and 2102 (15th step)).
Figure)). Thus, (0) rie tel tn cal
1 has been sent to the initial command queue 1903. As described above, the character string input to the command buffer 120 is temporarily stored, and when a command input instruction is subsequently given by the user, the character string that has been input so far is used as one command. I have. In this example, the method name is input using the keyboard. However, if the candidate voice data 501 representing this method name is stored in advance as follows, an undefined method or a voice can be input. That is, the voice representing the method name is stored in the voice data 502 of the candidate voice data 501, the object method is stored in the data type 503, and “call” is stored in the character string 504. In this way, if the method call is input by voice following the argument of the command (0), the voice recognition processing (the seventh
From the candidate audio data extracted in FIG. A), in the audio event generation process (FIG. 7B), an event whose event type is a command input is generated. The generated event is equivalent to the event generated by the keystroke “call + line feed code” of the method name input using the keyboard described above. As described above, in the case of a command processing target or an argument, the type of data in the candidate data for the input voice is an object instance, and in this case, the input voice is stored in the command buffer. However, as described above, since the voice input representing the method can be treated as the end of the command to be input, the candidate data for the voice input should be combined with the candidate data for the already input voice into one command. Can be. As described above, if it is determined whether the input of the command is completed by determining the type of voice input by the operator, the operator does not need to use a special input such as a line feed code. The voice input above can be applied to the defined method as it is, of course. The character string stored in the initial command queue 1903 as described above is displayed on the program control screen 106 together with the command number (FIG. 26A). In the following:
Command input is performed using the method described above.

B. Executing the command The operator can control the program control screen 106 (Fig. 1).
Move the mouse cursor 201 over the “execute” button 215
And press the mouse button (not shown) here. The event dispatch process 127 starts the program control process 132 (FIG. 2) for the event data generated by this operation, and the object method execution process 135 is started during the execution of the process 132 (step 2203 (step 2203)). 18)). Here, the command queue 121 at the top of the command queue stack 122 is the initial command queue 1903 (26B).
), The first command of which is the command (0) described above, and the method call is undefined. However, the system invokes an object method execution process 135 that executes the command, regardless of whether the method in the command is predefined. In the process 135, the instruction code is s, as shown in FIG.
Since it is not access_end (step 2801), it is not an embedded instruction (step 2804), and it is not registered as a method of the designated object person (step 2811), the object method generation processing 2812 is started (step 2812). This process 2
At 812, as shown in FIG.
, A new call command queue 1903 (that is, an area matrix storing a command sequence to be included in the method call requested by this command) 1903 is generated and pushed onto the command queue area queue stack 122 (step 2910). . Further, a corresponding method programming screen 3110 is generated (step 2).
920), a method name "call", a set rie [person] of the instruction target and its belonging object, and a set tel [telephone] and tn [telephone number] of the argument and its belonging object are displayed on the object method programming screen (step 2930). ). This is preparation for defining a new method.

Thereafter, the operator inputs and executes each of the commands constituting the method "call". Here, (1) go to the telephone (rie tel # position goto) (2) pick up the telephone (rie tel tel_take) (3) make a call to the given telephone number. (Rie tel tn connect) (4) If the above is achieved, it is a success. (Success_end) is programmed. The operator issues a command (1)-
(4) is sequentially input. Here, # is for expressing the slot value of the object instance. Commands (1)-(4) are call command queues 31
21 in turn. call method queue 31
The command stored in 21 is displayed on the corresponding method programming screen 3110 each time it is stored. When the mouse cursor 201 is brought over the “execute” button 215 (FIG. 26A) on the program control screen 106 and the mouse button is pressed, a command queue 3121 corresponding to the call method is executed by the object method execution processing 135. Execute the first command (1). Here, the method goto is success_
The object method is not end (Step 2801), is not a built-in instruction (Step 2804), and is not defined as a person object (Step 2811). That is, here
(1) The method programming is suspended and goto method programming is performed. Therefore, this processing 28
12, the goto command queue 3020 (second
7B) is formed at the top of the command queue stack 122 (step 2910 (FIG. 24)). further,
The method programming screen 3210 for the goto method shown in FIG.
A method name, a processing target, arguments, and the like are displayed on the programming screen 3210 in the same manner as the screen 3110 according to the command (1) according to the command (1). (Step 2930). Thus, the queues in the command waiting area queue stack 122 are equivalent to the number of method programming screens and their corresponding methods, and serve to control method programming. The goto method is programmed to move a human to a certain position. Here, the following command sequence is input. (10) Create a temporary object instance that stores the distance dist scalar @ create (11) Create a temporary object instance that stores the direction dir vector @ create (12) Check the distance between rie and tel rie tel # position dist getdistance (13) Investigate the direction of te1 from rie. Rie tel # takeposiiton dirgetdirection (14) Turn rie in the direction of tel based on the result of (13). rie dir rotate (15) Make rie walk the distance of (1) rie dist walk (16) Turn rie in the direction of the phone rie tel # direction rotate (17) If the above is successful, use the goto method. finish. The success_end commands (10) and (11) are a distance object that stores a distance dist (this is a scalar amount) between the person icon 3102 and the telephone icon 3104, and a direction dir (this is a vector amount) of the telephone viewed from the person. This is a command for generating a direction object to be stored. In this embodiment, the command for generating an object is (object name) @create, which is a built-in command. In FIG. 27A, icons 3202 and 3203 represent a distance object and a direction object, respectively, and these icons are displayed according to the definition of these objects. When the objects are generated as described above, the names (dist, dir) of the respective objects are input by voice. Then, the voice input to the voice data of the candidate voice data is stored as an object instance in the data type, and character strings (“dist” and “dir”) each representing a name in the character string data, and stored in the candidate voice database. In the commands (12) and (13), the instances of the temporary objects generated by the commands (10) and (11) are respectively embedded instructions getdistance and getdirect.
The values of the distance dist and the direction dir are stored by using ion. Then, by the command (14), the person icon 31
02 is rotated by the direction dir, and is walked by the distance dist by the command (15). Next, in (16), the user is directed to the telephone. Finally, the command (17) (success
_End).

The above command string is the last command (1
After the input of 7), it is stored in the goto command string area 3230. Next, when execution is instructed, program execution processing 1
35 (FIG. 22), this command queue is executed one after another, and there is no cause for failure and success
_End (step 2801), go
The method execution end processing 2301 of the to method is performed (step 2802). Method execution end processing 2301
First, as shown in FIG. 23, the top of the command queue stack 122 is checked (step 230).
2). Since this is the goto command queue 3230, the command sequence in this is stored as a goto method (step 2303). Then, the command queue stack 122 is popped and the goto method programming screen 3210 is deleted. Also, dist320 which is temporary data generated by the goto method description
2, dir3203 into object library 124,
The three-dimensional shape database 123 is deleted from the object display screen 102 (step 2304). At this point, the top of the command queue stack returns to the call command queue 3121. call command queue 3
The remaining commands in 121 are (2) rie tel tel_take (3) rie tel TN connect (4) success_end. That is, cal which was temporarily suspended from here
I will resume l-method programming. Next, when the execution is instructed again, the call command queue 3
Execute the first command (2) of 121. tel_ta
Since the ke method is not defined in the instruction object human, the object method generation processing 2812
Is performed. That is, the call method programming is interrupted again, and the tel_take method programming is performed.

As a result of this process 2812, at the top of the command queue stack 122, there is a tel_take command queue 3320 as shown in FIG. 28A.
The method programming screen includes a screen 3110 for a call method and a tel_tak as shown in FIG. 28B.
There is a screen 3301 for the e-method. Go from earlier
As a result of execution of the to method, the person 3102
Standing in a position suitable for picking up 4 and facing the phone. Therefore, here, the right hand of the person 3102 is called the telephone 31.
The operation of extending the handset to the receiver of No. 04, picking up the handset, and bringing the right hand of the picked up hand near the right ear of the person 3102 is programmed. To make the upper right arm and the lower right arm of the person 3102 pick up the telephone 3104, first, the upper right arm and the lower right arm are alternately selected, and the motion control processing 128 is performed on them. Here, the x-axis rotation valuator 220 (FIG. 1), the y-axis rotation valuator 221 and the z-axis rotation valuator 222 on the motion control screen are operated.
As shown in FIG. 3A, since the upper right arm and the lower right arm have a rotatable attribute, they can be rotated at the right shoulder and the right elbow. Further, neither of them has a movable attribute, and if the user tries to move the attribute, the whole person moves as can be seen from the figure, so that no movement is performed. In this manner, the upper right arm and the lower right arm are properly operated, and the right hand is placed near the telephone handset by the motion control process 128 shown in FIG. The attribute is given in this way to express that a joint such as a human can only rotate and cannot move.

After the above motion control, "ri"
e. "ruarm" is input and stored as a processing target in the command buffer 120, and the program control screen 1
Mouse cursor 20 on the “Remember” button 216 on
By bringing the "1" and pressing the mouse button 202 here, the program control 132 is started, and the state storage process 134 is started during the process, step 2204 (FIG. 18). In the state storage processing 134, the processing target (rie, ruarm) is stored in the command buffer 120.
Therefore, the storage processing 210 is started for the processing target (step 2402 (FIG. 20)). In this processing, since the processing target in the command buffer is the upper right arm of the person 3102, which has only the rotatable attribute,
Rotate the upper right arm to the currently indicated rotation angle. Next command (20) rie. ruarm current rotation angle rotat
e is generated and the tel_take command queue 332 is
0 (steps 2504 to 2506). Similarly, by performing the processing target state storage processing 210 on the lower right arm of the person 3102, the following command (21) rie. rlarm current rotation angle rotat
e is added to the tel_take command queue 3320. Next, the handset of te13104 is held in the right hand of rie3102. The embedded instruction 1ink for this is used. The command is (22) rie. hand tel. receive
er link. This method link is realized by designating an argument object in a lower-level structure to be specified. As a result, thereafter, the receiver moves and rotates according to the position of the right hand of the person 3102. Further, even if the telephone 3104 is moved or rotated, the handset is not affected. In this state, the handset also moves in accordance with the movement of the telephone 3104. Therefore, the command (23) tel. body tel. receive
Run runlink. As a result, the object structures of the person 3102 and the telegram 3104 are as shown in FIG. 28A. Next, the upper right arm and the lower right arm of the person 3102 are again attached to the valuator rot_x (2
20), rot_y (221), rot_z (222)
, So that the handset is positioned exactly near the right ear.
Start As a result, the command (24) rie. ruarm current rotation angle rotat
e (25) rie. rlarm current rotation angle rotat
e is added to the tel_take command queue 3320. Add (26) success_end as the last command for this method. Then, these commands are executed one after another, and when reaching the success_end, the method execution end processing 2301 is performed. The command sequence (20)-(26) describing the tel_take method is stored, the command queue stack 122 is popped,
The l_take method programming screen 3301 is deleted. The next command in the call command queue 3121 is (3) rie teln connect. The connect method is also a human object 27
01 (FIG. 3A), a new command queue 3430 is created, voice registration is performed, and object method creation processing 2812 is performed. As a result, a connect command queue 3430 is created at the top of the command queue stack 122 (FIG. 29B), and the method programming screen displays two, 3110 corresponding to call and 3420 corresponding to connect. (FIG. 29A). Here, (30) a connection request is issued to the specified telephone number. Tel TN connect_request (31) If the connection is successful, the effect is output. success_end.

By executing this command sequence, (32)
When it reaches success_end, a method execution end process 2301 is performed, the command sequence (30)-(32) is stored as a connect method, the command queue stack 122 is popped, and the connect
The t-method programming screen 3420 is deleted. Here, the top of the command queue stack is the call command queue 3121, and the top command is (4) success_end. When this is executed, the method execution end processing 2301 is started, and (1)-
(4) The command sequence is stored as a call method, the command queue stack 122 is popped, and the call method programming screen 3110 is erased. Thus, the programming of the call method originally intended is completed. The top of the command queue stack 122 here is the initial command queue 312.
0. This completes the call method programming. From here on, enter "Execute" and enter "Person object instance Phone object instance"
If the command "telephone number object instance call" is sent to the object method execution processing 135, the person object instance moves to the telephone position, picks up the receiver, issues a connection request to the specified telephone number, and connects. If it succeeds, it will say "connected". Also, "person object instance vector object instance goto", "person object instance
Phone object instance tel_take ",
"Person object instance Phone object instance Phone number object instance co
Each command of “nect” is also defined in the process of the call method definition. Since the execution of the command is output as an operation of the requested object instance or as a voice, it is easy to recognize the state. However, it may fail somewhere while executing the command. In this example, the next command that issues a phone connection request,
(3) In rie tel TN connect (30) rie tel TN connect_re
It is conceivable that the quest fails because the other party is busy. However, one of the features of the present invention is that it is not necessary to write a program until a failure actually occurs.

If the command (30) fails while the command is being executed several times, the program can be described by using the failed example. In the example of FIG. 29A, it is assumed that a failure has occurred when the person 3102 uses the telephone and uses the telephone number object instance tn1. The command is (40) rie teln
1 connect, and the command (3
0) replaced with the current example (30) rie
tel tn1 connect_request.

The object method generation processing 2812 (FIG. 24) of the command (40) is performed in the same manner as described above. The program here is for a failed phone connection. Here, the command sequence is as follows. (41) A message is output with a voice saying “Failed” rie “Failed” spoke (42) Rotate the handset to a position that fits into the main unit rie. receiver The direction in which the handset fits in the telephone. rotate (43) Place the handset in the position of the main body rie. receiver tel # position translation (44) Disconnect the handset from rie's right hand rie. receiver tel. body link (45) Return handset and body to original relationship rie. hand rie. receiver unlink (46) erect upright rie neutral (47) If more than success, end success_end.

Here, when the commands of (44) and (45) are executed, the structure of the object becomes as shown in FIG.
It becomes as shown in. This is the state defined first by the person object 2701 and the telephone object 2750. From then on, the person and the phone will be the normal person object 2
701, operates as a telephone object 2750. The above commands are sequentially executed (47) success_e
If it reaches the nd command, method execution end processing 2
301 is performed, and the command sequence (41)-(47) is newly stored as the connect method of the person object 2701 without losing the one defined in FIG. 29B. Now, con for the person object
The command sequence (30)-(3
2) and two command strings (41) to (47) are defined. Since the method execution is performed in the order of definition, when the "person object instance telephone object instance telephone number instance connect" command is executed, a specified connection request is first issued using the command sequence defined in (30)-(32). as a result,
If it succeeds, an audio output of "successful" is output. If the operation is not successful, the process proceeds to steps (41) to (47), outputs a voice message "connection failed", and stands up with the handset placed. The secretary now has a method of making a phone call. Person object 2
701 is, as shown in FIG. 3A, a call method (2
712), goto method (2713), tel_t
An ake method (2714), a connect (1) method (2715), a connect (2) method (2716), and a group of embedded instruction codes shown in FIG. .

2. Program Using Communication with Large-Scale General-Purpose Computer 116 Next, features of the embodiment of the present invention will be described using an example in which reservation of a conference room is programmed using a large-scale general-purpose computer. It also refers to programming the camera's field of view. Person object instance r
ie (3102), terminal object instance te
rm (3612), date object instance da
te3610, room object instance room
The method yoyaku is described using 3611. Here, it is assumed that the object instances are arranged in a three-dimensional space as shown in FIG. 31A. First, the camera object instance is view 1 (370
1) and the terminal object instance te
It is assumed that the rm (3612) is not displayed in a partition 3602 in the object display screen as shown in the object display screen 1 (3601) in FIG. 31A. Therefore, this cannot be selected by the mouse 109.

The method is defined as (50) rie term date room yo
and yaku. The term (3612) is input by voice or a keyboard. When this is executed, the object method generation processing 2812 is started, the yoyaku command queue 3630 is pushed onto the command queue stack 122, and the method programming screen 3620 is generated. Here, if the terminal is not actually displayed, it becomes impossible to know what the user is operating. Therefore, the valuator on the view control screen 104 is operated to change the view to the view 2 (3702) in which the term 3612 is visible. Then, as shown in the object display screen 2 (3602), the term36
If 12 is displayed, it can also be selected with the mouse 109. As described above, when an object that is defined as a three-dimensional shape in the computer is selected using the display device and the mouse, if there is an object that is not projected on the display device, the view is changed to change the view. And then select it. In this program, the operator executes a state storage process 134 for the camera on the assumption that the operator wants to work in this view. For this purpose, the processing target of the command buffer may be left empty. When the state storage processing 134 is executed, (51) a position in which the camera term can be viewed A direction in which the term can be viewed A camera_set command is generated. This is a special command for changing the view. Next, using the predefined method goto, (52) rie term # position got
o command and add rie3102 to term3612
Walk to the place of. The command code of the next command (53) term parentminal is a built-in command, which opens a communication path with a large general-purpose machine and logs on to it. At the same time, the terminal screen 107 is displayed on the display device 101. Next, the command (54) term date room get re
servation is executed to execute the reservation application program on the large general-purpose computer, and date3610 and room361 are executed.
1. Make a reservation for the conference room according to 1. If the above is successful, add the command (55) success_end to the yoyaku command queue 3630. When this is executed, the command sequence (51)-(55) becomes yo
Stored as a yaku method, command queue stack 122 is popped and yoyaku method programming screen 3620 is cleared. In this way, it is possible to set a specific view position, place the camera at that position, and specify to execute the object method. In addition, work using a large general-purpose computer can be performed.

D. Modifications Up to now, three-dimensional
The example in which the dimensional shape is displayed on the object display screen 102 has been described, but this is not necessarily essential to the present invention. Here, a method for realizing a system that does not use a three-dimensional shape will be described. As an example, 1. The following describes the program for making a telephone call described in. The problem in realizing a program for making a telephone call in a two-dimensional system is how to express that a requested object instance goes to a telephone object instance and has a telephone. To do this, as shown in FIG. 33, the requested object instance 3801 is moved to the object instance 3802 to be processed, and an icon representing the call is displayed in order to express that the call has been taken. May be emphasized such as blinking. Alternatively, both may be emphasized without moving. Also, as shown in FIG.
Icon 39 of the requested object instance
01 and the icon 3902 to be processed may be changed. Further, as shown in FIG. 35, an arrow may be displayed from the requested icon 4001 toward the processing target icon 4002, or from the processing target icon 4002 toward the requested icon 4001. .

As described above, the essential point of the present invention is that when the execution state of a program is represented on a display device, the program for managing the execution of the program library is displayed in a graphic form, and this is displayed in the execution state. That is, the display is changed accordingly. Further, in this embodiment, a command (2) is input to make the person icon 3102 pick up the telephone, and a command string (2
0) to (26). However, for the purpose of informing the operator of the execution state of the program, the command sequence (20) to (26) is embedded in advance in the method of the command connect of the command (3) for making a telephone call. In response to the input of the command (3), the person 3102 can be moved to the telephone near the operation of making a telephone call and made to pick up the receiver. Further, in the present embodiment, the handset is moved by linking to the person icon 3102, but in response to the command connect of the command (3),
You can move the handset and do it. This also allows the operator to know what the method of the other object requested by the program being created is.

[0108]

According to the present invention, the display of the icon corresponding to the program changes according to the stage of execution of the program in relation to the icon representing another processing object, so that the user can intuitively grasp the execution state of the program. be able to. In the case where the icon corresponding to the program is in the form of a person, there is a secretary in the computer as viewed from the user, and the execution of the program can be instructed as if requesting a job. In general, when the icons used for the computer power are converted into three-dimensional images, an interface that is more intuitive and easy to understand can be realized. In addition, a computer operation environment that can be easily operated, particularly when voice input / output is used for general computer input, will be provided.

[Brief description of the drawings]

FIG. 1 is an overall configuration diagram of a system for realizing the present invention.

FIG. 2 shows functional blocks of a program for creating a program held in a main storage device shown in FIG. 1 and a database related thereto.

FIG. 3 is a diagram showing various object instance information used in the present embodiment.

FIG. 4 is a diagram showing an example of graphic data in the three-dimensional database shown in FIG. 2,

FIG. 5 is an explanatory diagram concerning the structure and processing of candidate audio data in the audio database of FIG. 2;

FIG. 6 is a structure and explanatory diagram of event data processed by the program of FIG. 2;

FIG. 7 is a flowchart relating to voice input processing.

FIG. 8 is a flowchart relating to mouse input processing.

FIG. 9 is a flowchart relating to keyboard input.

FIG. 10 is a format diagram of a command.

FIG. 11 is a structural diagram of a command queue.

FIG. 12 is a structural diagram of a command queue staff.

FIG. 13 is a flowchart of an event dispatch process for interpreting a user input.

FIG. 14 is a flowchart relating to a process of activating an event dispatch process.

FIG. 15 is a flowchart relating to a process of activating an event dispatch process.

FIG. 16 is a flowchart relating to a process of activating an event dispatch process.

FIG. 17 is a flowchart relating to a process of activating an event dispatch process.

FIG. 18 is a flowchart relating to a process of activating an event dispatch process.

FIG. 19 is a flowchart relating to a process of activating an event dispatch process.

FIG. 20 is a flowchart relating to a state storage process in which a program control process is started.

FIG. 21 is a flowchart relating to a state storage process in which a program control process is started.

FIG. 22 is a flowchart relating to a method execution ending process called out in the object method execution process.

FIG. 23 is a flowchart relating to a method execution end process called out in the object method execution process.

FIG. 24 is a flowchart relating to a method execution ending process called out in the object method execution process.

FIG. 25 is a diagram showing a group of embedded instructions used in the program of FIG. 2;

FIG. 26 is an explanatory diagram of an embodiment using a specific example, explaining a state on a display device and a main state on a main storage device.

FIG. 27 is an explanatory diagram of an embodiment using a specific example, explaining a state on a display device and a main state on a main storage device.

FIG. 28 is an explanatory diagram of an embodiment using a specific example, illustrating a state on a display device and a main state on a main storage device.

FIG. 29 is an explanatory diagram of an embodiment using a specific example, illustrating a state on a display device and a main state on a main storage device.

FIG. 30 is an explanatory diagram of an embodiment using a specific example, illustrating a state on a display device and a main state on a main storage device.

FIG. 31 is an explanatory diagram of an embodiment using a specific example, explaining a state on a display device and a main state on a main storage device.

FIG. 32 is a diagram illustrating an arrangement of an object instance in a three-dimensional space and a view for projecting the arrangement.

FIG. 33 is a diagram illustrating a method of easily realizing the present invention in a two-dimensional system.

FIG. 34 is a diagram illustrating a method of easily realizing the present invention in a two-dimensional system.

FIG. 35 is a diagram illustrating a method of easily realizing the present invention in a two-dimensional system.

[Explanation of symbols]

135: Object method execution processing, 123: Three-dimensional shape database, 124: Object library, 122: Command queue stack, 125: Candidate voice database, 142: Voice recognition processing, 143: Voice event generation processing

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] May 7, 2001 (2001.5.7)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Name of invention

[Correction method] Change

[Correction contents]

Patent application title: Program operation status display method and apparatus

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] Claims

[Correction method] Change

[Correction contents]

[Claims]

[Procedure amendment 3]

[Document name to be amended] Statement

[Correction target item name] 0001

[Correction method] Change

[Correction contents]

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer
Related to the technology of displaying the situation.

[Procedure amendment 4]

[Document name to be amended] Statement

[Correction target item name] 0013

[Correction method] Change

[Correction contents]

[0013]

In the second or third known technique, the operator is notified of the execution stage of the program by changing the display of the icon corresponding to the processing target according to the execution stage. However, the second and third known technologies do not have the concept of an object, a method, or a message as in the first known technology. That is, only a program consisting of commands for directly specifying a processing target is created. On the other hand, in the first known technique, as a method (program) to be created for a certain object, a program for executing a process including a process by a method assigned to another object can be created. In that sense, it can be said to be a very general-purpose or advanced program creation method. However, in the case of the first known technique, the execution state of the program being created cannot be known. The purpose of the present invention is to
A program that executes the process assigned to the process target
Operation status that allows the operator to know the execution status of the program
It is to provide a display method and an apparatus .

[Procedure amendment 5]

[Document name to be amended] Statement

[Correction target item name] 0015

[Correction method] Deleted

[Procedure amendment 6]

[Document name to be amended] Statement

[Correction target item name] 0016

[Correction method] Change

[Correction contents]

[0016]

According to the structure of the first aspect,
Thus, the above problem can be improved.

[Function] According to the execution of a program,
The display of the corresponding icon indicates the processing content of the program
The program execution status.
You can know the state.

[Procedure amendment 7]

[Document name to be amended] Statement

[Correction target item name] 0017

[Correction method] Deleted

[Procedure amendment 8]

[Document name to be amended] Statement

[Correction target item name] 0108

[Correction method] Change

[Correction contents]

[0108]

According to the present invention, it is possible to respond to the execution of a program.
The icon corresponding to the program is displayed
Changes depending on the processing contents of the program
You can know the execution status of the program.

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Yoshiki Matsuda 1-280 Higashi-Koigabo, Kokubunji-shi, Tokyo F-term in Central Research Laboratory, Hitachi, Ltd. 5E501 AA02 BA05 CA03 CB02 CB09 CB15 EA11 FA04 FB22 FB25 FB44

Claims (14)

[Claims] 1. A processing status display method in a processing device having a display unit, wherein an icon indicating an operation status of a program is displayed on the display unit, and a predetermined processing procedure is executed in response to an instruction for the icon. Regardless of whether the display position of the icon has changed,
A processing status display method, wherein a display mode of the icon is changed according to a progress status of the processing procedure. 2. The processing status display method according to claim 1, wherein the instruction for the icon is performed based on an input voice. 3. The method according to claim 1, wherein a plurality of events are allocated in accordance with an instruction for the icon, and a display mode of the icon is changed in accordance with the progress in response to the event. Processing status display method. 4. The processing status display method according to claim 1, wherein the change in the display of the icon is at least one of a swing command, a swing command, and a rotation command. . 5. The processing status display method according to claim 1, wherein when the instruction is an instruction to make a telephone call, the display mode of the telephone icon is changed to a state where the handset is lifted. 6. The processing status display method according to claim 1, wherein the change in the display mode of the icon displays the icon near another icon. 7. A processing status display device using a processing device having a display means, a display control means for displaying an icon representing an operation target and an icon for operating the operation target on the display means, and an instruction for the icon Execution means for executing a predetermined processing procedure in response to the display control means, wherein the display control means executes the processing of the icon in accordance with the progress of the processing procedure irrespective of whether or not the display position of the icon has changed. A processing status display device characterized by changing a display mode. 8. The processing status display device according to claim 1, wherein the instruction for the icon is performed based on an input voice. 9. The display control unit according to claim 2, wherein a plurality of events are assigned in accordance with an instruction for the icon, and a display mode of the icon is changed in accordance with the progress according to the progress. The processing status display device according to claim 1. 10. The processing status display device according to claim 1, wherein the change in the display of the icon is at least one of a swing command, a swing command, and a rotation command. 11. The processing status display device according to claim 1, wherein when the instruction is an instruction to make a telephone call, the display mode of the telephone icon is changed to a state in which the handset is lifted. 12. The processing status display device according to claim 1, wherein the change in the display mode of the icon is performed by partially displaying the icon with another icon. 13. A processing status display method in a processing device having a display means, wherein an icon indicating an operation status of a program is displayed on the display means, and a predetermined processing procedure is executed in response to an instruction for the icon. A processing status display method characterized by the following. 14. A processing status display method using a processing device having a display means, wherein an electronic secretary icon representing an electronic secretary is displayed on the display means, and a predetermined processing procedure is performed in response to an instruction for the electronic secretary. Executing, irrespective of the presence or absence of a change in the display position of the electronic secretary icon, changing the display mode of the electronic secretary icon based on a set procedure according to the progress of the processing procedure. The processing status display method to be performed.