JP7546390B2 - Information processing device and information processing system - Google Patents

Description

The embodiments disclosed in this specification and the drawings relate to an information processing device and an information processing system.

In the field of pathology and the like, there is known a technique for storing, as digital data, a specimen image in which an entire tissue slice serving as a specimen is captured, and a specimen image in which a tissue specimen cut out from the specimen is captured at a higher magnification than the specimen image. Such pathological specimen images and specimen images are used, for example, by doctors for pathological tissue diagnosis.

JP 2019-000340 A JP 2019-095853 A

One of the problems that the embodiments disclosed in this specification and the drawings attempt to solve is to make it easier for doctors and other such personnel to grasp the correspondence between the cut-out position of the specimen tissue in the specimen depicted in the specimen image and the region of interest, such as a lesion region, on the specimen tissue depicted in the specimen image. However, the problems that the embodiments disclosed in this specification and the drawings attempt to solve are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be positioned as other problems.

The information processing device according to the embodiment includes an acquisition unit, a tissue region recognition unit, a region of interest recognition unit, an excision position recognition unit, a correspondence unit, a mapping unit , and a display control unit . The acquisition unit acquires a first image, which is an image of an entire specimen, and a second image, which is an image of at least one specimen tissue excised from the specimen at a higher magnification than the first image. The tissue region recognition unit recognizes a tissue region in which the specimen tissue is depicted from the second image. The region of interest recognition unit recognizes a region of interest included in the tissue region. The excision position recognition unit recognizes an excision position of the specimen tissue in the first image. The correspondence unit associates the excision position in the first image with the tissue region in the second image. The mapping unit maps the region of interest in the second image with a target region corresponding to the region of interest in the first image based on the correspondence between the excision position and the tissue region. The display control unit associates the first image and the second image, in which the region of interest and the target region have been mapped by the mapping unit, and displays them as separate images on the same screen of the display unit, and associates the cut-out position on the first image with the display position on the second image of the specimen tissue cut out from the cut-out position.

FIG. 1 is a diagram illustrating an example of an overall configuration of an information processing system according to an embodiment. FIG. 2 is a diagram illustrating a sample image and a specimen image according to the embodiment. FIG. 3 is a diagram illustrating an example of a specimen image according to the embodiment. FIG. 4 is a diagram illustrating an example of a region of interest according to the embodiment. FIG. 5 is a diagram illustrating a process of recognizing a region of interest by deep learning according to an embodiment. FIG. 6 is a diagram illustrating an example of a cut-out position according to the embodiment. FIG. 7 is a diagram showing an example of association between extraction positions and tissue regions according to the embodiment. FIG. 8 is a diagram illustrating an example of a label depicted in a specimen image according to the embodiment. FIG. 9 is a diagram showing an example of a specimen region ID setting operation screen according to the embodiment. FIG. 10 is a diagram illustrating an example of mapping according to the embodiment. FIG. 11 is a diagram showing an example of an arrangement of centers of gravity of tissue regions in a specimen image according to the embodiment. FIG. 12 is a diagram showing another example of an arrangement of the centroids of tissue regions in a specimen image according to the embodiment. FIG. 13 is a diagram illustrating an example of a mapping display screen according to the embodiment. FIG. 14 is a diagram showing another example of the mapping display screen according to the embodiment. FIG. 15 is a diagram showing another example of the mapping display screen according to the embodiment. FIG. 16 is a diagram showing an example of a setting screen for the cut-out position according to the embodiment. FIG. 17 is a diagram illustrating an example of division of the cut-out position according to the embodiment. FIG. 18 is a diagram illustrating an example of an extraction position ID when the extraction line is divided according to the embodiment. FIG. 19 is a flowchart showing an example of the flow of the cut-out position setting process according to the embodiment. FIG. 20 is a flowchart showing an example of the flow of a mapping process between a sample image and a specimen image according to the embodiment. FIG. 21 is a diagram showing an example of a display mode of a target region according to the first modification. FIG. 22 is a diagram showing an example of a marker according to the second modification. FIG. 23 is a diagram showing an example of an operation for changing the length of the range bar according to the third modification. FIG. 24 is a diagram showing an example of an operation screen according to the third modification. FIG. 25 is a diagram showing an example of an operation screen according to the fourth modification. FIG. 26 is a diagram showing an example of the relationship between the extraction position and the tissue region according to the fifth modification. FIG. 27 is a diagram showing an example of a specimen image and a sample image according to the sixth modification. FIG. 28 is a diagram showing an example of mapping of depth information of a region of interest for a sample image according to Modification Example 7. FIG. 29 is a diagram showing an example of a cut-out position setting screen according to the ninth modification.

Below, embodiments of an information processing device and an information processing system will be described in detail with reference to the drawings.

FIG. 1 is a diagram showing an example of the overall configuration of an information processing system S according to an embodiment. As shown in FIG. 1, the information processing system S includes an information processing device 100, a sample image storage device 201, and a specimen image storage device 202. The information processing device 100 is communicatively connected to the sample image storage device 201 and the specimen image storage device 202 via a network 300 such as an in-hospital LAN (Local Area Network).

The information processing system S may further include a hospital information system (HIS), a laboratory information system (LIS), a radiology information system (RIS), etc. Alternatively, the information processing system S may be part of a hospital information system. The information processing system S may further include a terminal device such as a personal computer (PC) or a tablet terminal.

The information processing system S is provided, for example, in a medical institution such as a hospital, a research institute such as a university, or an examination center. In addition, some or all of the devices constituting the information processing system S may be provided in a cloud environment.

The specimen image storage device 201 is a device that stores specimen images. The specimen image is an image of the entire tissue piece that is the specimen. The specimen image is a digital image captured by an imaging device such as a digital camera. The specimen image is an example of a first image in this embodiment.

In this embodiment, the specimen is a portion of tissue collected from a patient's body, an animal, etc. For example, the specimen is the mucosa and submucosa of the digestive tract removed by endoscopic submucosal dissection (ESD), but is not limited thereto, and may be a piece of tissue removed by laparotomy, etc.

The specimen image storage device 202 is a device that stores specimen images. A specimen image is a captured image of a specimen tissue cut out from a tissue slice that is a specimen. More specifically, a specimen image is an image of at least one specimen tissue cut out from a tissue slice that is a specimen, captured at a higher magnification than the specimen image. For example, a specimen image is a whole slide imaging (WSI) in which the entire specimen tissue or a part of it placed on a glass slide is digitally imaged with high precision. The specimen image may also be a micrograph captured by an electron microscope, or an image captured by other techniques. The specimen image is an example of a second image in this embodiment.

The specimen image storage device 201 and the specimen image storage device 202 are, for example, a server device or a PC. The specimen image storage device 201 and the specimen image storage device 202 may be collectively referred to as an image storage device. Although the specimen image storage device 201 and the specimen image storage device 202 are shown as separate devices in FIG. 1, the specimen image storage device 201 and the specimen image storage device 202 may be configured as an integrated device. The specimen image storage device 201, the specimen image storage device 202, and the information processing device 100 may be configured as an integrated device.

Figure 2 is a diagram illustrating a sample image 51 and a specimen image 61 according to an embodiment. For example, a sample 5 collected from a patient P by a diagnostician is transferred to a pathologist or a laboratory technician for pathological diagnosis. At this time, a pathology number is assigned for each request (order) for pathological diagnosis. Also, a different specimen number is assigned for each specimen to be the subject of pathological diagnosis. Note that, when there are multiple specimens to be diagnosed in one request for pathological diagnosis, multiple specimen numbers are associated with one pathology number. Hereinafter, in this embodiment, a pathologist or a laboratory technician is referred to as a pathologist, etc.

The processing performed by the information processing device 100 in this embodiment does not necessarily have to be aimed at pathological diagnosis. For example, it may be aimed at research by a research institute, or at the creation of a report at an examination center that performs pathological examinations on behalf of a medical institution.

A specimen image ID is assigned as identification information for the specimen image 51 obtained by capturing an image of the specimen 5 using a digital camera or the like. The specimen image ID may be registered as additional information for the specimen image 51. Information related to the specimen image 51, such as the pathology number of the specimen 5 that is the subject of the image of the specimen image 51, the specimen number of the specimen 5, the patient ID of the patient P from whom the specimen 5 was obtained, and the image date and time of the specimen image 51, may be registered as additional information for the specimen image 51. The specimen image ID and other information related to the specimen image 51 may be displayed as text information on the specimen image 51.

The pathologist or the like cuts out the specimen tissues 6a-6e from the specimen 5, and after performing processing such as staining, places, for example, the cross-sections of the specimen tissues 6a-6e on the slide glass 7 facing the imaging device. Hereinafter, when there is no need to distinguish between the individual specimen tissues 6a-6e, they will simply be referred to as specimen tissue 6. The specimen tissue 6 is, for example, a thin slice of the specimen 5, but is not limited to this.

In FIG. 2, as an example, the specimen tissue 6 is imaged with its cross section facing the imaging device, but the orientation of the specimen is not limited to this. For example, the specimen tissue 6 may be imaged in the same orientation as the specimen 5 when the specimen image 51 is captured.

A WSI image or the like of the specimen tissue 6 placed on the glass slide 7 is the specimen image 61. A specimen image ID is assigned as identification information for the specimen image 61. Note that in FIG. 2, the specimen tissues 6a-6e cut from one specimen 5 are placed on one glass slide 7, but they may be placed separately on multiple glass slides 7. Since the specimen image 61 is imaged for each glass slide 7, if the specimen tissues 6a-6e are placed separately on multiple glass slides 7, multiple specimen images 61 are imaged. In this case, a different specimen image ID is assigned to each specimen image 61.

The specimen image ID may be registered as additional information of the specimen image 61. In addition, information related to the specimen image 61, such as the pathology number of the specimen 5 from which the specimen tissue that is the subject of the imaging of the specimen image 61 was obtained, the specimen number of the specimen 5, the patient ID of the patient P from which the specimen 5 was obtained, and the imaging date and time of the specimen image 61, may be registered as additional information of the specimen image 61. In addition, the specimen image ID and other information related to the specimen image 61 may be depicted as text information on the specimen image 61.

Returning to FIG. 1, the information processing device 100 is, for example, a server device or a PC, and has a network interface 110, a memory circuit 120, an input interface 130, a display 140, and a processing circuit 150.

The NW interface 110 is connected to the processing circuit 150 and controls the transmission and communication of various data between the information processing device 100 and the specimen image storage device 201 or the specimen image storage device 202. The NW interface 110 is realized by a network card, a network adapter, a NIC (Network Interface Controller), etc.

The memory circuitry 120 prestores various types of information to be used by the processing circuitry 150. The memory circuitry 120 also stores various programs.

The input interface 130 is realized by a trackball, a switch button, a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touch screen that integrates a display screen and a touchpad, a non-contact input circuit using an optical sensor, and a voice input circuit. The input interface 130 is connected to the processing circuit 150, and converts input operations received from an operator into electrical signals and outputs them to the processing circuit 150. Note that in this specification, the input interface is not limited to those that have physical operating parts such as a mouse and a keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs these electrical signals to the processing circuit 150 is also included as an example of the input interface 130.

The display 140 is a liquid crystal display, an organic electro-luminescence (OEL) display, or the like. The input interface 130 and the display 140 may be integrated. For example, the input interface 130 and the display 140 may be realized by a touch panel. The display 140 is an example of a display unit in this embodiment.

The processing circuit 150 is a processor that reads out a program from the memory circuit 120 and executes it to realize a function corresponding to each program. The processing circuit 150 of this embodiment includes an acquisition function 151, a reception function 152, a tissue area recognition function 153, an area of interest recognition function 154, a cut-out position recognition function 155, a matching function 156, a mapping function 157, a display control function 158, a transmission function 159, and a cut-out position setting function 160. The acquisition function 151 is an example of an acquisition unit. The reception function 152 is an example of a reception unit. The tissue area recognition function 153 is an example of a tissue area recognition unit. The area of interest recognition function 154 is an example of an area of interest recognition unit. The cut-out position recognition function 155 is an example of a cut-out position recognition unit. The matching function 156 is an example of a matching unit. The mapping function 157 is an example of a mapping unit. The display control function 158 is an example of a display control unit. The transmission function 159 is an example of a transmission unit. The cut-out position setting function 160 is an example of a cut-out position setting unit.

Here, for example, each processing function of the components of the processing circuit 150, namely, the acquisition function 151, the reception function 152, the tissue area recognition function 153, the area of interest recognition function 154, the cut-out position recognition function 155, the association function 156, the mapping function 157, the display control function 158, the transmission function 159, and the cut-out position setting function 160, is stored in the memory circuit 120 in the form of a program executable by a computer. The processing circuit 150 is a processor. For example, the processing circuit 150 realizes a function corresponding to each program by reading and executing the program from the memory circuit 120. In other words, the processing circuit 150 in the state in which each program has been read has each function shown in the processing circuit 150 of FIG. 1. In FIG. 1, the processing functions performed by the acquisition function 151, the reception function 152, the tissue area recognition function 153, the area of interest recognition function 154, the cut-out position recognition function 155, the association function 156, the mapping function 157, the display control function 158, the transmission function 159, and the cut-out position setting function 160 are described as being realized by a single processor, but the processing circuit 150 may be configured to be realized by combining multiple independent processors and each processor executing a program. Also, in FIG. 1, the single storage circuit 120 is described as storing a program corresponding to each processing function, but the processing circuit 150 may be configured to read out the corresponding program from each storage circuit by distributing multiple storage circuits.

In the above description, an example was described in which the "processor" reads out and executes a program corresponding to each function from a storage circuit, but the embodiment is not limited to this. The term "processor" refers to circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), and a Field Programmable Gate Array (FPGA)). When the processor is, for example, a CPU, the processor realizes the function by reading out and executing a program stored in a storage circuit. On the other hand, when the processor is an ASIC, instead of storing a program in the storage circuit 120, the function is directly incorporated as a logic circuit in the circuit of the processor. Note that each processor in this embodiment is not limited to being configured as a single circuit for each processor, and may be configured as a single processor by combining multiple independent circuits to realize the function. Additionally, multiple components in FIG. 1 may be integrated into a single processor to achieve the functions.

The acquisition function 151 acquires a sample image 51 from the sample image storage device 201. The acquisition function 151 also acquires a specimen image 61 from the specimen image storage device 202. Note that the source of acquisition of the sample image 51 and the specimen image 61 is not limited to the sample image storage device 201 and the specimen image storage device 202.

The specimen image 51 and the specimen image 61 to be acquired are specified, for example, by a user operation accepted by the reception function 152. The user of the information processing device 100 is a pathologist or the like.

The user specifies the sample image 51 or specimen image 61 to be acquired by, for example, inputting a pathology number, a patient ID, a specimen number, or an imaging date, or a combination of these. Furthermore, one pathology number is associated with one or more specimen numbers and specimen image IDs, and one specimen number is associated with one or more specimen image IDs. Therefore, when a user inputs a pathology number to process a request for a pathology diagnosis, one or more sample images 51 and one or more specimen images 61 are acquired.

The user may also select a desired specimen image 51 from a list of specimen images 51 stored in the specimen image storage device 201. The list may be a list in which information about the specimen image 51, such as the pathology number, is expressed as text information, or a list in which the specimen image 51 is displayed as an image. In this case, the acquisition function 151 acquires from the specimen image storage device 202 a specimen image 61 associated with the same specimen number as the specimen number of the specimen image 51 selected by the user.

The timing of acquiring the specimen image 51 or specimen image 61 is not limited to a user operation, but may be, for example, the timing when the specimen image 61 is saved in the specimen image storage device 202.

In this embodiment, the acquisition function 151 is described as acquiring the specimen image 51 and the specimen image 61, but the function of acquiring the specimen image 51 and the function of acquiring the specimen image 61 may be configured as separate functions.

The reception function 152 receives various operations by the user via the input interface 130. For example, the reception function 152 receives a cut-out position in the specimen image 51 specified by the user. The reception function 152 also receives a user operation to correct the correspondence between the cut-out position in the specimen image 51 and the tissue area in the specimen image 61. The cut-out position and tissue area will be described later.

The tissue area recognition function 153 recognizes the tissue area in which the specimen tissue 6 is depicted from the specimen image 61.

Figure 3 is a diagram showing an example of a specimen image 61 according to an embodiment. The diagram on the left side of Figure 3 is a specimen image 61 in which specimen tissues 6a to 6c are depicted. In the specimen image 61, the slide glass 7 reflected as the background and the specimen tissues 6a to 6c are depicted.

In the example shown in FIG. 3, the specimen tissues 6a to 6c are imaged with their cross sections facing the imaging device. When the specimen 5 is the mucosa and submucosa of the digestive tract, the top of the specimen image 61 is the mucosa side of the specimen tissues 6a to 6c, and the bottom of the specimen image 61 is the submucosa side of the specimen tissues 6a to 6c.

The tissue area recognition function 153 uses image processing to recognize tissue areas 62a-62c, which are areas in which specimen tissues 6a-6c are depicted in the specimen image 61. The diagram on the right side of Figure 3 shows the result of tissue area recognition function 153 recognizing tissue areas 62a-62c from the specimen image 61 shown in the diagram on the left side. The tissue area recognition function 153 recognizes tissue areas 62a-62c by identifying the coordinates of pixels contained in the areas in which specimen tissues 6a-6c are depicted in the specimen image 61. Hereinafter, when there is no need to distinguish between tissue areas 62a-62c, they will simply be referred to as tissue area 62.

In addition, in the diagram on the right side of FIG. 3, the contour of the tissue region 62 is displayed for the purpose of explanation, but the contour of the tissue region 62 does not have to be depicted. In addition, the user may be able to select whether to display or hide the contour of the tissue region 62.

The method for recognizing the tissue region 62 from the specimen image 61 is not particularly limited, but for example, the tissue region recognition function 153 may employ rule-based image processing such as Selective Search, which selects candidate regions by grouping similar regions at the pixel level.

Alternatively, image processing using machine learning such as deep learning may be employed as a method for recognizing the tissue region 62 from the specimen image 61. When deep learning is employed, the tissue region recognition function 153 inputs the specimen image 61 to a trained model stored in the memory circuitry 120, and obtains the recognition result of the tissue region 62 output from the trained model. The trained model used in the recognition process of the tissue region 62 is a model in which the correspondence between multiple specimen images 61 and the tissue regions 62 in the multiple specimen images 61 has been learned. The trained model may be incorporated into the tissue region recognition function 153.

Returning to FIG. 1, the region of interest recognition function 154 recognizes a region of interest contained in the tissue region 62 from the specimen image 61.

The region of interest is a specific tissue in the specimen tissue 6 that is considered important in pathological diagnosis. As an example, the region of interest is a lesion site in the specimen tissue 6. The lesion site is, for example, a tumor region such as an adenoma or adenocarcinoma.

The region of interest may also include any one of the following tissues: tumor, mucosa, muscularis mucosa, submucosa, or muscularis proper. The mucosa, muscularis mucosa, submucosa, or muscularis proper are normal regions that are free of tumors and are used for measuring tumors, etc. The region of interest may include any one of the following tissues, or may include multiple types of tissue.

Figure 4 is a diagram showing an example of a region of interest 63 according to an embodiment. The diagram on the left side of Figure 4 is a specimen image 61 in which specimen tissue 6 is depicted. A tissue region 62 is recognized in the specimen image 61.

The diagram on the right side of Figure 4 shows the result of region of interest recognition function 154 recognizing region of interest 63 from specimen image 61 shown in the diagram on the left side. As shown in the diagram on the right side of Figure 4, the image region in which region of interest 63 is depicted is recognized by identifying the coordinates of pixels contained in the image region in tissue region 62 in which region of interest 63 is depicted.

In addition, in the diagram on the right side of FIG. 4, the contour of the region of interest 63 is displayed for the purpose of explanation, but the contour of the region of interest 63 does not have to be drawn. In addition, the user may be able to select whether to display or hide the contour of the region of interest 63.

The method for recognizing the region of interest 63 from the specimen image 61 is not particularly limited, but rule-based image processing such as Selective Search may be used.

Alternatively, image processing using machine learning such as deep learning may be used as a method for recognizing the region of interest 63 from the specimen image 61.

FIG. 5 is a diagram illustrating the recognition process of the region of interest 63 by deep learning according to the embodiment. When deep learning is employed, the region of interest recognition function 154 inputs the specimen image 61 to the trained model 90 stored in the memory circuitry 120 as shown in FIG. 5, and obtains the recognition result of the region of interest 63 output from the trained model 90. The recognition result of the region of interest 63 is, for example, coordinate information indicating the range of the region of interest 63 in the specimen image 61.

The trained model 90 used in the recognition process of the region of interest 63 is, for example, a model that has learned the correspondence between multiple specimen images 61 and the regions of interest 63 in the multiple specimen images 61. The trained model 90 may be incorporated into the region of interest recognition function 154.

Returning to FIG. 1, the cut-out position recognition function 155 recognizes the cut-out position of the specimen tissue 6 in the specimen image 51. The cut-out position is the position where the specimen tissue 6 is cut out from the specimen 5. In this embodiment, the cut-out position simply refers to the cut-out position on the specimen 5 depicted in the specimen image 51.

Figure 6 is a diagram showing an example of an excision position according to an embodiment. When the specimen 5 is the mucosa and submucosa of the digestive tract, the specimen 5 is imaged with the mucosa facing the imaging device.

In the example shown in FIG. 6, cut-out lines 53a-53f drawn on the specimen image 51 are assigned cut-out position IDs "1"-"6", respectively. The cut-out lines 53a-53f indicate the Y-direction positions at which the specimen tissue 6 is cut out from the specimen 5. Hereinafter, when there is no need to distinguish between the individual cut-out lines 53a-53f, they will simply be referred to as cut-out lines 53.

The number of cutout lines 53 per sample image 51 is not particularly limited, but for example, in this embodiment, one or more cutout lines 53 are drawn per sample image 51.

The cut-out line 53 and cut-out position ID shown in the left diagram of FIG. 6 are set by the cut-out position setting function 160, which will be described later. Details of how to set the cut-out line 53 and the cut-out position ID will be described later.

For example, the cut-out position setting function 160 stores cut-out position information in the memory circuitry 120, which associates the sample image ID of the sample image 51, the coordinates of the cut-out lines 53 on the sample image 51, and the cut-out position ID. The cut-out position recognition function 155 reads out the cut-out position information from the memory circuitry 120 and identifies the positions of the cut-out lines 53 on the sample image 51 and the cut-out position IDs of each cut-out line 53. The cut-out position information may be stored in association with the sample image 51 as supplementary information of the sample image 51.

The cut-out position recognition function 155 also recognizes the specimen area 52 in the specimen image 51. The specimen area 52 is an image area in which the specimen 5 is depicted on the specimen image 51. The method for recognizing the specimen area 52 is not particularly limited, but rule-based image processing such as Selective Search may be adopted, or image processing using machine learning such as deep learning may be adopted. Furthermore, if the background image of the specimen image 51 is predetermined, the cut-out position recognition function 155 may delete the background image from the specimen image 51 and recognize the remaining image area as the specimen area 52.

The central diagram in Figure 6 is an example of a specimen region 52 recognized from a specimen image 51 by the cut-out position recognition function 155.

Furthermore, the cut-out position recognition function 155 corrects the cut-out line 53 based on the identified cut-out line 53 and cut-out position ID, and the recognized specimen region 52. Specifically, the cut-out position recognition function 155 aligns the width of the cut-out line 53 with the width of the specimen region 52 by deleting the portion of the cut-out line 53 that protrudes from the specimen region 52. Here, the width of the specimen region 52 and the width of the cut-out line 53 refer to the lengths of the specimen region 52 and the cut-out line 53 in the X direction in FIG. 6.

The corrected cut-out line 53 represents the cut-out position where the specimen tissue 6 is cut out from the specimen 5 depicted in the specimen image 51.

The method for recognizing the excision position is not limited to the above example. For example, the excision position recognition function 155 may recognize the excision position from a line image or text information drawn in the specimen image 51. The text information is, for example, information such as an instruction for the position to excision the tissue piece or a record of the excision position. The means for recognizing the text information may be, for example, OCR (Optical Character Recognition/Reader). Furthermore, the line image or text information is not limited to that drawn in the specimen image 51 itself, but may be written on a form or the like associated with the specimen image 51.

In addition, in FIG. 6, the excision position is recognized from the specimen image 51 obtained by capturing an image of the specimen 5 before the specimen tissue 6 is excised, but the excision position recognition function 155 may recognize the excision position from the specimen image 51 obtained by capturing an image of the specimen 5 after the specimen tissue 6 has been excised by image processing. In this case, the excision position recognition function 155 assigns an excision position ID to the excision position based on a predetermined operational rule.

The predetermined operation rule is, for example, "assigning cut-out position IDs in ascending order starting from 1 from the top of the specimen image 51". The predetermined operation rule is an operation rule that specifies the order in which the specimen tissues 6 are cut out and the order in which the specimen tissues 6 are arranged on the slide glass 7. The operation rule may be an operation rule specific to the medical institution or testing center in which the information processing device 100 is installed. In this case, the reception function 152 may receive the operation rule input by the user and store it in the memory circuitry 120 in advance. Alternatively, for example, a configuration may be adopted in which multiple operation rules are registered in the memory circuitry 120 before the user sets it, and the user can select one operation rule to adopt from the multiple operation rules.

Returning to FIG. 1, the matching function 156 matches the cut-out position in the specimen image 51 with the tissue area 62 in the specimen image 61 depicting the specimen tissue 6 cut out from the cut-out position.

Figure 7 is a diagram showing an example of the association between the cut-out position and the tissue region 62 according to this embodiment. As shown in Figure 7, the association function 156 associates the cut-out position recognized by the cut-out position recognition function 155 with the tissue region 62 cut out from the cut-out position on a one-to-one basis.

More specifically, the association function 156 assigns a specimen area ID to each tissue area 62 included in the specimen image 61. The association function 156 assigns specimen area IDs based on an operational rule such as, for example, "assign specimen area IDs in ascending order from 1 from the top of the specimen image 61." Furthermore, as shown in FIG. 7, when multiple specimen images 61a, 61b are associated with one specimen image 51, the association function 156 assigns specimen area IDs to the multiple specimen images 61a, 61b in the order of the earliest image capture date and time.

The operational rules used to assign specimen region IDs are the same as those described in the explanation of the excision position recognition function 155, and are operational rules that stipulate the order in which the specimen tissues 6 are excised and the order in which the specimen tissues 6 are placed on the slide glass 7.

For this reason, the specimen area ID assigned to each tissue area 62 in the specimen image 61 basically matches the cut-out position ID of the cut-out position where the specimen tissue depicted in the tissue area 62 was cut out. The association function 156 associates the cut-out position ID with the specimen area ID having the same number as the cut-out position ID. In the example shown in FIG. 7, the cut-out positions with cut-out position IDs "1" to "6" in the specimen image 51 are associated with the tissue areas 62 with specimen area IDs "1" to "6" in the specimen image 61, respectively.

The method of assigning the specimen area ID is not limited to the above example. FIG. 8 is a diagram showing an example of a label 71 depicted in a specimen image 61 according to this embodiment. For example, as shown in FIG. 8, if an excision position ID or a specimen area ID indicating the position from which the specimen tissue 6 was excised is written on the label 71 depicted in the specimen image 61, the association function 156 recognizes the excision position ID by a method such as OCR.

The user may also input a specimen area ID from a GUI (Graphical User Interface) displayed on the display 140. If the specimen tissue ID assigned by the association function 156 is incorrect, the user may also change the specimen area ID from the GUI. In this case, the reception function 152 accepts the input or change of the specimen area ID by the user. The association function 156 corrects the association between the excision position and the tissue area 62 based on the accepted user operation.

Figure 9 is a diagram showing an example of a specimen area ID setting operation screen 141 according to this embodiment. The setting operation screen 141 is a type of GUI. The display process of the setting operation screen 141 is performed by a display control function 158, which will be described later. The setting operation screen 141 includes a specimen image 51 and a specimen image 61 associated with the specimen image 51.

As shown in FIG. 9, a list box 60 or the like may allow the user to input or change the specimen area ID.

For example, when the user presses the number correction button 80 on the screen with a mouse or a stylus pen, the cut-out position ID and specimen area ID set by the association function 156 are displayed. Note that if the specimen area ID has not yet been set by the association function 156, the specimen area ID does not need to be displayed.

For example, when the user presses the number of the cut-out position ID or the cut-out line 53, the tissue region 62 for which a specimen region ID with the same number as the cut-out position ID can be set is displayed for selection. Selection methods include, for example, a drop box or radio buttons.

In addition, the tissue area 62 that the user clicks on after pressing the cut-out position ID number or the cut-out line 53 may be associated with the cut-out position ID.

The excision position ID and the tissue region 62 may be associated by the user dragging and dropping the number of the excision position ID or the excision line 53 onto the tissue region 62, or by the user dragging and dropping the tissue region 62 onto the number of the excision position ID or the excision line 53. The illustrated operations are merely examples, and the methods of associating the excision position in the specimen image 51 with the tissue region 62 in the specimen image 61 depicting the specimen tissue 6 excised from the excision position and the methods of changing the association are not limited to these.

In addition, when the number of tissue regions 62 is greater than the number of excision positions, such as when the specimen tissue 6 is divided after excision, the association function 156 may automatically assign a symbol such as an asterisk to the remaining tissue regions 62 as a provisional specimen region ID. In this case, the user may manually correct the specimen region ID. In addition, when the specimen tissue 6 is divided after excision, the user may divide the excision line 53 of the specimen image 51 to match the divided specimen tissue 6.

Returning to FIG. 1, the mapping function 157 maps the region of interest 63 in the specimen image 61 to a target region in the specimen image 51 that corresponds to the region of interest 63 in the specimen image 61 based on the correspondence between the cut-out position and the tissue region 62 by the correspondence function 156.

Figure 10 is a diagram showing an example of mapping according to this embodiment. The mapping function 157 associates the range of the region of interest 63 depicted in the specimen image 61 with the specimen image 61 based on the cut-out position ID and the specimen area ID. The mapping function 157 also draws a figure representing the target area 54 near the cut-out position associated with the tissue area 62 that includes the region of interest 63. The figure may be, for example, a line segment or a rectangle.

More specifically, the mapping function 157 draws a figure representing the target region 54 on the side of the cut-out line 53 indicating the cut-out position associated with the tissue region 62 including the region of interest 63, where the specimen tissue 6 is cut out. For example, the tissue region 62b of the specimen region ID "2" associated with the cut-out position ID "2" includes the region of interest 63a and the region of interest 63b. If the specimen tissue 6 depicted in the tissue region 62b is cut out from above the cut-out line 53b, the mapping function 157 draws the target region 54a corresponding to the region of interest 63a and the target region 54b corresponding to the region of interest 63b above the cut-out line 53b to which the cut-out position ID "2" is assigned in the specimen image 51.

Tissue region 62b with specimen region ID "3" associated with cut-out position ID "3" includes region of interest 63c. Mapping function 157 draws target region 54c corresponding to region of interest 63c above cut-out line 53c with cut-out position ID "3" in specimen image 51. Hereinafter, when there is no need to distinguish between individual target regions 54a to 54c, they will simply be referred to as target region 54.

In FIG. 10, the target area 54 is drawn linearly, but the drawing manner is not limited to this.

Whether the target region 54 is mapped above or below the cut-out line 53 depends on the orientation of the cut surface when the specimen tissue 6 is placed on the slide glass 7. The mapping function 157 determines whether the target region 54 is mapped above or below the cut-out line 53, for example, based on an operational rule. Alternatively, the user may be able to select whether the target region 54 is mapped above or below the cut-out line 53. The mapping function 157 may also draw the target region 54 superimposed on the cut-out line 53.

In addition, since the specimen image 61 is captured at a higher magnification than the specimen image 51, when the mapping function 157 draws the target area 54 corresponding to the region of interest 63 in the specimen image 51, the scale of the region of interest 63 is corrected to match the scale of the specimen image 51.

Specifically, the mapping function 157 corrects the dimensions of the region of interest 63 based on the difference between the dimensions of the specimen region 52 at the cut-out position of the specimen image 51 and the dimensions of the tissue region 62 corresponding to the cut-out position. The mapping function 157 maps the region of interest 63 with the corrected dimensions onto the target region 54 on the specimen image 51. More specifically, the dimensions of the specimen region 52 are the length of the specimen region 52 in the width direction of the specimen image 51, that is, the width of the specimen region 52. More specifically, the dimensions of the tissue region 62 are the length of the tissue region 62 in the same direction as the width direction of the specimen region 52.

The mapping function 157 calculates the dimensions and position of the corrected target area 54, for example, using equation (1).

In formula (1), i is the cut-out position ID of the specimen image 51 and the specimen area ID of the specimen image 61. R _macro,i indicates the range of the target area 54 associated with the cut-out position ID "i". _{R micro,i} is the range of the region of interest 63 associated with the specimen area ID "i" on the specimen image 61. The range of the target area 54 and the range of the region of interest 63 are expressed by the coordinates on the specimen image 51 or the specimen image 61. l _macro,i is the length (width) in the X direction of the corrected cut-out line 53 associated with the cut-out position ID "i" on the specimen image 51. Also, l _micro,i is the length (width) in the X direction of the tissue area 62 on the specimen image 61. Also, offset is the amount of movement for drawing the corrected tissue area 62 as the target area 54 on the specimen image 51. For example, the offset is the coordinate of the left end of the corrected cut-out line 53 associated with the cut-out position ID "i".

In addition, if the magnification of the microscope that captured the specimen image 61 and the magnification of the digital camera that captured the specimen image 51 can be identified, the mapping function 157 may correct the length (width) of the cutout line 53 in the X direction based on the difference in magnification.

In addition, since the tissue regions 62 on the specimen image 61 are not necessarily aligned in the same direction as the cut-out line 53 of the specimen image 51, the mapping function 157 identifies the direction of the tissue regions 62 that corresponds to the X direction (width direction) of the specimen image 51 based on the arrangement of the centers of gravity of the tissue regions 62.

Figure 11 is a diagram showing an example of an arrangement of centers of gravity of tissue regions in a specimen image according to an embodiment. Centers of gravity 65a to 65c are the centers of distribution of the pixels contained in tissue regions 62a to 62c, respectively. For example, the coordinates of centers of gravity 65a to 65c are the average values of the coordinates of the pixels contained in tissue regions 62a to 62c, respectively.

The mapping function 157 determines the direction in which the centers of gravity 65a to 65c are aligned, and recognizes the direction in which the centers of gravity 65a to 65c are aligned as the Y-axis direction, and the direction perpendicular to the direction in which the centers of gravity 65a to 65c are aligned as the X-axis direction (width direction). The mapping function 157 determines the length (width) L1 in the X-direction of the tissue region 62 along the recognized X-axis direction. The X-axis direction (width direction) is also called the reference axis direction.

In FIG. 11, the direction in which the tissue regions 62 are arranged in the specimen image 61 is substantially the same as the cut-out line 53 in the sample image 51, but the direction in which the tissue regions 62 are arranged is not limited to this. FIG. 12 is a diagram showing another example of the arrangement of the centroids of the tissue regions 62 in a specimen image according to an embodiment. As shown in FIG. 12, the tissue regions 62 may be arranged in the vertical direction of the specimen image 61. In this case, the mapping function 157 determines that the vertical direction of the specimen image 61 is the width direction of the tissue regions 62 based on the direction in which the centroids 65a to 65c are arranged.

The direction in which the centers of gravity 65a to 65c are arranged may be determined by, for example, calculating the ratio of the variance of the X-coordinate and the Y-coordinate in the coordinate group, or by principal component analysis. Specifically, the mapping function 157 may perform principal component analysis on the coordinates of the centers of gravity 65a to 65c, and may set the direction perpendicular to the first principal component direction as the reference axis direction. If the ratio of the dimensions of the first principal component to the second principal component is greater than a threshold, the mapping function 157 may notify the display control function 158 described below that the direction in which the centers of gravity 65a to 65c are arranged cannot be determined. In this case, the display control function 158 may display a screen such as a GUI that allows the user to input the reference axis direction. The threshold for the ratio of the dimensions of the first principal component to the second principal component is, for example, 0.8, but is not limited to this.

The mapping function 157 may also determine the length (width) L1 in the X-direction of the tissue region 62 according to rules such as "if the centers of gravity 65a-65c are aligned vertically in the specimen image 61, the measurement of the width of the tissue region 62 is based on the horizontal direction of the specimen image 61" and "if the centers of gravity 65a-65c are aligned horizontally in the specimen image 61, the measurement of the width of the tissue region 62 is based on the vertical direction of the specimen image 61." In this case, if the centers of gravity 65a-65c are aligned diagonally or if the centers of gravity 65a-65c are aligned without a fixed directionality, the user is to input the reference axis direction.

In this embodiment, the mapping function 157 draws the target area 54, but the mapping function 157 may simply identify the coordinates of the target area 54 on the sample image 51.

Returning to FIG. 1, the display control function 158 causes the display 140 to display various screens such as a GUI. The display control function 158 also causes the display 140 to display the specimen image 51 and the sample image 61, in which the region of interest 63 and the target region 54 have been mapped by the mapping function 157, in association with each other.

Figure 13 is a diagram showing an example of a mapping display screen 142 according to this embodiment. For example, as shown in Figure 13, the display control function 158 displays the sample image 51 and sample images 61a and 61b, which are images of sample tissue 6 cut out from the sample 5 depicted in the sample image 51, on one screen as the mapping display screen 142. The display control function 158 also displays the target area 54 mapped on the sample image 51 by the mapping function 157 as, for example, a linear figure. The display form of the target area 54 and the form of the mapping display screen 142 are not limited to this.

FIG. 14 is a diagram showing another example of the mapping display screen 142 according to this embodiment. The image 610 shown in FIG. 14 is an image obtained by cutting out a portion of the specimen image 61 in which a tissue region 62 corresponding to the cut-out position indicated by the cut-out line 53 on the specimen image 51 selected by the user is depicted. For example, the display control function 158 may display the tissue region 62 corresponding to each cut-out position on the specimen image 51 as shown in FIG. 14.

Also, FIG. 15 is a diagram showing another example of the mapping display screen 142 according to this embodiment. The specimen image 51 and the specimen image 61 do not have to be displayed at the same size. For example, when the user selects the specimen image 61, the display control function 158 displays the specimen image 61 in an enlarged manner as shown in FIG. 15, and the specimen image 51 is displayed smaller than the specimen image 61. Also, when the user selects the specimen image 51, the display control function 158 displays the specimen image 51 in an enlarged manner, and the specimen image 61 is displayed smaller than the specimen image 51.

The mapping display screen 142 may also serve as an operation screen that can accept various change operations by the user, such as changing the mapping or changing the cut-out line 53.

The user can also perform an operation to save or output the results of the changes made on the operation screen. For example, when the user presses the save button 81 shown in FIG. 15, information such as mapping and cutout line 53 is stored in the memory circuitry 120 in the state after the user's changes, in association with the specimen image 51 and the sample image 61.

When the user presses the output button 82 shown in FIG. 15, the sample image 51 and specimen image 61 after mapping are output. The output destination may be any storage destination in the memory circuit 120, or may be another information processing device connected to the information processing device 100 via the network 300. The other information processing device may be, for example, an information processing device constituting an HIS, LIS, RIS, etc., but is not limited to these. When the output destination is outside the information processing device 100, the sample image 51 and specimen image 61 after mapping designated as the output target by the user are transmitted to the output destination by the transmission function 159 described below.

In addition, in this embodiment, the display control function 158 displays the mapping display screen 142 or other GUI on the display 140 of the information processing device 100, but the mapping display screen 142 or other GUI may also be displayed on a display of a terminal device such as a PC or tablet terminal connected to the information processing device 100 via the network 300. In this case, the display of the terminal device such as a PC or tablet terminal is an example of a display unit.

Returning to FIG. 1, the transmission function 159 transmits to an external device the sample image 51 on which the target region 54 has been mapped by the mapping function 157 as corresponding to the region of interest 63.

The sample image 51 on which the target area 54 corresponding to the region of interest 63 has been mapped by the mapping function 157 is an image in which a figure indicating the target area 54 is drawn on the sample image 51. In addition, the sample image 51 on which the target area 54 corresponding to the region of interest 63 has been mapped by the mapping function 157 may be image data in which the position and dimensions of the target area 54 are registered as supplementary information of the sample image 51.

In addition, at this time, the transmission function 159 may also transmit the specimen image 61 associated with the specimen image 51 to the external device together with the specimen image 51. In this case, it is assumed that the specimen image 51 and the specimen image 61 are associated with cut-out position information and mapping information.

The external device to which the data is to be sent is, for example, another information processing device connected to the information processing device 100 via the network 300. The other information processing device is, for example, an information processing device constituting an HIS, LIS, RIS, etc., but is not limited to these. The sending function 159 may automatically send the specimen image 51 and the specimen image 61 in which the region of interest 63 and the target region 54 have been mapped by the mapping function 157, or may send the specimen image 51 or the specimen image 61 specified by the user.

The cut-out position setting function 160 sets the cut-out position in the sample image 51 based on the cut-out position specified by the user. The cut-out position specified by the user is accepted by the reception function 152.

Figure 16 is a diagram showing an example of the cut-out position setting screen 143 according to this embodiment. The cut-out position setting screen 143 is displayed on the display 140 by the display control function 158. The cut-out position setting screen 143 displays the specimen image 51 before the specimen tissue 6 is cut out.

As shown in FIG. 16, when a user draws an excision line 53 on the sample image 51 displayed on the excision position setting screen 143 by operating a mouse or the like, the excision position setting function 160 records the coordinates of the excision line 53 on the sample image 51 as the excision position. In addition, the excision position setting function 160 assigns an excision position ID to each excision line 53 based on the operation rules. For example, the excision position setting function 160 assigns excision position IDs in ascending order from the top of the sample image 51.

The cut-out line 53 does not have to be parallel to the width direction of the specimen image 51, and can be set at any angle by the user. In addition, the cut-out position setting function 160 may recognize the upper side of the cut-out line 53 as the direction in which the specimen tissue 6 is collected when the angle θ of the cut-out line 53 with respect to the width direction of the specimen image 51 is less than a threshold value, and may recognize the lower side of the cut-out line 53 as the direction in which the specimen tissue 6 is collected when the angle θ is equal to or greater than the threshold value. The threshold value of the angle θ is, for example, 270 degrees, but is not limited to this. In FIG. 16, the line parallel to the width direction of the specimen image 51, which is drawn to indicate the start position of the angle θ, may not actually be displayed on the cut-out position setting screen 143.

The cut-out position setting function 160 may also automatically set cut-out lines 53 at regular intervals according to predetermined guidelines.

The sample image 51 with the cutout line 53 set is stored in the memory circuit 120, for example, by a user operation. The sample image 51 with the cutout line 53 set is the sample image 51 associated with cutout position information. The cutout position information is information in which the sample image ID of the sample image 51, the coordinates of the cutout line 53 on the sample image 51, and the cutout position ID are associated with each other. The cutout position information may be registered in one image data as additional information of the sample image 51, or may be stored as a separate data file. In addition, the storage destination of the sample image 51 with the cutout line 53 set is not limited to the memory circuit 120, and may be the sample image storage device 201. The cutout position information is also called annotation information (correct answer information).

Based on the specimen image 51 with the cut-out line 53 set, the pathologist or the like cuts out the specimen tissue 6 from the specimen 5, performs processing such as staining on the specimen tissue 6, and then captures the specimen image 61.

The cut-out position setting function 160 may also accept operations such as dividing the cut-out position by the user when correcting the correspondence between the cut-out position of the specimen image 51 and the tissue area 62 of the specimen image 61 not only before the specimen tissue 6 is cut out from the specimen 5 but also after the cut-out.

Figure 17 is a diagram showing an example of the division of the cut-out position according to this embodiment. For example, when the user presses the division button 86 on the operation screen 146 displayed on the display 140 by the display control function 158, a marker 97 indicating the division position of the cut-out line 53 is displayed on the sample image 51. The user can set the division position by moving the marker 97 by operating the mouse or the like. When the cut-out line 53 is divided, the cut-out position ID is represented by a sub-number for each divided cut-out line.

Figure 18 is a diagram showing an example of the cut-out position ID when the cut-out line 53 according to this embodiment is divided. In the example shown in Figure 18, the third cut-out line 53 is divided into cut-out line 531 and cut-out line 532. In this case, the cut-out position setting function 160 sets the cut-out position ID "3-1" for the cut-out line 531 and the cut-out position ID "3-2" for the cut-out line 532. In this case, the tissue region IDs of the tissue region 62c and tissue region 62d corresponding to the cut-out line 531 and the cut-out line 532 are also changed to "3-1" and "3-2" by the above-mentioned association function 156 or the user's operation. The cut-out line 53 may be divided in advance before the specimen image 61 is captured.

In this embodiment, the cut-out position setting function 160 has been described as an example of a function of the information processing device 100, but the cut-out position setting function 160 may be provided in a device other than the information processing device 100.

Next, we will explain the flow of the cut-out position setting process executed by the information processing device 100 configured as described above.

Figure 19 is a flowchart showing an example of the flow of the cut-out position setting process according to this embodiment. The process of this flowchart is premised on the assumption that the sample image 51 has been captured by a digital camera or the like and stored in the sample image storage device 201.

First, the reception function 152 receives an operation by the user to specify a sample image 51 (S1). The user specifies the sample image 51 to be acquired by inputting, for example, a pathology number, a patient ID, a sample number, or an imaging date, or a combination of these. The user may also select a desired sample image 51 from a list of sample images 51 stored in the sample image storage device 201.

Then, the acquisition function 151 acquires the sample image 51 specified by the user from the sample image storage device 201 (S2). Here, the display control function 158 causes the display 140 to display the cut-out position setting screen 143 including the sample image 51.

The reception function 152 also receives an operation by the user to specify an excision position in the sample image 51 (S3). The operation by the user to specify an excision position in the sample image 51 is, for example, an operation to draw an excision line 53 on the sample image 51 on the excision position setting screen 143.

The cut-out position setting function 160 assigns a cut-out position ID to the cut-out position specified by the user, for example, based on operational rules (S4).

The reception function 152 also determines whether or not a user operation to change the cut-out position has been received (S5).

When the reception function 152 determines that a user operation to change the cut-out position has been received (S5 "Yes"), the cut-out position setting function 160 changes the cut-out position based on the change operation (S6). In addition to changing the cut-out position, when a user operation to divide the cut-out line is received, the cut-out position setting function 160 also divides the cut-out line based on the user operation. Then, the process returns to S5.

If the reception function 152 determines that the user's operation to change the cut-out position has not been accepted (S5 "No"), it determines whether the user's operation to save the cut-out position has been accepted (S7). If the reception function 152 determines that the user's operation to save the cut-out position has not been accepted (S7 "No"), it returns to the processing of S5.

If the reception function 152 determines that the user's operation to save the cut-out position has been received (S7 "Yes"), the cut-out position setting function 160 stores the sample image 51 with the cut-out position, i.e., the sample image 51 associated with the cut-out position information, in the memory circuit 120 (S8). The cut-out position setting function 160 may transmit the sample image 51 with the cut-out position to the sample image storage device 201 via the transmission function 159. At this point, the processing of this flowchart ends.

Next, we will explain the flow of the mapping process for the specimen image 51 and the sample image 61 according to this embodiment.

Figure 20 is a flowchart showing an example of the flow of the mapping process of the sample image 51 and the specimen image 61 according to this embodiment. As a premise for the processing of this flowchart, it is assumed that the sample image 51 and the specimen image 61 have been captured and stored in the sample image storage device 201 and the specimen image storage device 202. In addition, if the cut-out position setting function 160 of the information processing device 100 sets the cut-out position of the sample image 51, the sample image 51 may be stored in the memory circuit 120 of the information processing device 100.

First, the reception function 152 receives an operation by the user to specify a sample image 51 and a specimen image 61 (S101). For example, the user specifies the sample image 51 and the specimen image 61 to be acquired by inputting, for example, a pathology number, a patient ID, a specimen number, or an imaging date, or a combination of these. The user may also select a desired sample image 51 from a list of sample images 51 stored in the sample image storage device 201. In this case, a specimen image 61 associated with the same specimen number as the selected sample image 51 is automatically selected.

Then, the acquisition function 151 acquires the sample image 51 designated as the acquisition target from the sample image storage device 201 (S102). Note that, if the sample image 51 is stored in the memory circuitry 120, the acquisition function 151 acquires the sample image 51 designated as the acquisition target from the memory circuitry 120.

The acquisition function 151 also acquires the specimen image 61 specified as the acquisition target from the specimen image storage device 202 (S103).

Next, the tissue area recognition function 153 recognizes the tissue area 62 from the acquired specimen image 61 by deep learning or the like (S104).

In addition, the region of interest recognition function 154 recognizes a region of interest 63 from the acquired specimen image 61 by deep learning or the like (S105).

Next, the cut-out position recognition function 155 recognizes the cut-out position from the acquired sample image 51 (S106). For example, if the cut-out line 53 and the cut-out position ID of the sample image 51 are set by the cut-out position setting function 160, the cut-out position recognition function 155 identifies the cut-out line 53 and the cut-out position ID from the cut-out position information associated with the sample image 51. Also, if the cut-out line 53 and the cut-out position ID of the sample image 51 are not set by the cut-out position setting function 160, for example, the cut-out position recognition function 155 identifies the cut-out line 53 and the cut-out position ID based on OCR or a predetermined operation rule. Then, the cut-out position recognition function 155 recognizes the sample region 52 in the sample image 51, and recognizes the cut-out position in the sample image 51 by correcting the cut-out line 53 based on the identified cut-out line 53 and the cut-out position ID and the recognized sample region 52.

Next, the association function 156 associates the tissue region 62 of the specimen image 61 with the cut-out position of the specimen image 51 (S107). For example, the association function 156 sets a tissue region ID to the tissue region 62 based on the operation rules. The association function 156 associates the tissue region ID with the cut-out position ID of the cut-out position in a one-to-one relationship.

Next, the mapping function 157 maps the region of interest 63 that includes the tissue region 62 of the specimen image 61 onto the specimen image 51 (S108). For example, the mapping function 157 draws a figure indicating the target region 54 in the target region 54, which is an image range on the specimen image 51 that corresponds to the region of interest 63. Here, the display control function 158 causes the display 140 to display, for example, the mapping display screen 142 described in FIG. 13.

Then, the reception function 152 judges whether or not an operation to change the association by the user has been accepted (S109). When the reception function 152 judges that an operation to change the association by the user has been accepted (S109 "Yes"), the reception function 152 changes the association by changing the tissue area ID or the cut-out position ID based on the user's operation (S110). Then, the process returns to S109.

If the reception function 152 determines that a user-initiated operation to change the association has not been accepted (S109 "No"), it determines whether a user-initiated save operation has been accepted (S111). If the reception function 152 determines that a user-initiated operation to change the association has not been accepted (S111 "No"), it returns to the process of S109.

When the reception function 152 determines that a save operation by the user has been received (S111 "Yes"), the mapping function 157 associates the mapped sample image 51 and specimen image 61 and saves them in the memory circuitry 120 (S112).

In this way, the information processing device 100 of this embodiment corresponds the cut-out position in the specimen image 51 to the tissue region 62 in the specimen image 61 in which the specimen tissue 6 cut out from the cut-out position is depicted, and maps the region of interest 63 in the specimen image 61 to the target region 54 in the specimen image 51 that corresponds to the region of interest 63 in the specimen image 61. Therefore, according to the information processing device 100 of this embodiment, it is possible for a doctor or the like to easily grasp the correspondence between the cut-out position of the specimen tissue 6 in the specimen 5 depicted in the specimen image 51 and the region of interest 63, such as a lesion region, on the specimen tissue 6 depicted in the specimen image 61.

For example, in the past, when a pathologist or the like prepared a pathology diagnosis report, the pathologist or the like would enlarge and visually inspect a portion of the tissue of the specimen using a microscope or the like, visually grasp the area of the specimen image corresponding to the region of interest contained in the portion of the tissue, and manually write the grasped result on the specimen image. In such a task, it may be difficult for the pathologist or the like to grasp the correspondence between the region of interest and the specimen image with high accuracy. In addition, since the pathologist or the like must alternate between checking the microscope and the specimen image to transcribe the region of interest, the workload may be high. In contrast, the information processing device 100 of this embodiment maps the target region 54 corresponding to the region of interest 63 of the specimen image 61 onto the specimen image 51, enabling the pathologist or the like to grasp the correspondence between the region of interest and the specimen image with high accuracy.

The information processing device 100 of this embodiment also displays the specimen image 51 and the specimen image 61, in which the region of interest 63 and the target region 54 are mapped, in correspondence with each other on the display 140. Therefore, according to the information processing device 100 of this embodiment, a doctor or the like can visually grasp the correspondence between the region of interest 63 depicted in the specimen image 61, such as a lesion region on the specimen tissue 6.

The information processing device 100 of this embodiment also accepts a user operation to correct the correspondence between the cut-out position in the specimen image 51 and the tissue region 62 in the specimen image 61, and corrects the correspondence between the cut-out position and the tissue region 62 based on the accepted user operation. Therefore, according to the information processing device 100 of this embodiment, even if the position at which the specimen tissue 6 is placed on the slide glass 7 does not follow the operational rules, the correspondence between the cut-out position and the tissue region 62 can be flexibly corrected by the user's operation.

The information processing device 100 of this embodiment also accepts a cut-out position in the sample image 51 designated by the user, and sets the cut-out position in the sample image 51 based on the accepted cut-out position. Therefore, according to the information processing device 100 of this embodiment, it is possible to set the cut-out position at a position desired by the user, and it is also possible to accurately identify the cut-out position when associating the cut-out position with the tissue region 62.

In addition, the information processing device 100 of this embodiment corrects the dimensions of the region of interest 63 based on the difference between the dimensions of the specimen region 52 at the cut-out position of the specimen image 51 and the dimensions of the tissue region 62, and maps the region of interest 63 with the corrected dimensions to the target region 54 on the specimen image 51. Therefore, according to the information processing device 100 of this embodiment, the difference in scale between the specimen image 61 and the specimen image 51 can be corrected, and the position corresponding to the region of interest 63 on the specimen image 51 can be represented as the target region 54, so that a doctor or the like can grasp the position and range of the region of interest 63 in the specimen 5 with high accuracy.

In addition, the information processing device 100 of this embodiment draws a figure representing the target region 54 near the cut-out position associated with the tissue region 62 including the region of interest 63. Therefore, according to the information processing device 100 of this embodiment, a doctor or the like can visually grasp the position and range in the specimen 5 of the region of interest 63 recognized in the specimen image 61.

Furthermore, the information processing device 100 of this embodiment draws a figure representing the target region 54 on the side of the cut-out line 53 indicating the cut-out position associated with the tissue region 62 including the region of interest 63, where the specimen tissue 6 is cut out. Therefore, according to the information processing device 100 of this embodiment, a doctor or the like can easily grasp on which side of the cut-out line 53 the region of interest 63 recognized from the specimen image 61 is located.

In addition, the information processing device 100 of this embodiment inputs the specimen image 61 to a trained model 90 that has learned the correspondence between multiple specimen images and image regions in which lesion sites are depicted in the multiple specimen images, and recognizes the image region output from the trained model 90 as the image region in which the region of interest 63 is depicted. Therefore, the information processing device 100 of this embodiment can reduce the work of a doctor or the like manually writing the range of the region of interest on the specimen image, and can efficiently identify the region of interest 63.

The information processing device 100 of this embodiment also transmits the sample image 51 onto which the target region 54 corresponding to the region of interest 63 is mapped to an external device. Therefore, according to the information processing device 100 of this embodiment, the mapping result of the target region 54 in the sample image 51 can be registered in an external device such as an HIS.

The trained model 90 described in this embodiment includes a "self-learning model" that further updates the internal algorithm of the trained model 90 by obtaining user feedback. In addition to the trained model 90, the other trained models described in this embodiment also include "self-learning models". The trained model 90 and other trained models described in this embodiment may be constructed using integrated circuits such as ASICs and FPGAs. Instead of the trained model 90 and other trained models described in this embodiment, a mathematical model, a lookup table, a database, or the like may be applied.

The processing described in this embodiment as being executed by the mapping function 157 may be executed by the display control function 158. For example, instead of generating and saving in advance an image in which a figure representing the target area 54 is drawn, the display control function 158 may temporarily display the figure representing the target area 54 when displaying it on the display 140.

(Variation 1)
In the above embodiment, the mapping function 157 draws, on the sample image 51, a figure representing the target region 54 of the sample image 51 corresponding to the region of interest 63 of the specimen image 61. In this modified example, when the region of interest 63 includes multiple types of tissue, the mapping function 157 draws the target region 54 on the sample image 51 in a different display mode for each type of tissue.

For example, as described in the above embodiment, the region of interest 63 may include any one of the following tissues: tumor, mucosa, muscularis mucosa, submucosa, or muscularis propria, or may include multiple types of tissue.

Figure 21 is a diagram showing an example of the display mode of the target region 54 according to the first modification. As shown in the upper part of Figure 21, the tissue region 62 of the specimen image 61 includes three regions of interest 63g, 63x, and 63h. Of these, the regions of interest 63g and 63h are benign tumors, and the region of interest 63x is a malignant tumor. In other words, the regions of interest 63g and 63h are the same type of tissue, but the region of interest 63x is a different type of tissue from the regions of interest 63g and 63h.

The region of interest recognition function 154 recognizes the tissue type of each of the regions of interest 63g, 63x, and 63h by rule-based image processing such as Selective Search or deep learning. For example, the region of interest recognition function 154 recognizes that the regions of interest 63g and 63h are benign tumors and that the region of interest 63x is a malignant tumor.

The mapping function 157 determines whether the region of interest 63 includes multiple types of tissues based on the recognition result of the tissue type of the region of interest 63 by the region of interest recognition function 154. If the region of interest 63 includes multiple types of tissues, the mapping function 157 draws a figure representing the target region 54 on the specimen image 51 in a different display mode for each type of tissue.

For example, in display examples 1 to 3 shown in FIG. 21, the mapping function 157 depicts the target regions 54g, 54h corresponding to the regions of interest 63g, 63h, and the target region 54x corresponding to the region of interest 63x in different display modes. For example, the mapping function 157 displays the target regions 54g, 54h and the target region 54x with line segments of different colors. Note that the difference in display mode is not limited to this.

In addition, in display example 1, the mapping function 157 displays the target region 54x superimposed on the target regions 54g and 54h. Because malignant tumors have a higher priority in pathological diagnosis than benign tumors, the mapping function 157 prevents the target region 54x from being hidden by the target regions 54g and 54h by drawing it in this manner. Note that the priority at the time of drawing may be determined by the region of interest recognition function 154. Furthermore, the priority at the time of drawing may be changed by an operation by the user to change the priority.

In addition, in display example 2, the mapping function 157 shifts the positions of the line segments representing the target regions 54x, 54g, and 54h. As a result, the line segments corresponding to all of the target regions 54x, 54g, and 54h are displayed on the sample image 51.

In addition, in display example 3, the mapping function 157 changes and draws the line segment representing the target area 54x and the line segments representing the target areas 54g and 54h. As a result, the line segments corresponding to all of the target areas 54x, 54g, and 54h are displayed on the sample image 51.

Which of the display examples 1 to 3 in which the figure representing the target area 54 on the sample image 51 is drawn may be selectable by the user, for example, or may be predetermined.

(Variation 2)
In the above embodiment, the mapping function 157 maps the target region 54 corresponding to the region of interest 63 onto the sample image 51, but the target of mapping is not limited to this.

Fig. 22 is a diagram showing an example of markers 66, 56 according to Modification 2. In this modification, the reception function 152 receives a user operation to input a marker 66 on a portion of the specimen image 61 that requires attention, for example, as shown in the upper part of Fig. 22.

The mapping function 157 identifies a position on the specimen image 51 that corresponds to the marker 66 drawn on the specimen image 61, and draws the marker 56 at the identified position. Through this process, the mapping function 157 maps the marker 66 drawn on the specimen image 61 onto the specimen image 51.

The position on the specimen image 51 corresponding to the marker 66 written on the specimen image 61 is, for example, coordinates on the specimen image 51 corresponding to the representative coordinates of the pixel where the marker 66 is drawn on the specimen image 61. The representative coordinates are coordinates indicating any one of the coordinates of the multiple pixels where the marker 66 is drawn on the specimen image 61. For example, as shown in FIG. 22, if the shape of the marker 66 is triangular, the coordinates of the tip of the triangle are the representative coordinates. Also, if the shape of the marker 66 is circular, the coordinates of the center of the circle are the representative coordinates. Also, as explained in equation (1), the offset may be based on the coordinates of the left end of the corrected cut-out line 53.

According to the information processing device 100 of this modified example, a doctor or the like can easily grasp which position on the specimen image 51 corresponds to the area requiring attention on the specimen image 61.

In this modified example, the marker 56 is input by the user, but the marker 56 may be set automatically. For example, the region of interest recognition function 154 may set the marker 56 in a region that is recognized as requiring attention by rule-based image processing such as Selective Search or deep learning. A region requiring attention is, for example, a region where a malignant tumor has reached a blood vessel, but is not limited to this. Furthermore, the location indicated by the marker 56 may be inside or outside the region of interest 63.

(Variation 3)
In the above embodiment, it has been described that the information processing device 100 can accept a user's operation to change the cropping position, etc. Furthermore, the information processing device 100 may be capable of accepting a user's operation to modify the dimensions of the region of interest 63.

Figure 23 is a diagram showing an example of an operation for changing the length of range bars 91 to 93 according to Modification Example 3. Range bars 91 to 93 indicate the width and widthwise position of a region of interest 63 included in a tissue region 62. Note that in Figure 23, the contour of the region of interest 63 is not displayed, but the contour of the region of interest 63 may be displayed in addition to the range bars 91 to 93.

The reception function 152 receives an operation to change the length and position of the range bars 91 to 93, for example, by the user dragging and dropping the end of the range bars 91 to 93 with the mouse pointer 9 or the like. The operation to change the length and position of the range bars 91 to 93 is an example of a user operation to correct the dimensions of the region of interest 63.

The mapping function 157 changes the dimensions of the target region 54 on the specimen image 51 based on the dimensions of the region of interest 63 corrected by the user.

FIG. 24 is a diagram showing an example of an operation screen 144 relating to variant example 3. The operation screen 144 is displayed by the display control function 158. The operation screen 144 includes, for example, a specimen image 51 and a specimen image 61. Also, for example, when the reception function 152 receives a user operation of pressing the region of interest correction button 83, the display control function 158 displays range bars 94 and 95 on the operation screen 144.

The region of interest 63 associated with the range bar 94 shown in FIG. 24 corresponds to the target region 54a on the sample image 51. In this case, when the length of the range bar 94 is changed by a user operation, the mapping function 157 changes the width of the target region 54a on the sample image 51.

23 and 24, only the width and widthwise position of the region of interest 63 are changed, but the user may also be able to change the length of the region of interest 63 in a direction perpendicular to the widthwise direction. For example, in FIG. 23, a cross section of the specimen tissue 6 is imaged, but if the specimen 5 is the mucosa and submucosa of the digestive tract, the user may be able to change or newly set the range of the region of interest 63 in the vertical direction of the specimen image 61, i.e., the depth direction of the mucosa and submucosa.

Furthermore, the user's operation to correct the dimensions of the region of interest 63 is not limited to the operation of changing the range bars 91-93, but may be able to specify the dimensions of the region of interest 63 in units of pixels or in units of image regions of a certain size. Furthermore, in addition to modifying the dimensions of the region of interest 63, the user may also be able to delete the region of interest 63.

(Variation 4)
Furthermore, the information processing device 100 may display numerical information such as the area, maximum width, or depth of the region of interest 63 on the specimen image 51 or the specimen image 61 in response to a user operation.

FIG. 25 is a diagram showing an example of an operation screen 145 according to Modification Example 4. The operation screen 145 includes a specimen image 51 and a specimen image 61. For example, the reception function 152 receives a user operation to press the length measurement button 84, and then receives a user operation to select one of the tissue regions 62. In this case, the display control function 158 displays the numerical value of the longest width of the region of interest 63 in the selected tissue region 62.

For example, the display control function 158 calculates the maximum width of the actual region of interest 63 based on the maximum width of the region of interest 63 on the specimen image 61 and the magnification ratio of the imaging device that captured the specimen image 61, and displays the calculation result.

The calculation of the maximum width may be performed by the region of interest recognition function 154. Furthermore, numerical information such as the area, maximum width, or depth of the region of interest 63 may be drawn on the specimen image 51 or the sample image 61 by the mapping function 157, rather than being displayed by the display control function 158.

In FIG. 25, the display control function 158 displays the numerical information on the specimen image 61, but the numerical information may also be displayed on the sample image 51.

(Variation 5)
Furthermore, depending on the operating rules of a medical institution or the like, the position of the imaging plane of the sample tissue 6 relative to the cut-out position may differ depending on the cut-out position.

Figure 26 is a diagram showing an example of the relationship between the cut-out positions and tissue regions 62 according to variant example 5. For cut-out position IDs "1" to "3" shown in Figure 26, the upper part of the cut-out position corresponds to tissue regions 62i to 62k. For cut-out position ID "4", the lower part of the cut-out position corresponds to tissue region 62l. In Figure 26, the number "4" indicating the cut-out position ID is inverted upside down to show that the lower part of the cut-out position for cut-out position ID "4" corresponds to tissue region 62l, but this display is not essential. Also, it may be shown by other methods whether the upper or lower part of the cut-out position corresponds to tissue region 62.

For example, if a specimen tissue 6 is further excised from above the excision position ID "4", two specimen area IDs, one above and one below, may be associated with one excision position ID "4". In this case, the two specimen area IDs, one above and one below, are represented by sub-numbers, for example, "4-1" and "4-2".

(Variation 6)
In the above-described embodiment, the specimen image 51 is an image of the surface of the specimen 5 captured from above, and the specimen image 61 is an image of a cross-section of the specimen tissue 6 excised from the specimen 5 captured at high magnification, but the imaging directions are not limited to these.

For example, the specimen image 51 may be an image of a cross section of the specimen 5. In this case, the mapping function 157 may map the range corresponding to the outline of the region of interest 63 recognized in the specimen image 61 onto the specimen image 51 as the target region 54.

Figure 27 is a diagram showing an example of a specimen image 51 and a specimen image 61 according to the sixth modification. The specimen image 51 and the specimen image 61 shown in Figure 27 are both images in which the cross section of the specimen 5 is the imaging plane. For this reason, the mapping function 157 corrects the outline of the region of interest 63 in the specimen image 61 to match the magnification of the specimen image 51, and maps the result onto the specimen image 51 as the target region 54. According to this modification, when the imaging direction of the specimen 5 is the same in the specimen image 51 and the specimen image 61, the region of interest 63 can be mapped onto the specimen image 51 with higher accuracy without converting it into one-dimensional information such as the width length.

(Variation 7)
The mapping function 157 may further render the target region 54 on the specimen image 51 in different manners depending on the depth direction length of the region of interest 63 in the specimen image 61 .

Figure 28 is a diagram showing an example of mapping of depth information of the region of interest 63 for the sample image 51 according to the seventh modification. For example, the mapping function 157 draws the target region 54 with the display mode changed like a heat map according to the depth length of the sample 5 in the region of interest 63. For example, the mapping function 157 may draw the target region 54 so that the color becomes darker as the depth length of the sample 5 in the region of interest 63 becomes longer. Furthermore, the mapping function 157 may display the target region 54 with a gradation of different colors so that the color of the target region 54 approaches red as the depth length of the sample 5 in the region of interest 63 becomes longer.

The mapping function 157 may, for example, draw the target area 54 in a different display manner depending on the relative length difference within the specimen image 61.

The region of interest recognition function 154 may also calculate the actual depth length of the region of interest 63 based on the magnification of the microscope that captured the specimen image 61. In this case, the mapping function 157 may draw the target region 54 so that the display mode differs depending on the absolute value of the calculated actual depth length of the region of interest 63.

In addition, similar to the method in the above embodiment in which the dimensions of the region of interest 63 are corrected using equation (1) and the correction result is recognized as the dimensions of the target region 54, the mapping function 157 may calculate the widthwise magnification of the specimen image 51 and the specimen image 61 from the difference between the widthwise length of the specimen region 52 at the cut-out position of the specimen image 51 and the widthwise length of the tissue region 62 corresponding to the cut-out position in the specimen image 61, and draw the display mode of the target region 54 so that it differs depending on the value obtained by multiplying the depthwise length of the region of interest 63 by the calculated magnification.

In this modified example, the relative or absolute depth of the lesion tissue depicted in the region of interest 63 is displayed on the specimen image 51, providing reference information for doctors and others to use when making a pathological diagnosis.

(Variation 8)
Furthermore, the mapping function 157 may depict the undulations of the surface of the specimen tissue 6 depicted in the specimen image 61 in the specimen image 51. For example, when the specimen 5 is the mucosa and submucosa of the digestive tract, the undulations of the mucosal surface of the specimen 5 can be identified by the specimen image 61 obtained by capturing a cross section of the specimen tissue 6.

The mapping function 157 plots the relative or absolute height difference of the undulations using contour lines or the like on the specimen image 51 captured using the membrane side of the specimen 5 as the imaging surface. Note that the plotting method is not limited to contour lines, and the height difference may be represented by a color gradation, such as a heat map.

(Variation 9)
In the above embodiment, the user sets the cut-out line 53 at an arbitrary position. However, the information processing device 100 may support the user when setting the cut-out line 53 .

FIG. 29 is a diagram showing an example of the cut-out position setting screen 143 relating to the ninth modified example. The cut-out position setting function 160 estimates an area in the specimen image 51 that is likely to contain a lesion, for example, by rule-based image processing such as selective search or deep learning. The display control function 158 displays the area estimated by the cut-out position setting function 160 on the cut-out position setting screen 143. The area 500 shown in FIG. 29 is an example of an area in the specimen image 51 that is likely to contain a lesion.

The information processing device 100 of this modified example can assist the user in setting the cut-out position so that a cross-section of the area 500 that is likely to contain a lesion can be obtained.

(Variation 10)
In the above embodiment, the sample image 51 after mapping processing and the like are transmitted from the information processing device 100 to an external device, but the external device may be permitted to access the sample image 51 after mapping processing stored in the memory circuitry 120 of the information processing device 100. For example, if the external device is a PC or the like, a user of the PC can view the sample image 51 after mapping processing stored in the memory circuitry 120 on the display 140 of the PC. In addition, the sample image 51 after mapping processing and the like stored in the memory circuitry 120 of the information processing device 100 may be permitted to be downloaded to an external device.

According to at least one of the embodiments described above, it is possible to easily allow a doctor or the like to grasp the correspondence between the cut-out position of the specimen tissue 6 in the specimen 5 depicted in the specimen image 51 and the region of interest 63, such as a lesion region, on the specimen tissue 6 depicted in the specimen image 61.

Although several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, modifications, and combinations of embodiments can be made without departing from the spirit of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and spirit of the invention.

5 Specimen 6, 6a to 6e Specimen tissue 51 Specimen image 52 Specimen region 53, 53a to 53f, 531, 532 Crop line 54, 54a to 54c, 54g, 54h, 54x Target region 56, 66 Marker 61, 61a, 61b Specimen image 62, 62a to 62f, 62i to 62l, 621, 622 Tissue region 63, 63a to 63c, 63g, 63h, 63x Region of interest 65a to 65c Center of gravity 91 to 94 Range bar 100 Information processing device 110 NW interface 120 Memory circuit 130 Input interface 140 Display 141 Setting operation screen 142 Mapping display screen 143 Crop position setting screen 144 to 146 Operation screen 150 Processing circuit 151 Acquisition function 152 Reception function 153 Tissue area recognition function 154 Area of interest recognition function 155 Extraction position recognition function 156 Correspondence function 157 Mapping function 158 Display control function 159 Transmission function 160 Extraction position setting function 201 Sample image storage device 202 Specimen image storage device 300 Network P Patient S Information processing system

Claims (15)

an acquisition unit that acquires a first image, which is an image of an entire specimen, and a second image, which is an image of at least one specimen tissue cut out from the specimen, at a higher magnification than the first image;
a tissue area recognition unit that recognizes a tissue area in which the sample tissue is depicted from the second image;
a region of interest recognition unit for recognizing a region of interest included in the tissue region;
an excision position recognition unit that recognizes an excision position of the sample tissue in the first image;
a correspondence unit that corresponds the cut-out position in the first image with the tissue region in the second image;
a mapping unit that maps the region of interest in the second image to a target region corresponding to the region of interest in the first image based on the correspondence between the extraction position and the tissue region;
a display control unit that causes the first image and the second image, in which the region of interest and the target region are mapped by the mapping unit, to correspond to each other and display them as separate images on the same screen of a display unit, and that causes the cut-out position on the first image to correspond to a display position on the second image of the sample tissue cut out from the cut-out position;
An information processing device comprising: a receiving unit that receives a user's operation to correct the correspondence between the extraction position in the first image and the tissue region in the second image,
The associating unit corrects the association between the cut-out position and the tissue region based on the user's operation accepted by the accepting unit.
The information processing device according to claim 1 . a receiving unit that receives a user's designation of the cut-out position in the first image;
a cut-out position setting unit that sets a cut-out position in the first image based on the designation of the cut-out position accepted by the accepting unit,
3. The information processing device according to claim 1 or 2 . the mapping unit corrects a size of the region of interest based on a difference between a size of a specimen region in which the specimen is depicted at the cut-out position of the first image and a size of the tissue region corresponding to the cut-out position, and maps the region of interest with the corrected size onto the target region.
The information processing device according to claim 1 . The mapping unit draws a figure representing the target region in the vicinity of the extraction position associated with the tissue region including the region of interest.
The information processing device according to claim 1 . the mapping unit draws a figure representing the target region on a side of a cut-out line indicating the cut-out position associated with the tissue region including the region of interest, where the sample tissue is cut out;
The information processing device according to claim 1 . a reception unit that receives a user's operation to correct a dimension of the region of interest,
The mapping unit changes a size of the target region based on a size of the region of interest corrected by the user.
The information processing device according to claim 1 . the region of interest is a lesion site in the specimen tissue,
The region of interest recognition unit inputs the second image to a trained model in which a correspondence relationship between a plurality of specimen images and an image region in which a lesion site is depicted in the plurality of specimen images is trained, and recognizes an image region output from the trained model as an image region in which the region of interest is depicted.
The information processing device according to claim 1 . The mapping unit renders the target region in the first image in a different manner depending on a length of the region of interest in the second image in a depth direction of the specimen.
The information processing device according to claim 1 . the region of interest comprises any of the following tissues: a tumor, a mucosa, a muscularis mucosa, a submucosa, or a muscularis propria;
When the region of interest includes a plurality of types of tissue, the mapping unit renders the target region in the first image in a different display manner for each type of tissue.
The information processing device according to claim 1 . A transmission unit that transmits the first image onto which the target region corresponding to the region of interest is mapped by the mapping unit to an external device.
The information processing device according to claim 1 . the display control unit displays the first image and the second image in which the sample tissue cut out from the specimen depicted in the first image is captured on one screen of the display unit, and displays the target region mapped on the first image as a linear figure.
The information processing device according to claim 1 . the first image includes a plurality of the cropping positions;
the display control unit displays, for each of the cut-out positions on the first image, an image in which a portion of the second image in which the tissue region corresponding to the cut-out position is depicted is cut out.
The information processing device according to claim 1 . the display control unit, when a user selects the first image, displays the first image larger than the second image, and when the user selects the second image, displays the second image larger than the first image.
The information processing device according to claim 1 . An information processing system including an image storage device and an information processing device,
The image storage device includes:
A first image is an image of the entire specimen, and a second image is an image of at least one specimen tissue cut out from the specimen and captured at a higher magnification than the first image;
The information processing device includes:
an acquisition unit that acquires the first image and the second image from the image storage device;
a tissue area recognition unit that recognizes a tissue area in which the sample tissue is depicted from the second image;
a region of interest recognition unit for recognizing a region of interest included in the tissue region;
an excision position recognition unit that recognizes an excision position of the sample tissue in the first image;
a correspondence unit that corresponds the cut-out position in the first image with the tissue region in the second image;
a mapping unit that maps the region of interest in the second image to a target region corresponding to the region of interest in the first image based on the correspondence between the extraction position and the tissue region;
a display control unit that causes the first image and the second image, in which the region of interest and the target region are mapped by the mapping unit, to be associated with each other and to be displayed as separate images on the same screen of a display unit, and causes the cut-out position on the first image to be associated with a display position on the second image of the sample tissue cut out from the cut-out position.
Information processing system.

Images