JP2022534916A - Passive Ultrasonic Sensor-Based Initialization for Image-Based Device Segmentation - Google Patents

Abstract

A controller 220 for determining the shape of an interventional medical device during an interventional medical procedure based on the position of the interventional medical device includes a memory 221 that stores instructions and a processor 222 that executes the instructions. Upon instruction, the system 200, including the controller 220, performs a process including obtaining the position of the interventional medical device 201 (S320) and obtaining an image of the volume containing the interventional medical device (S330). The process further includes applying image processing (S340) to the image based on the location of the interventional medical device 201 to identify the interventional medical device 201 including the geometry of the interventional medical device 201 . The process further includes segmenting the interventional medical device 201 to obtain a segmented representation of the interventional medical device 201 (S350). A segmented representation of the interventional medical device 201 is superimposed on the image (S360).

Description

Ultrasound tracking techniques detect passive ultrasound within the field of view for diagnostic ultrasound B-mode images by analyzing the signal received by a passive ultrasound sensor as an imaging beam from an ultrasound probe sweeps through the field of view (FOV). Estimate the position of the sensor (eg, PZT, PVDF, copolymer, or other piezoelectric material). Passive ultrasonic sensors are sound pressure sensors, such passive ultrasonic sensors are used to determine the position of an interventional medical device to which it is attached. Time-of-flight measurements provide the axial/radial distance of the passive ultrasonic sensor from the imaging array of the ultrasonic probe, while amplitude measurements and knowledge of the direct beam firing sequence provide the lateral/angular distance of the passive ultrasonic sensor. Provide location.

FIG. 1 shows a known system for tracking interventional medical devices using passive ultrasound sensors. In FIG. 1, an ultrasound probe 102 emits an imaging beam 103 that sweeps across a passive ultrasound sensor 104 at the tip of an interventional medical device 105 . An image 107 of tissue is fed back by the ultrasound probe 102 . The position of the passive ultrasonic sensor 104 at the tip of the interventional medical device 105 is provided as the tip position 108 during determination by the signal processing algorithm. The tip location 108 is superimposed on the tissue image 107 as an overlay image 109 . Tissue image 107 , tip location 108 , and overlay image 109 are all displayed on display 100 .

Known technologies for passive ultrasonic sensors provide the position of the passive ultrasonic sensor 104 but not the geometry of the interventional medical device 105 . For example, in many clinical situations, such as cardiac and vascular interventions, it is advantageous to determine the shape of interventional medical device 105 .

It is an object of the present invention to provide, at least in part, this shape.

According to one aspect of the present disclosure, a controller for determining a shape of an interventional medical device during an interventional medical procedure based on a position of the interventional medical device includes a memory that stores instructions and a processor that executes the instructions. . When executed by the processor, the instructions cause the system, including the controller, to perform processes including obtaining a position of the interventional medical device and obtaining an image of a volume containing the interventional medical device. A process performed when the processor executes the instructions imparts image processing to the image based on the location of at least one point of the interventional medical device to identify the interventional medical device including the geometry of the interventional medical device. further comprising: Processes performed when the processor executes the instructions further include segmenting the interventional medical device to obtain a segmented representation of the interventional medical device. A segmented representation of the interventional medical device is superimposed on the image.

According to another aspect of the disclosure, a tangible non-transitory computer-readable storage medium stores a computer program. When executed by a processor, the computer program causes a system including a tangible, non-transitory computer-readable storage medium to perform a process for determining the shape of an interventional medical device during an interventional medical procedure based on the position of the interventional medical device. Let A process executed when a processor executes a computer program from a tangible, non-transitory computer-readable storage medium includes obtaining a position of at least one point of an interventional medical device and imaging a volume containing the interventional medical device. and obtaining. The process performed when the computer program is executed by the processor performs image processing based on the position of at least one point of the interventional medical device to identify the interventional medical device including the geometry of the interventional medical device. further comprising granting to Processes performed when the computer program is executed by the processor further include segmenting the interventional medical device to obtain a segmented representation of the interventional medical device. A segmented representation of the interventional medical device is superimposed on the image.

According to yet another aspect of the present disclosure, a system for determining the shape of an interventional medical device during an interventional medical procedure based on the position of a passive ultrasound sensor located using an ultrasound imaging probe includes: , an ultrasound imaging probe, a passive ultrasound sensor, and a controller. An ultrasound imaging probe emits a beam during an interventional medical procedure. A passive ultrasound sensor is secured to an interventional medical device during an interventional medical procedure. The controller includes a memory that stores instructions and a processor that executes the instructions. When executed by a processor, the instructions direct the system to acquire the position of a passive ultrasound sensor based on the emission of a beam from an ultrasound imaging probe and to image a volume containing the interventional medical device and the passive ultrasound sensor. and performing a process including obtaining Processes performed when the processor executes the instructions perform image processing based on the position of the passive ultrasound sensor to identify the interventional medical device, including the geometry of the interventional medical device and the position of the interventional medical device. further comprising granting to Processes performed when the processor executes the instructions further include segmenting the interventional medical device to obtain a segmented representation of the interventional medical device. A segmented representation of the interventional medical device is superimposed on the image along with the position of the passive ultrasound sensor.

The illustrative embodiments are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that the various features are not necessarily drawn to scale. In fact, dimensions may be arbitrarily increased or decreased for clarity of discussion. Like reference numerals refer to like elements wherever applicable and practical.

1 shows a known system for tracking interventional medical devices using passive ultrasonic sensors; FIG. FIG. 2 illustrates a system for passive ultrasonic sensor-based initialization for image-based device segmentation, according to a representative embodiment; FIG. 4 illustrates a process of passive ultrasonic sensor-based initialization for image-based device segmentation, according to a representative embodiment; FIG. 10 illustrates visualization progress in passive ultrasound sensor-based initialization for image-based device segmentation, according to representative embodiments; FIG. 4B illustrates another process of passive ultrasonic sensor-based initialization for image-based device segmentation, according to a representative embodiment; FIG. 4B illustrates another process of passive ultrasonic sensor-based initialization for image-based device segmentation, according to a representative embodiment; FIG. 3 illustrates a set of two-dimensional or X-plane images used to optimize views for passive ultrasound sensor-based initialization for image-based device segmentation, according to representative embodiments; FIG. 10 illustrates a visualization of an interventional medical device mesh superimposed on an ultrasound volume in passive ultrasound sensor-based initialization for image-based device segmentation, according to a representative embodiment;

In the following detailed description, for purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth to provide a thorough understanding of embodiments in accordance with the present teachings. Descriptions of known systems, devices, materials, methods of operation, and methods of manufacture may be omitted so as not to obscure the description of the representative embodiments. Nevertheless, systems, devices, materials, and methods within the purview of those skilled in the art are within the present teachings and used in accordance with the exemplary embodiments. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. The defined terms are in addition to their technical and scientific meanings as commonly understood and accepted in the technical field of the present teachings.

The terms first, second, third, etc. are used herein to describe various elements or components, which elements or components are not limited by these terms. It should be understood that this should not be done. These terms are only used to distinguish one element or component from another. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in this specification and the appended claims, singular terms are intended to include both singular and plural unless the context clearly dictates otherwise. Additionally, as used herein, the terms "comprising, including, having," and/or "including, including, having," and/or similar terms are explicitly stated. The presence of any feature, element and/or component does not preclude the presence of one or more other features, elements, components and/or groups thereof. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.

Unless otherwise specified, when an element or component is referred to as being “connected to,” “coupled,” or “adjacent to” another element or component, the element or component is referred to as the other element or component. It will be understood that components may be directly connected or coupled, or there may be intervening elements or components. That is, these and similar terms encompass where one or more intermediate elements or components may be utilized to connect two elements or components. However, when an element or component is said to be "directly connected" to another element or component, this means that the two elements or components are connected to each other without intermediate or intervening elements or components. Contains only cases.

In view of the foregoing, therefore, the present disclosure, through one or more of its various aspects, embodiments, and/or specific features or sub-components, achieves one of the advantages as specifically noted below. intended to bring out one or more For purposes of explanation and not limitation, example embodiments disclosing specific details are set forth to provide a thorough understanding of embodiments in accordance with the present teachings. However, other embodiments consistent with the present disclosure that depart from the specific details disclosed herein remain within the scope of the appended claims. Moreover, descriptions of well-known apparatus and methods may be omitted so as not to obscure the description of the example embodiments. Such methods and apparatus are within the scope of this disclosure.

As described herein, the shape of an interventional medical device (eg, wire) is readily obtained using the position of passive ultrasound sensors. The shape is quantified using a system that identifies the positions of passive ultrasonic sensors. The position of the passive ultrasound sensor is used to initialize an image processing algorithm that determines the shape of the device using, for example, spatial filtering or cross-correlation with a known shape. After the geometry of the interventional medical device is determined, a mesh of the interventional medical device is generated and overlaid to enhance visualization.

FIG. 2 illustrates a system for passive ultrasonic sensor-based initialization for image-based device segmentation, according to representative embodiments.

In FIG. 2, ultrasound system 200 includes ultrasound imaging probe 210, controller 220, console 290, interventional medical device 201, and passive ultrasound sensor S1. It is noted that ultrasound system 200 represents a system that includes controller 220 used to acquire the position of an interventional medical device and an image of a volume containing the interventional medical device. However, other types of location and imaging systems may be used to perform the features described herein and still be within the spirit and scope of the disclosure. Similarly, the ultrasound system 200 is configured to apply image processing to the images based on the position of the interventional medical device 201 and to segment the interventional medical device to obtain a segmented representation of the interventional medical device 201. represents a system including a controller 220 of . However, other types of imaging systems may be used to carry out the features described herein and still be within the spirit and scope of the disclosure. Accordingly, the systems described herein need not be ultrasound systems.

Controller 220 includes a memory 221 that stores instructions and a processor 222 that executes instructions. It should also be noted that the controllers 220 described herein may include multiple processors each including a combination of memory and processor to perform one or more of the characteristic functions ascribed to the controllers 220 herein. devices.

Console 290 includes a memory 291 that stores instructions and a processor 292 that executes the instructions. Console 290 further includes monitor 295 and touch panel 296 . Memory 291 and processor 292 can be considered a sensor unit that determines the position of passive ultrasonic sensor S1 and provides the position of passive ultrasonic sensor S1 to controller 220 . Alternatively, another combination of memory and processor (not shown) receives voltage readings from passive ultrasonic sensor S1 and beam timing from ultrasonic imaging probe 210 to determine the position of passive ultrasonic sensor S1. It may be used to determine and provide to controller 220.

By using the ultrasound system 200 or other embodiments consistent with those described herein, the shape of the interventional medical device 201 can be determined using the passive ultrasound sensor S1 at the tip of the interventional medical device 201 (or at the interventional medical device 201). quantified by locating another position of the device 201). The tip of the interventional medical device 201 is located using a passive ultrasound sensor S1 in two-dimensional ultrasound space or three-dimensional ultrasound space. Image processing techniques such as spatial filtering are then applied to the ultrasound image to enhance structures that are potentially identifiable as the body of the interventional medical device 201 . The interventional medical device 201 is segmented and superimposed on the image based on the most prominent device-like structures appearing near the known location of the passive ultrasound sensor S1.

Controller processor 222 or processor 292 is tangible and non-transitory. The term "non-transitory" as used herein should be interpreted as a property of a state that lasts for a period of time rather than as a permanent property of the state. The term "non-transitory" specifically denies transitory properties such as carriers or signals or other forms of properties that exist only temporarily at all times and places. A processor is a product and/or a machine component. Processor 222 of controller 220 is configured to execute software instructions to perform functions as described in various embodiments herein. Processor 22 of controller 220 may be a general purpose processor or may be part of an application specific integrated circuit (ASIC). Controller processor 222 may also be a microprocessor, microcomputer, processor chip, controller, microcontroller, digital signal processor (DSP), state machine, or programmable logic device. Controller processor 222 may also be a logic circuit including a programmable gate array (PGA), such as a field programmable gate array (FPGA), or another type of circuit including discrete gate and/or transistor logic. The processor 222 of the controller can be a central processing unit (CPU), a graphics processing unit (GPU), or both. Additionally, the processors described herein may include multiple processors, parallel processors, or both. Multiple processors may be included in a single device, multiple devices, or may be combined. As used herein, "processor" encompasses electronic components capable of executing programs or machine-executable instructions. References to a computing device that includes a "processor" should be interpreted as possibly including more than one processor or processing core. A processor may be, for example, a multi-core processor. A processor also refers to a collection of processors within a single computer system or distributed among multiple computer systems. The term computing device should also be interpreted to refer to a collection or network of computing devices, each of which includes one or more processors. Many programs have instructions that are executed by multiple processors, which may be within the same computing device or even distributed across multiple computing devices. be.

Memory, such as memory 221 or memory 291 described herein, is a tangible storage medium that stores data and executable instructions and is non-transitory while the instructions are stored therein. The term "non-transitory" as used herein should be interpreted as a property of a state that lasts for a period of time rather than as a permanent property of the state. The term "non-transitory" specifically denies transitory properties such as carriers or signals or other forms of properties that exist only temporarily at all times and places. The memory described herein is an article of manufacture and/or a machine component. Memory, as described herein, is a computer-readable medium from which data and executable instructions can be read by a computer. The memory described herein includes random access memory (RAM), read only memory (ROM), flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), register , hard disk, removable disk, tape, compact disk read-only memory (CD-ROM), digital versatile disk (DVD), floppy disk, Blu-ray disk, or any other form of storage known in the art is a medium. Memory can be volatile or non-volatile, secure and/or encrypted, non-secure and/or non-encrypted. "Memory" is an example of a computer-readable storage medium. Computer memory is memory that is directly accessible to a processor. Examples of computer memory include, but are not limited to, RAM memory, registers, and register files. References to "computer memory" or "memory" should be construed as potentially multiple memories. The memory is, for example, multiple memories within the same computer system. The memory may also be multiple memories distributed among multiple computer systems or computing devices.

For convenience, reference to features of ultrasound system 200 is used throughout this disclosure in and for other embodiments for the purpose of consistency. However, as noted above, ultrasound system 200 is merely one example of a system that performs the functions and functionality described herein.

FIG. 3 illustrates a process of passive ultrasonic sensor-based initialization for image-based device segmentation, according to representative embodiments.

At S310, the process of FIG. 3 begins by emitting a beam from an ultrasound imaging probe. For example, the ultrasound imaging probe 210 of FIG. 2 emits beams as part of a sequence of beams at S310.

Next, at S320, the process of FIG. 3 proceeds to obtain the position of the passive ultrasound sensor S1 located in the ultrasound volume. The position of the passive ultrasonic sensor S1 is determined in a predetermined three-dimensional coordinate system with an origin set for the ultrasonic image. Registration is performed by registering the three-dimensional coordinate system of the ultrasound image to another three-dimensional coordinate system, for example, the image from the viewpoint of the interventional medical device when the interventional medical device 201 is an imaging device such as an endoscope. This is done to align with other three-dimensional coordinate systems, such as for interventional medical device 201, including. Registration is performed by aligning image landmarks in two different underlying coordinate systems in order to import one coordinate system into the other by image transformation. Registration may also transform 2D or 3D ultrasound images into 2D or projection imaging space, e.g., by identifying common landmarks in each imaging space, or by identifying poses of ultrasound transducers in another imaging space. is performed to align with the

At S330, the process of FIG. 3 next includes acquiring an image of the ultrasound volume that includes the interventional medical device 201 and the passive ultrasound sensor S1. The image is an ultrasound image resulting from an ultrasound beam emitted by ultrasound imaging probe 210 during an interventional medical procedure.

At S340, the process of FIG. 3 uses constraints to identify the interventional medical device 201, including the shape of the interventional medical device 201 and the position of the interventional medical device 201, based on the position of the passive ultrasonic sensor S1. and applying image processing to the image. The image processing in S340 is applied based on the location of the points of the interventional medical device 201 or based on the location of at least one point of the interventional medical device 201 . Constraints used at S340 include one or more of pixel intensity characteristics, relative pixel locations, predetermined shape, and/or dimensional characteristics. As an example, the pixel intensity of the pixels corresponding to the location of the interventional medical device 201 is higher than the pixel intensity of the pixels corresponding to the anatomical features of the subject of the interventional medical procedure. As another example, the search is limited to pixels within a predetermined range measured in distance, pixels, or something else from the pixel corresponding to the position of the passive ultrasonic sensor S1. Image processing involves filtering pixels to remove pixels based on predetermined characteristics so that the remaining elements of the original image better represent the interventional medical device 201 . That is, elements of the device geometry (ie interventional medical device 201) remain after filtering. Therefore, image processing intentionally discards representations of shapes, such as anatomical features, but not necessarily based on shape, to the extent that discarding is based on pixel intensity or pixel position, or the like.

In other examples, the characteristics used for image filtering include predetermined shapes selected from a library of predetermined shapes corresponding to, for example, different interventional medical devices having different shapes. In another example, properties used for image filtering include dimensional properties of a given shape such as minimum or maximum length, width, height, diameter, radius, cross-sectional area, curvature, and the like. For example, a wire as the interventional medical device 201 has a very small cross-section, and searching for the wire looks for ends with an area or diameter less than a threshold value corresponding to the very small cross-section.

At S350, the process of FIG. 3 then reconstructs and segments the shape of the interventional medical device 201 or the predetermined shape of the interventional medical device to obtain a segmented representation of the interventional medical device 201. including. A given device shape is used as a given constraint, and as a result, a given device shape is used as a candidate shape for interventional medical device 201 . Elements of the device geometry (ie, interventional medical device 201) that remain after filtering are used for reconstruction of the device geometry. A segmentation is a representation of a structure, such as an anatomical feature, and/or a surface of an interventional medical device, such as the interventional medical device 201, for example, a set of points in three-dimensional (3D) coordinates on the surface of the structure and adjacent It consists of triangular planar segments defined by connecting groups of three points that correspond to each other, so that the entire structure is covered with a mesh of non-intersecting triangular planar surfaces. A three-dimensional model of the structure is obtained by segmentation. The segmentation described herein with respect to S350, and similar operations in other embodiments, segment the interventional medical device 201, anatomical structures, and/or other structures present in the three-dimensional ultrasound volume. including running

At S360, the process of FIG. 3 includes superimposing the segmented representation of the interventional medical device 201 together with the position of the passive ultrasonic sensor S1 onto the image.

The process of FIG. 3 and features of other embodiments herein are used for many types of interventional procedures. For example, in structural heart repair, the use of features herein is used to check to ensure correct geometry/path during placement of interventional medical device 201 . Additionally or alternatively, use of the features herein is used to detect whether the interventional medical device 201 has passed a desired path during an invasive procedure such as a septal puncture. For example, fiducial markers may be placed at the locations of the target anatomy. As an additional or alternative use of the features described herein for organic heart repair, the shape of a mitral valve or other implant is quantified in three dimensions during organic heart repair.

FIG. 4 shows a visualization progress in passive ultrasound sensor-based initialization for image-based device segmentation, according to a representative embodiment.

In visualization A, the progression of FIG. 4 shows the tip position of interventional medical device 201 .

In visualization B, the progression of FIG. 4 shows the effect or result of image processing. In FIG. 4, the effect is to greatly reduce the overall image detail, but greatly increase the detail of the interventional medical device 201 .

In visualization C, the progression of FIG. 4 shows the interventional medical device 201 superimposed on the original image. That is, the interventional medical device 201 identified and highlighted in visualization B is superimposed in visualization C. FIG. Naturally, the visualization of the interventional medical device 201 from visualization B is augmented, such as by filling in missing pixels, to fully generate the shape well expected by the model extracted from the library of models. . Therefore, the image processing used to reduce overall image detail in visualization B further includes enhancement to enhance detail of the interventional medical device.

In FIG. 4, the geometry of the interventional medical device 201 is obtained from image processing initialized based on the position of the passive ultrasound sensor S1. A tip or other position of the interventional medical device 201 is placed in the 3D volume using tracking of the passive ultrasonic sensor S1. Image processing techniques are applied in the embodiment of FIG. 4 to make the tubular structures appear sharper. The highest intensity tubular structure near the position of the passive ultrasound sensor S1 is assumed to be the body of the interventional medical device 201 and is then superimposed on the original image.

In one embodiment, the process comprising the progression of FIG. 4 is used with image processing algorithms applied to the three-dimensional volume to locate and enhance visualization of interventional medical device 201 . A two-dimensional imaging plane from ultrasound imaging probe 210 is used to sweep a three-dimensional space containing interventional medical device 201 . The resulting images are then analyzed to obtain a three-dimensional volume model of the interventional medical device 201 in space and/or in space. The spatial and/or three-dimensional volumetric model of the interventional medical device 201 is then segmented. As an example, an initial two-dimensional plane is set to contain the passive ultrasonic sensor, and then the two-dimensional plane is incrementally rotated to roughly pivot around the passive ultrasonic sensor. Two-dimensional image processing such as filtering and clustering is performed on each frame of the rotational sweep to identify portions of the interventional medical device 201 . Analyzing each plane's view of the interventional medical device 201 to determine which plane from the sweep provides the best representation of the device, such as the longest section of the device, or the extreme extremity near the passive ultrasound sensor. It is decided whether to show the part. A segmentation of the interventional medical device is then displayed.

FIG. 5 illustrates another process of passive ultrasonic sensor-based initialization for image-based device segmentation, according to representative embodiments.

At S510, the process of FIG. 5 begins by obtaining the position of the passive ultrasound sensor S1 within the ultrasound volume.

At S520, the process of FIG. 5 next includes setting filter constraints and searching for constraints. That is, searching for the interventional medical device 201 in the image begins with locating the passive ultrasonic sensor S1. A constraint search is a search for regions in the image that satisfy a given constraint. A region that satisfies a given constraint includes one or more such pixel characteristics that are within a specified radius or distance from the passive ultrasound sensor and have pixel intensities higher than a given threshold. In one example, the predetermined constraint is a particular shape that is part of the shape of the interventional medical device 201, e.g., a corner or set of corners, one or more angles in the profile of the interventional medical device 201 that appear in the ultrasound image; and other types of constraints, etc. The constraint may be a predetermined constraint or may vary depending on the characteristics of the subject of the interventional medical procedure, including the type of interventional medical device 201, the type of surgery, and anatomical characteristics.

At S530, the process of FIG. 5 then moves to apply a filter to the ultrasound volume using the filter constraints. A process performed by controller 220 or ultrasound system 200 including controller 220 includes filtering images from an ultrasound imaging system to remove representations of a subject of an interventional medical procedure. For example, since the object sought for image processing is the interventional medical device 201, filtering is performed to remove representations of tissues, bones, and other anatomical features of the subject of the interventional medical procedure.

At S540, the process of FIG. 5 includes searching regions of maximum intensity surrounding the location of the passive ultrasonic sensor and identifying the point with maximum intensity. The maximum intensity is the pixel intensity above a predetermined threshold, or the relative intensity of neighboring pixel intensities above a predetermined threshold.

At S550, the process of FIG. 5 next includes connecting adjacent points. Neighboring points are for pixels with intensities above the threshold and within a predetermined distance from the position of the passive ultrasonic sensor S1.

At S560, the process of FIG. 5 ends by superimposing the shape of the interventional medical device 201 on the ultrasound image. The superimposed interventional medical device 201 is a representation of the interventional medical device 201 superimposed on the ultrasound image and highlighted, such as by a highlighted outline.

The process of FIG. 5 and other embodiments herein are used for many types of interventional procedures. For example, in peripheral vascular interventions, the features herein are used to monitor the geometry of an interventional medical device 201, such as a wire, during stenosis or occlusion crossing to detect buckling of the interventional medical device 201. . Additionally or alternatively, features herein are used to detect the advancement of an interventional medical device 201, such as a wire, relative to a vessel to check if the interventional medical device 201 has exited the vessel wall.

FIG. 6 illustrates another process of passive ultrasonic sensor-based initialization for image-based device segmentation, according to representative embodiments.

The process of FIG. 6 begins at S610 by obtaining the position of the passive ultrasound sensor within the ultrasound volume.

Next, at S620, the process of FIG. 6 includes setting filter constraints and identifying a model of interventional medical device 201. FIG. Filter constraints may be predefined constraints or constraints set dynamically for different interventional medical procedures. Filter constraints are applied to identify regions that satisfy predetermined constraints, so that image processing to be performed is performed only in the identified regions, or at least starts in the identified regions. do.

At S630, the process of FIG. 6 next includes fixing the tip of the identified interventional medical device 201 model to the position of the passive ultrasound sensor within the ultrasound volume.

At S640, the process of FIG. 6 includes determining the optimal angle of interventional medical device 201 relative to the position of passive ultrasound sensor S1 within the ultrasound volume. This angle is used to position the interventional medical device 201 starting from the position of the ultrasonic sensor S1 and aligning the direction of the angle from the passive ultrasonic sensor S1.

At S650, the process of FIG. 6 concludes by superimposing the geometry of the interventional medical device 201 on the ultrasound image. For example, the mesh resulting from segmenting the shape of a given structure as the interventional medical device 201 is registered based on the process of S640, and then representations of the corners or other extremities of the interventional medical device 201 are made into passive ultra It is arranged to overlap the representation of the passive ultrasonic sensor S1 in or on the acoustic image.

FIG. 7 shows a set of two-dimensional or X-plane images used to optimize views for passive ultrasound sensor-based initialization for image-based device segmentation, according to a representative embodiment.

As shown in FIG. 7, five different two-dimensional or X-plane images are labeled A, B, C, D, and E, each with varying interventional medical device 201 for different viewpoints. Shows the view in which The most complete view is the view of the two-dimensional or X-plane image “C”, which corresponds to the view with the greatest amount of detail of interventional medical device 201 .

In the embodiment of FIG. 7, images from two-dimensional or X-plane views can be observed to see which provides the best detail of the interventional medical device 201 . The images from FIG. 7 are acquired before or after segmentation of the interventional medical device 201, and then, for example, identify or determine the best placement, orientation, and posture of the interventional medical device 201 on the underlying 3D ultrasound volume. used for verification. As an example consistent with the example described in the context of FIG. 4, an initial two-dimensional plane is set to contain the passive ultrasonic sensor, and then a two-dimensional plane is shown schematically around the passive ultrasonic sensor. Incrementally rotated to pivot. In embodiments where position is determined by a mechanism other than a passive ultrasonic sensor, the position of the point on interventional medical device 201 is still used as the starting point of the sweep. Two-dimensional image processing such as filtering and clustering is performed on each frame of the rotational sweep to identify portions of the interventional medical device 201 . Analyze each plane's view of the interventional medical device 201 to determine which plane from the sweep provides the best representation of the device, such as the longest section of the device, or the extreme end near the passive ultrasound sensor. It is decided whether to show the part. A segmentation of the interventional medical device is then displayed.

FIG. 8 shows visualization of a mesh of an interventional medical device 201 superimposed within an ultrasound volume in passive ultrasound sensor-based initialization for image-based device segmentation, according to a representative embodiment.

In the embodiment of Figure 8, the interventional medical device 201 is an SHD device, which is segmented into meshes and overlaid on a three-dimensional ultrasound volume. The shape of the interventional medical device 201 is known, rigid, and adapted to the ultrasound image of the three-dimensional ultrasound volume. The positions of the passive ultrasound sensors are used to initialize the search volume to identify the proper orientation, pose, and general placement of the interventional medical device 201 of FIG.

As an example of the embodiment of FIG. 8, the interventional medical device 201 is rigid, such as in the case of an atrial septal puncture needle or a mitral valve repair device. Image processing techniques are used to search for specific features of the device in the area near the passive ultrasonic sensor. The mesh of the known device is then superimposed on a 2D or 3D ultrasound image as shown in FIG. A superimposed mesh of a known device is superimposed on the ultrasound image and highlighted, such as by color, brightness, or other visual characteristics, so that the superimposed mesh is clearly distinguished in the combined image.

As a result, passive ultrasound sensor-based initialization for image-based device segmentation can identify the interventional medical device 201 in the ultrasound image and place a model of the interventional medical device 201 in or on the ultrasound image. be possible. Passive ultrasound sensor-based initialization for image-based segmentation, such as ensuring that the interventional medical device 201 is positioned along the correct path in the correct posture, for example during an interventional medical procedure (e.g., and/or to quantify interventional medical device 201 in three dimensions during interventional medical procedures. be.

Although passive ultrasonic sensor-based initialization for image-based device segmentation has been described with reference to several exemplary embodiments, the words used are words of description and illustration, rather than words of limitation. It should be understood that Deviating from the scope and spirit of passive ultrasonic sensor-based initialization for image-based device segmentation within its aspects, as presently described and as amended, within the scope of the appended claims Changes may be made without Although passive ultrasonic sensor-based initialization for image-based device segmentation has been described with reference to specific means, materials, and embodiments, passive ultrasonic sensor-based initialization for image-based device segmentation is , is not intended to be limited to the details disclosed, but rather passive ultrasonic sensor-based initialization for image-based device segmentation is intended to cover all functions such as those falling within the scope of the appended claims. to substantially equivalent structures, methods, or uses.

For example, the examples above illustrate the use of the features herein for organic heart repair or peripheral vascular intervention. Other practical applications of the features herein include detecting bending of interventional medical devices 201, such as needles, during deep tissue biopsies. Other practical applications of the features herein include providing a reliable intra-body two-dimensional projection reference for registration between ultrasound and X-rays.

The following examples are provided.

Example 1
A controller (220) for determining a shape of an interventional medical device (201) during an interventional medical procedure based on a position of the interventional medical device (201), comprising:
a memory (221) for storing instructions;
A processor (222) that executes instructions that, when executed on the processor, cause the system, including the controller (220), to:
obtaining (S320) the location of a point of the interventional medical device (201);
obtaining (S330) an image of a volume containing an interventional medical device (201);
applying image processing (S340) to the image based on the position of the interventional medical device to identify the interventional medical device, including the shape of the interventional medical device;
segmenting (S350) the interventional medical device to obtain a segmented representation of the interventional medical device, wherein the segmented representation of the interventional medical device is superimposed (S360) on the image; a controller (220), including a processor (222), which implements a process including: performing (S350);

Example 2
The controller (220) of Example 1, wherein the process implemented by the system comprises:
Obtaining (S320) the position of the passive ultrasonic sensor based on the emission of the beam from the ultrasonic imaging probe, wherein the position of the interventional medical device corresponds to the position of the passive ultrasonic sensor (S320). and
starting (S340) searching for an interventional medical device in the image from the position of the passive ultrasound sensor and searching for a region in the image that satisfies predetermined constraints to apply image processing to the image;
A controller (220) in which the segmented representation of the interventional medical device is superimposed on the image together with the position of the passive ultrasound sensor.

Example 3
The controller (220) of Example 2, wherein the process implemented by the system comprises:
further comprising applying (S340) a predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the geometry of the interventional medical device;
A controller (220) wherein the predetermined constraints include pixel intensity characteristics and pixel position relative to the position of the passive ultrasonic sensor.

Example 4
The controller (220) of Example 2, wherein the process implemented by the system comprises:
further comprising applying (S340) a predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the geometry of the interventional medical device;
A controller (220), wherein the predetermined constraints include at least one predetermined shape to be used as a candidate shape for the interventional medical device.

Example 5
The controller (220) of Example 2, wherein the process implemented by the system comprises:
further comprising applying (S340) a predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the geometry of the interventional medical device;
A controller (220), wherein the predetermined constraint includes at least one-dimensional properties of the predetermined shape.

Example 6
The controller (220) of Example 1, wherein the process implemented by the system comprises:
filtering (S340) the image to remove a representation of a subject of an interventional medical procedure into which the interventional medical device is inserted, wherein elements of the shape of the interventional medical device remain in the image after filtering; filtering (S340);
Reconstructing (S350) the shape of the interventional medical device based on elements of the shape of the interventional medical device remaining in the image after filtering to obtain a reconstruction of the shape of the interventional medical device. , the controller (220).

Example 7
The controller (220) of Example 6, wherein the segmentation is performed on the reconstruction of the shape of the interventional medical device, and the segmented representation of the interventional medical device is the segmentation of the reconstruction of the shape of the interventional medical device a controller (220), including the rendered representation.

Example 8
The controller (220) of Example 1, wherein the segmentation is performed into a predetermined shape based on image processing, and the segmented representation of the interventional medical device is segmented into the predetermined shape of the interventional medical device. A controller (220) that includes a representation of the shape so that a segmented representation of the predetermined shape is superimposed on the image.

Example 9
A tangible, non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, transfers to a system including the tangible, non-transitory computer-readable storage medium an interventional medical device during an interventional medical procedure. causing a process for determining a shape to be performed based on the position of the interventional medical device, the process being performed when the processor (222) executes a computer program from a tangible non-transitory computer-readable storage medium;
obtaining (S320) a location of an interventional medical device;
obtaining an image of a volume containing an interventional medical device (S330);
applying image processing to the image based on the position of at least one point of the interventional medical device to identify the interventional medical device including the shape of the interventional medical device (S340);
segmenting (S350) the interventional medical device to obtain a segmented representation of the interventional medical device, wherein the segmented representation of the interventional medical device is superimposed (S360) on the image; and (S350) a tangible non-transitory computer-readable storage medium.

Example 10
The tangible non-transitory computer-readable storage medium of Example 9, wherein the process performed by the system comprises:
searching for an interventional medical device in the image starting from the position of the passive ultrasound sensor (S340) and searching for regions in the image that satisfy predetermined constraints to apply image processing to the image; A non-transitory computer-readable storage medium.

Example 11
Example 10. The tangible non-transitory computer-readable storage medium of Example 10, wherein the process performed by the system comprises:
Obtaining (S320) the position of the passive ultrasonic sensor based on the emission of the beam from the ultrasonic imaging probe, wherein the position of the interventional medical device corresponds to the position of the passive ultrasonic sensor (S320). and
further comprising applying (S340) a predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the geometry of the interventional medical device;
the predetermined constraints include pixel intensity characteristics and pixel positions relative to positions of the passive ultrasonic sensor;
A tangible non-transitory computer-readable storage medium in which a segmented representation of an interventional medical device is superimposed on an image together with the position of a passive ultrasound sensor.

Example 12
Example 10. The tangible non-transitory computer-readable storage medium of Example 10, wherein the process performed by the system comprises:
further comprising applying (S340) a predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the geometry of the interventional medical device;
A tangible non-transitory computer-readable storage medium, wherein the predetermined constraints include at least one predetermined shape to be used as a candidate shape for an interventional medical device.

Example 13
Example 10. The tangible non-transitory computer-readable storage medium of Example 10, wherein the process performed by the system comprises:
further comprising applying (S340) a predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the geometry of the interventional medical device;
A tangible non-transitory computer-readable storage medium, wherein the predetermined constraint includes at least one dimensional characteristic of the predetermined shape.

Example 14
The tangible non-transitory computer-readable storage medium of Example 9, wherein the process performed by the system comprises:
Filtering (S340) the image to remove representations of the subject of the interventional medical procedure into which the interventional medical device is inserted, where elements of shape remain in the image after filtering (S340). ) and
Reconstructing (S350) the shape of the interventional medical device based on elements of the shape of the interventional medical device remaining in the image after filtering to obtain a reconstruction of the shape of the interventional medical device. , a tangible non-transitory computer-readable storage medium.

Example 15
The tangible non-transitory computer-readable storage medium of Example 14, comprising:
A tangible non-transitory computer readable, wherein the segmentation is performed on the reconstruction of the shape of the interventional medical device, and the segmented representation of the interventional medical device comprises the segmented representation of the reconstruction of the shape of the interventional medical device storage medium.

Example 16
The tangible non-transitory computer-readable storage medium of Example 9, comprising:
Segmentation is performed on the predetermined shape based on image processing, the segmented representation of the interventional medical device comprising a segmented representation of the predetermined shape of the interventional medical device, such that the predetermined shape A tangible non-transitory computer-readable storage medium on which the segmented representation is superimposed on the image.

Example 17
A system (200) for determining the shape of an interventional medical device (201) during an interventional medical procedure based on the position of a passive ultrasound sensor (S1) located using an ultrasound imaging probe (210). There is
an ultrasound imaging probe (210) that emits a beam during an interventional medical procedure;
a passive ultrasonic sensor (S1) fixed to an interventional medical device (201) during an interventional medical procedure;
A controller (220) comprising a memory (221) for storing instructions and a processor (222) for executing the instructions,
When executed by processor (222), the instructions cause the system to:
obtaining (S320) the position of the passive ultrasonic sensor based on the emission of the beam from the ultrasonic imaging probe;
acquiring an image of a volume containing an interventional medical device and a passive ultrasound sensor (S330);
applying image processing (S340) to the image based on the position of the passive ultrasound sensor to identify the interventional medical device including the shape of the interventional medical device and the position of the interventional medical device; and segmenting the interventional medical device. segmenting (S350) the interventional medical device to obtain a rendered representation, wherein the segmented representation of the interventional medical device is superimposed on the image together with the positions of the passive ultrasound sensors (S360). , segmenting (S350).

Example 18
The system of Example 17, comprising:
a sensor unit (291/292) for determining the position of the passive ultrasonic sensor and providing the position of the passive ultrasonic sensor to the controller (220);
a display (295) displaying the segmented representation of the interventional medical device based on the segmentation by the controller (220) and displaying the position of the passive ultrasound sensor determined by the sensor unit.

Example 19
Example 17. The system of Example 17, wherein the process performed by the system comprises:
The system further comprising searching for an interventional medical device within the image starting from the position of the passive ultrasound sensor (340) and searching for regions within the image that satisfy predetermined constraints to apply image processing to the image. .

Example 20
Example 17. The system of Example 17, wherein the process performed by the system comprises:
Filtering (S340) the image to remove representations of the subject of the interventional medical procedure into which the interventional medical device is inserted, where elements of shape remain in the image after filtering (S340). ) and
reconstructing (S350) the shape based on the shape elements remaining in the image after filtering to obtain a reconstruction of the shape of the interventional medical device.

The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of the disclosure described herein. Many other embodiments will be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from this disclosure, resulting in structural and logical substitutions and modifications without departing from the scope of this disclosure. Additionally, the illustrations are representative only and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. As a result, the present disclosure and figures are to be considered illustrative rather than restrictive.

One or more embodiments of the present disclosure are disclosed herein merely for convenience and without the intention of voluntarily limiting the scope of this application to any particular invention or inventive concept: Individually and/or collectively referred to by the term "invention". Moreover, although specific embodiments are illustrated and described herein, subsequent constructions designed to accomplish the same or similar purposes may be substituted for the specific embodiments illustrated. It should be understood that This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein will be apparent to those of ordinary skill in the art upon reviewing the description.

An abstract of this disclosure is available at 37 C.F. F. R. It is provided to comply with § 1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Additionally, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. There is This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter is directed to less than all features of any of the disclosed embodiments. Thus, the following claims are incorporated into this Detailed Description, with each claim standing on its own as defining separately claimed subject matter.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. For that reason, the above-disclosed subject matter is to be considered illustrative rather than limiting, and the appended claims cover all such modifications, extensions, and modifications that fall within the true spirit and scope of this disclosure. and other embodiments. Accordingly, the scope of the disclosure, to the fullest extent permitted by law, is to be determined by the broadest allowable interpretation of the following claims and their equivalents, either limited or limited by the preceding detailed description. shall not be limited.

Claims (22)

A controller for determining a shape of the interventional medical device during an interventional medical procedure based on a position of the interventional medical device, the controller comprising:
a memory for storing instructions;
a processor that executes the instructions, and when executed on the processor, the instructions cause a system including the controller to:
obtaining a position of a point of the interventional medical device;
acquiring an image of a volume containing the interventional medical device;
applying image processing to the image based on the position of the interventional medical device to identify the interventional medical device including the shape of the interventional medical device;
segmenting the interventional medical device to obtain a segmented representation of the interventional medical device, wherein the segmented representation of the interventional medical device is superimposed on the image; and optionally,
outputting the image overlaid with the segmented representation for display to a user of the system, eg, an engineer, a doctor, or a patient. The process performed by the system includes:
Obtaining a position of a passive ultrasonic sensor based on emission of a beam from an ultrasonic imaging probe, wherein the position of the interventional medical device corresponds to the position of the passive ultrasonic sensor. When,
searching for the interventional medical device within the image starting from the position of the passive ultrasound sensor and searching for the region within the image that satisfies predetermined constraints to apply the image processing to the image; further comprising
2. The controller of claim 1, wherein the segmented representation of the interventional medical device is superimposed on the image together with the position of the passive ultrasound sensor. The process performed by the system includes:
further comprising imposing the predetermined constraint on the image to isolate regions within the image that are potentially considered to contain a portion of the shape of the interventional medical device;
3. The controller of claim 2, wherein the predetermined constraints include pixel intensity characteristics and pixel position relative to the position of the passive ultrasonic sensor. The process performed by the system includes:
further comprising imposing the predetermined constraint on the image to isolate regions within the image that are potentially considered to contain a portion of the shape of the interventional medical device;
3. The controller of claim 2, wherein the predetermined constraints include at least one predetermined shape to be used as the shape candidate for the interventional medical device. The process performed by the system includes:
further comprising imposing the predetermined constraint on the image to isolate regions within the image that are potentially considered to contain a portion of the shape of the interventional medical device;
3. The controller of claim 2, wherein said predetermined constraint comprises at least one dimensional property of a predetermined shape. The process performed by the system includes:
filtering the image to remove a representation of a subject of the interventional medical procedure into which the interventional medical device is inserted, wherein elements of the shape of the interventional medical device are present in the image after the filtering; remaining filtering and
Reconstructing the shape of the interventional medical device based on the elements of the shape of the interventional medical device remaining in the image after the filtering to obtain a reconstruction of the shape of the interventional medical device. 3. The controller of claim 1, further comprising: The segmentation is performed on the reconstruction of the shape of the interventional medical device, and the segmented representation of the interventional medical device is a segmented representation of the reconstruction of the shape of the interventional medical device. 7. The controller of claim 6, comprising: wherein said segmentation is performed into a predetermined shape based on said image processing, said segmented representation of said interventional medical device comprising a segmented representation of said predetermined shape of said interventional medical device; 2. The controller of claim 1, wherein as a result the segmented representation of the predetermined shape is superimposed on the image. A tangible, non-transitory, computer-readable storage medium storing a computer program that, when executed by a processor, provides a system containing the tangible, non-transitory computer-readable storage medium for interventional medical intervention during an interventional medical procedure. causing a process for determining a shape of a device to be performed based on the position of said interventional medical device, said process being performed when said processor executes said computer program from said tangible non-transitory computer-readable storage medium; ,
obtaining a position of the interventional medical device;
acquiring an image of a volume containing the interventional medical device;
applying image processing to the image based on the position of at least one point of the interventional medical device to identify the interventional medical device including the shape of the interventional medical device;
segmenting the interventional medical device to obtain a segmented representation of the interventional medical device, wherein the segmented representation of the interventional medical device is superimposed on the image; A tangible, non-transitory computer-readable storage medium, comprising: The process performed by the system includes:
Searching for said interventional medical device within said image starting from a position of a passive ultrasound sensor and searching for a region within said image that satisfies a predetermined constraint to apply said image processing to said image. 10. The tangible non-transitory computer-readable storage medium of claim 9. The process performed by the system includes:
obtaining a position of a passive ultrasonic sensor based on emission of a beam from an ultrasonic imaging probe, wherein the position of the interventional medical device corresponds to the position of the passive ultrasonic sensor; ,
applying the predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the shape of the interventional medical device;
the predetermined constraints include pixel intensity characteristics and pixel locations relative to the location of the passive ultrasonic sensor;
11. The tangible non-transitory computer-readable storage medium of claim 10, wherein the segmented representation of the interventional medical device is superimposed on the image along with the position of the passive ultrasound sensor. The process performed by the system includes:
further comprising applying the predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the shape of the interventional medical device;
11. The tangible non-transitory computer-readable storage medium of claim 10, wherein the predetermined constraints include at least one predetermined shape to be used as the shape candidate for the interventional medical device. The process performed by the system includes:
further comprising applying the predetermined constraint to the image to isolate regions within the image that are potentially considered to contain a portion of the shape of the interventional medical device;
11. The tangible non-transitory computer-readable storage medium of claim 10, wherein said predetermined constraint comprises at least one dimensional property of a predetermined shape. The process performed by the system includes:
filtering the image to remove a representation of a subject of the interventional medical procedure into which the interventional medical device is inserted, wherein the shape elements remain in the image after the filtering. and
Reconstructing the shape of the interventional medical device based on the elements of the shape of the interventional medical device remaining in the image after the filtering to obtain a reconstruction of the shape of the interventional medical device. 10. The tangible non-transitory computer-readable storage medium of claim 9, further comprising: The segmentation is performed on the reconstruction of the shape of the interventional medical device, and the segmented representation of the interventional medical device is a segmented representation of the reconstruction of the shape of the interventional medical device. 15. The tangible non-transitory computer-readable storage medium of claim 14, comprising: wherein said segmentation is performed into a predetermined shape based on said image processing, said segmented representation of said interventional medical device comprising a segmented representation of said predetermined shape of said interventional medical device; 10. The tangible non-transitory computer-readable storage medium of claim 9, wherein the segmented representation of the predetermined shape is superimposed on the image. 1. A system for determining the shape of an interventional medical device during an interventional medical procedure based on the position of a passive ultrasonic sensor located using an ultrasonic imaging probe, said system comprising:
an ultrasound imaging probe that emits a beam during the interventional medical procedure;
a passive ultrasonic sensor secured to the interventional medical device during the interventional medical procedure;
A controller comprising a memory storing instructions and a processor executing the instructions, the instructions, when executed by the processor, causing the system to:
obtaining the position of the passive ultrasonic sensor based on the emission of beams from the ultrasonic imaging probe;
acquiring an image of a volume containing the interventional medical device and the passive ultrasound sensor;
applying image processing to the image based on the position of the passive ultrasound sensor to identify the interventional medical device including the shape of the interventional medical device and the position of the interventional medical device; segmenting the interventional medical device to obtain a segmented representation of the medical device, wherein the segmented representation of the interventional medical device is combined with the position of the passive ultrasonic sensor; superimposed on said image, a controller performing a process comprising segmenting; and optionally:
a display for receiving the image overlaid with the segmented representation from the controller and displaying the image to a user of the system, such as an engineer, physician, or patient. a sensor unit for determining the position of the passive ultrasonic sensor and providing the position of the passive ultrasonic sensor to the controller;
a display displaying said segmented representation of said interventional medical device based on said segmentation by said controller and displaying said position of said passive ultrasound sensor determined by said sensor unit. 18. The system according to 17. The process performed by the system includes:
searching for the interventional medical device in the image starting from the position of the passive ultrasound sensor and searching the region in the image that satisfies predetermined constraints to apply the image processing to the image; 18. The system of claim 17, further comprising: The process performed by the system includes:
filtering the image to remove a representation of a subject of the interventional medical procedure into which the interventional medical device is inserted, wherein the shape elements remain in the image after the filtering. and
and reconstructing the shape based on elements of the shape remaining in the image after the filtering to obtain a reconstruction of the shape of the interventional medical device. System as described. A method for determining the geometry of an interventional medical device during an interventional medical procedure, the method comprising:
receiving or obtaining a position of a point of the interventional medical device;
receiving or acquiring an image of a volume containing the interventional medical device;
applying image processing to the image based on the position of at least one point of the interventional medical device to identify the interventional medical device including the shape of the interventional medical device;
segmenting the interventional medical device to obtain a segmented representation of the interventional medical device, wherein the segmented representation of the interventional medical device is superimposed on the image; step and, optionally,
and providing the image overlaid with the segmented representations to a user of the method, for example an engineer, a doctor, or a patient. 22. A computer program product comprising computer readable instructions which, when executed by a processor of a controller, causes the controller to perform the method of claim 21.

Images