Nothing Special   »   [go: up one dir, main page]

EP2088556A1 - Verfahren und System zur Registrierung zwecks Ausrichtung mindestens eines präoperativen Bildes an mindestens einem während einer Operation aufgenommenen Bild - Google Patents

Verfahren und System zur Registrierung zwecks Ausrichtung mindestens eines präoperativen Bildes an mindestens einem während einer Operation aufgenommenen Bild Download PDF

Info

Publication number
EP2088556A1
EP2088556A1 EP08002128A EP08002128A EP2088556A1 EP 2088556 A1 EP2088556 A1 EP 2088556A1 EP 08002128 A EP08002128 A EP 08002128A EP 08002128 A EP08002128 A EP 08002128A EP 2088556 A1 EP2088556 A1 EP 2088556A1
Authority
EP
European Patent Office
Prior art keywords
image
image data
preoperative
registration
intraoperative
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP08002128A
Other languages
English (en)
French (fr)
Other versions
EP2088556B1 (de
Inventor
Stefanie Demirci
Oliver Kutter
Nassir Prof. Navab
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens Healthcare GmbH
Original Assignee
Siemens AG
Siemens Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens AG, Siemens Corp filed Critical Siemens AG
Priority to EP08002128.0A priority Critical patent/EP2088556B1/de
Publication of EP2088556A1 publication Critical patent/EP2088556A1/de
Application granted granted Critical
Publication of EP2088556B1 publication Critical patent/EP2088556B1/de
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/38—Registration of image sequences
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
    • G06T7/32—Determination of transform parameters for the alignment of images, i.e. image registration using correlation-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10072—Tomographic images
    • G06T2207/10081—Computed x-ray tomography [CT]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30021—Catheter; Guide wire
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30101—Blood vessel; Artery; Vein; Vascular

Definitions

  • the present invention relates to a method and a system for registration to align at least one preoperative image to at least one intraoperatively acquired image.
  • CTA computed tomography angiography
  • a surgical instrument e.g. a pigtail catheter and guide wires, in particular a stent graft is placed inside the aorta.
  • the entire interventional catheter navigation is done under 2D angiography imaging where the physician is missing the important 3D information.
  • the image information available for the interventionalist has been improved.
  • image acquisitions cause very high radiation exposure and the produced images are of low quality and small size (30x40cm).
  • CTA Computerputed tomography angiography
  • Interventional fluoroscopy images however also display medical instruments such as catheters and stent shafts that cover large parts of the image information due to high focus settings of the X-Ray machine. These interventional instruments are not present within the CTA data and can thereby cause registration algorithms to obtain wrong results. Further dissimilarities can be produced by varying presence of contrasted blood vessels. As the CTA image is by specification contrasted, existing registration algorithms are not able to match an uncontrasted intraoperative fluoroscopy image to it.
  • a main aspect of the invention is an efficient method for image-based 2D-3D and 3D-3D registration that reduce the dissimilarities by image preprocessing, e.g. implicit detection and segmentation, and adaptive weights introduced into the registration procedure.
  • the inventive method for preferably image-based registration to align at least one preoperative image to at least one intraoperatively acquired image comprises the steps of:
  • said image dissimilarities can be caused by varying visible presence of surgical instruments and/or contrast agent.
  • Said intraoperatively acquired image can be represented by intraoperative three-dimensional image data.
  • Said intraoperatively acquired image can be represented by intraoperative two-dimensional (2D) image data.
  • step d) and e) can be simultanously performed.
  • FIG. 1 Further aspects of the invention are a system comprising software modules for performing the afore mentioned inventive method and a computer storage device comprising machine readable and executable program instructions which comprise steps of performing the afore mentioned inventive method.
  • the main advantage of the presented method is its robustness in terms of image dissimilarities which is achieved by image processing and appropriate registration algorithms.
  • Figure 1 In Figure 1 is shown: 1(a) shows a DRR image of an abdominal aortic aneurysm, 1(b) is generated with the same settings and enabled removal of the contrasted aorta. The medical instrument displayed inside the intraoperative 3D volume in 1(c) has been removed in 1(d).
  • Figure 2 generally depicts a schematic diagram for a 3D-3D registration solution.
  • Figure 3 schematically shows a Digitally Reconstruction Radiograph (DDR) process and
  • Figure 4 generally depicts a diagram for a 2D-3D registration solution
  • Figure 5 schematically shows an apparatus according to the invention.
  • DDR Digitally Reconstruction Radiograph
  • Figure 5 schematically shows an apparatus according to the invention which comprises a system for performing the in the following described inventive method.
  • FIG. 5 an apparatus, preferably a X-Ray apparatus, is shown.
  • a surgical instrument (not shown in the figure) can be navigated through a vascular structure of a body volume (region of interest) of a patient 7.
  • the system comprises a table 3 for supporting the patient and in the shown preferred embodiment it comprises a C-arm 4 with a X-ray source 5 and an image intensifier 6 for acquiring a series of images.
  • the image intensifier is connected with a processing system 10, preferably a computer, which can connect to a storage device 9 and a display unit 8.
  • Software modules for performing the inventive method can be implemented in the processing system.
  • the storage device can comprise an executable program of instructions to perform program steps for the inventive methods described in the following.
  • the algorithm iteratively optimizes the pose of a virtual X-Ray source by repeatedly generating DRR (Digitally Reconstructed Radiograph, see e.g. figure 3 ) images from the preoperative CTA while minimizing the image dissimilarity of the DRR and intraoperative X-Ray image (see result in the bottom right-handed picture of figure 4 ).
  • DRR Digitally Reconstructed Radiograph, see e.g. figure 3
  • DRR generation GPU Graphics Processing Unit
  • GLSL OpenGL Shading Language
  • the presented DRR renderer uses their scheme to determine the ray entry and exit points into the volume, however a single render pass to compute the X-Ray attenuation along each ray is employed.
  • a linear 1D transfer function is used implemented via dependent texture reads directly on the GPU [3].
  • a problem for image-based registration for aortic interventions is the presence of contrast in the preoperative CTA dataset and the presence of surgical instruments in the intraoperative X-Ray images.
  • DRR images generated from the CTA dataset contain a contrasted aorta, however intraoperative X-Ray images are not necessarily contrasted. This poses a problem for the image-based registrations as similarity metrics can hardly deal with such large dissimilarities.
  • a method to remove the contrast inside the aorta see top left-handed picture of figure 4 ) during DRR generation is implemented (see step 1 in figure 4 ). The idea is to do a lookup in a second volume, containing only the segmented aortic tree, before updating the attenuation along the ray. If the lookup yields a value greater than zero we scale the value interpolated from the original preoperative CTA dataset with a user set value.
  • the intraoperative X-Ray images contain all medical instruments that are inside the detector's field of view at acquisition time.
  • INK Insight Toolkit, Kitware [4]
  • figur 1(a) shows a DRR image of an abdominal aortic aneurysm
  • 1(b) is generated with the same settings and enabled removal of the contrasted aorta.
  • the medical instrument displayed inside the intraoperative 3D volume in 1(c) has been removed in 1(d).
  • the above described methods are embedded into a framework for optimizing the pose of the virtual X-Ray source by various image similarity metrics and optimization algorithms.
  • One of the most robust optimization algorithms is Nearest Neighbour and Powell-Brents Direction search method.
  • one of the most accurate similarity metric is normalized cross correlation [5].
  • the optimal pose calculation was further optimized by a good initial data/values that can be automatically extracted out of any mounted interventional angiography system.
  • the instruments are segmented in the interventional image by a connected threshold image filter provided by ITK (see top right-handed picture of figure 2 ).
  • I be the image volume of width w, height h and slices d consisting of voxels (x, y, z) and let S denote the set of segmented voxels.
  • ⁇ ( ⁇ 1 ,..., ⁇ w ) is the vector of knot points and N i , 3 , ⁇ are the basis functions that build up a certain B-Spline of degree 3.
  • C y and C z accordingly calculate the B-Spline interpolation in direction y and z.
  • the interpolated voxel value of V is then calculated by 1 3 ⁇ C x V + C y V
  • step 1 in figure 2 By applying the interpolation scheme (see step 1 in figure 2 ) to all segmented voxels all visible instruments are removed from the interventional volume (see bottom left-handed picture of figure 2 ) and the following 3D-3D registration procedure (see step 2 in figure 2 ) can be applied to align it to the CTA volume (see result in the bottom right-handed picture of figure 2 ).
  • normalized gradient correlation [5] is used as the most accurate similarity measure.
  • the optimization of the cost function is done by a best neighbour approach (also hill climbing) which is preferred to the Powell-Brent method because it is proved to be more independent of initial pose settings and therefore more robust.
  • the experiments is performed on an Intel Core 2 PC containing 2.66 GHz CPU and 4096 MB of main memory.
  • GeForce 8800 GTX (NVIDIA Corp.) GPU with 768 MB dedicated GDDR3 is used memory.
  • the evaluations of the proposed registration algorithms requires a selection of 2D and 3D datasets with varying presence of contrast agent and medical instruments. Each of them is then registered to the preoperative CTA volume by applying the appropriate registration algorithm.
  • CTA images are acquired by either Siemens Somatom Sensation 64 or Siemens Somatom Definition, interventional 2D and 3D X-Ray images are taken by Siemens AXIOM Artis dTA angiography suite with dynaCT software.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
EP08002128.0A 2008-02-05 2008-02-05 Verfahren und System zur Registrierung zwecks Ausrichtung mindestens eines präoperativen Bildes an mindestens einem während einer Operation aufgenommenen Bild Not-in-force EP2088556B1 (de)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP08002128.0A EP2088556B1 (de) 2008-02-05 2008-02-05 Verfahren und System zur Registrierung zwecks Ausrichtung mindestens eines präoperativen Bildes an mindestens einem während einer Operation aufgenommenen Bild

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP08002128.0A EP2088556B1 (de) 2008-02-05 2008-02-05 Verfahren und System zur Registrierung zwecks Ausrichtung mindestens eines präoperativen Bildes an mindestens einem während einer Operation aufgenommenen Bild

Publications (2)

Publication Number Publication Date
EP2088556A1 true EP2088556A1 (de) 2009-08-12
EP2088556B1 EP2088556B1 (de) 2017-02-01

Family

ID=40089881

Family Applications (1)

Application Number Title Priority Date Filing Date
EP08002128.0A Not-in-force EP2088556B1 (de) 2008-02-05 2008-02-05 Verfahren und System zur Registrierung zwecks Ausrichtung mindestens eines präoperativen Bildes an mindestens einem während einer Operation aufgenommenen Bild

Country Status (1)

Country Link
EP (1) EP2088556B1 (de)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102521832A (zh) * 2011-12-07 2012-06-27 中国科学院深圳先进技术研究院 图像分析方法和系统
CN102576457A (zh) * 2009-10-07 2012-07-11 直观外科手术操作公司 用于在临床图像上显示增强成像数据的方法和装置
CN104408734A (zh) * 2014-12-11 2015-03-11 山东师范大学 联合图像分割与形变配准技术的自适应目标区域转换方法
CN108765303A (zh) * 2018-04-08 2018-11-06 东南大学 一种数字减影血管成像图像的积分增强方法
JP2018187463A (ja) * 2011-10-05 2018-11-29 ニューヴェイジヴ，インコーポレイテッド 画像処理方法及び画像処理装置
EP3730055A1 (de) * 2019-04-22 2020-10-28 Canon Medical Systems Corporation Medizinische bildverarbeitungsvorrichtung, röntgendiagnosevorrichtung und computerimplementiertes verfahren
US11231787B2 (en) 2010-10-06 2022-01-25 Nuvasive, Inc. Imaging system and method for use in surgical and interventional medical procedures

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230154021A1 (en) * 2021-11-18 2023-05-18 Remex Medical Corp. Method and navigation system for registering two-dimensional image data set with three-dimensional image data set of body of interest

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020191735A1 (en) 2001-05-11 2002-12-19 Norbrrt Strobel Combined 3D angio-volume reconstruction method
DE10322738A1 (de) * 2003-05-20 2004-12-16 Siemens Ag Verfahren zur markerlosen automatischen Fusion von 2D-Fluoro-C-Bogen-Bildern mit präoperativen 3D-Bildern unter Verwendung eines intraoperativ gewonnenen 3D-Datensatzes
US20060002615A1 (en) * 2004-06-30 2006-01-05 Accuray, Inc. Image enhancement method and system for fiducial-less tracking of treatment targets
US20060188139A1 (en) 2004-12-20 2006-08-24 Ali Khamene Tree structure based 2D to 3D registration
US20060235287A1 (en) 2003-07-10 2006-10-19 Desmedt Paul A C Apparatus and method for navigating an instrument through an anatomical structure

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20020191735A1 (en) 2001-05-11 2002-12-19 Norbrrt Strobel Combined 3D angio-volume reconstruction method
DE10122875C1 (de) 2001-05-11 2003-02-13 Siemens Ag Kombiniertes 3D-Angio-Volumenrekonstruktionsverfahren
DE10322738A1 (de) * 2003-05-20 2004-12-16 Siemens Ag Verfahren zur markerlosen automatischen Fusion von 2D-Fluoro-C-Bogen-Bildern mit präoperativen 3D-Bildern unter Verwendung eines intraoperativ gewonnenen 3D-Datensatzes
US20060235287A1 (en) 2003-07-10 2006-10-19 Desmedt Paul A C Apparatus and method for navigating an instrument through an anatomical structure
US20060002615A1 (en) * 2004-06-30 2006-01-05 Accuray, Inc. Image enhancement method and system for fiducial-less tracking of treatment targets
US20060188139A1 (en) 2004-12-20 2006-08-24 Ali Khamene Tree structure based 2D to 3D registration

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
A. KHAMENE; P. BLOCH; W. WEIN; M. SVATOS; F. SAUER: "Automatic registration of portal images and volumetric CT for patient positioning in radiation therapy", MEDICAL IMAGE ANALYSIS, February 2006 (2006-02-01), pages 96 - 112, XP028013441, DOI: doi:10.1016/j.media.2005.06.002
G. P. PENNEY; J. WEESE; J. A. LITTLE; P. DESMEDT, D. L.; G. HILL; D. J. HAWKES: "A comparison of similarity measures for use in 2-d3-d medical image registration", IEEE TRANS. ON MEDICAL IMAGING (TMI), vol. 17, no. 4, 1998, pages 586 - 595, XP008118076, DOI: doi:10.1109/42.730403
J. KR"UGER; R. WESTERMANN: "Acceleration Techniques for GPU-based Volume Rendering", PROCEEDINGS IEEE VISUALIZATION 2003, 2003
L. IBANEZ; W. SCHROEDER; L. NG; J. CATES, THE ITK SOFTWARE GUIDE, 2005
MOURGUES F ET AL.: "3D reconstruction of the operating field for image overlay in 3D-endoscopic surgery", PROCEEDINGS IEEE AND ACM INTERNATIONAL SYMPOSIUM ON AUGMENTED REALITY IEEE COMPUT. SOC LOS ALAMITOS, CA, USA, 2001, pages 191 - 192, XP010568070, DOI: doi:10.1109/ISAR.2001.970537
MOURGUES F ET AL: "3D reconstruction of the operating field for image overlay in 3D-endoscopic surgery", PROCEEDINGS IEEE AND ACM INTERNATIONAL SYMPOSIUM ON AUGMENTED REALITY IEEE COMPUT. SOC LOS ALAMITOS, CA, USA, 2001, pages 191 - 192, XP010568070, ISBN: 0-7695-1375-1 *
O. KUTTER; S. KETTNER; E. U. BRAUN; N. NAVAB; R. LANGE; R. BAUERNSCHMITT: "Towards an integrated planning and navigation system for aortic stent-graft placement", PROC. OF COMPUTER ASSISTED RADIOLOGY AND SURGERY (CARS), June 2007 (2007-06-01)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102576457A (zh) * 2009-10-07 2012-07-11 直观外科手术操作公司 用于在临床图像上显示增强成像数据的方法和装置
CN102576457B (zh) * 2009-10-07 2015-11-25 直观外科手术操作公司 用于在临床图像上显示增强成像数据的方法和装置
US11231787B2 (en) 2010-10-06 2022-01-25 Nuvasive, Inc. Imaging system and method for use in surgical and interventional medical procedures
US11941179B2 (en) 2010-10-06 2024-03-26 Nuvasive, Inc. Imaging system and method for use in surgical and interventional medical procedures
JP2018187463A (ja) * 2011-10-05 2018-11-29 ニューヴェイジヴ，インコーポレイテッド 画像処理方法及び画像処理装置
JP2020110650A (ja) * 2011-10-05 2020-07-27 ニューヴェイジヴ，インコーポレイテッド 画像処理方法及び画像処理装置
CN102521832B (zh) * 2011-12-07 2014-05-28 中国科学院深圳先进技术研究院 图像分析方法和系统
CN102521832A (zh) * 2011-12-07 2012-06-27 中国科学院深圳先进技术研究院 图像分析方法和系统
CN104408734A (zh) * 2014-12-11 2015-03-11 山东师范大学 联合图像分割与形变配准技术的自适应目标区域转换方法
CN104408734B (zh) * 2014-12-11 2017-10-27 山东师范大学 联合图像分割与形变配准技术的自适应目标区域转换方法
CN108765303A (zh) * 2018-04-08 2018-11-06 东南大学 一种数字减影血管成像图像的积分增强方法
CN108765303B (zh) * 2018-04-08 2020-07-31 东南大学 一种数字减影血管成像图像的积分增强方法
EP3730055A1 (de) * 2019-04-22 2020-10-28 Canon Medical Systems Corporation Medizinische bildverarbeitungsvorrichtung, röntgendiagnosevorrichtung und computerimplementiertes verfahren
US11334998B2 (en) 2019-04-22 2022-05-17 Canon Medical Systems Corporation Medical image processing apparatus, X-ray diagnostic apparatus, and computer-implemented method

Also Published As

Publication number Publication date
EP2088556B1 (de) 2017-02-01

Similar Documents

Publication Publication Date Title
US8942455B2 (en) 2D/3D image registration method
EP2088556B1 (de) Verfahren und System zur Registrierung zwecks Ausrichtung mindestens eines präoperativen Bildes an mindestens einem während einer Operation aufgenommenen Bild
US7778686B2 (en) Method and apparatus for medical intervention procedure planning and location and navigation of an intervention tool
EP2800516B1 (de) Echtzeitdarstellung von gefässstrukturansichten für optimale vorrichtungsnavigation
EP2086415B1 (de) Kombination von röntgendaten mit intravaskulär gewonnenen daten
EP2349004B1 (de) Angiographisches bilderfassungssystem und verfahren mit automatischer blendenanpassung zum erhalt eines reduzierten sichtfelds zur abdeckung einer segmentierten zielstruktur oder läsion zur verringerung der röntgenstrahlendosis bei minimal invasiven röntgengeführten interventionen
US7853061B2 (en) System and method to improve visibility of an object in an imaged subject
US7346381B2 (en) Method and apparatus for medical intervention procedure planning
US8126239B2 (en) Registering 2D and 3D data using 3D ultrasound data
EP3258851B1 (de) Neukartierung von digitalbildern
US7912262B2 (en) Image processing system and method for registration of two-dimensional with three-dimensional volume data during interventional procedures
US8675996B2 (en) Catheter RF ablation using segmentation-based 2D-3D registration
CN104254874B (zh) 用于辅助2d‑3d图像配准的方法及系统
US20100061611A1 (en) Co-registration of coronary artery computed tomography and fluoroscopic sequence
US20090012390A1 (en) System and method to improve illustration of an object with respect to an imaged subject
US20190311505A1 (en) System And Method For Artifact Reduction In An Image
Wang et al. Dynamic 2-D/3-D rigid registration framework using point-to-plane correspondence model
JP2012115635A (ja) 画像処理方法、画像処理装置、画像システム及びプログラムコード
Demirci et al. Advanced 2D-3D registration for endovascular aortic interventions: addressing dissimilarity in images
Wagner et al. Continuous-sweep limited angle fluoroscopy guidance for percutaneous needle procedures
Baert et al. Precalibration versus 2D-3D registration for 3D guide wire display in endovascular interventions
Hoffmann et al. Contrast‐Based 3D/2D Registration of the Left Atrium: Fast versus Consistent
Imamura et al. Registration of preoperative CTA and intraoperative fluoroscopic image sequence for assisting endovascular stent grafting
Demirci et al. Towards an Computer Assisted Intervention System for Endovascular Abdominal Aortic Repair
Baert et al. 3D guide wire reconstruction from biplane image sequences for 3D navigation in endovascular interventions

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA MK RS

17P Request for examination filed

Effective date: 20100208

17Q First examination report despatched

Effective date: 20100226

AKX Designation fees paid

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: SIEMENS AKTIENGESELLSCHAFT

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: SIEMENS HEALTHCARE GMBH

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20160902

RIN1 Information on inventor provided before grant (corrected)

Inventor name: KUTTER, OLIVER

Inventor name: NAVAB, NASSIR

Inventor name: DEMIRCI, STEFANIE

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 866114

Country of ref document: AT

Kind code of ref document: T

Effective date: 20170215

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 10

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602008048635

Country of ref document: DE

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20170201

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 866114

Country of ref document: AT

Kind code of ref document: T

Effective date: 20170201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170501

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170502

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170601

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170501

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170601

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170228

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170228

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602008048635

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170205

26N No opposition filed

Effective date: 20171103

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 11

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20170228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170205

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170228

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170205

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20080205

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20190416

Year of fee payment: 12

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20170201

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20170201

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20200206

Year of fee payment: 13

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20200211

Year of fee payment: 13

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602008048635

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200901

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20210205

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210228

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210205