JP3790906B2 - Imaging device - Google Patents

Description

【００２８】
ただし、ｇ_i ^j ：ｊ番目の波長バンドパスフィルタを用いて入力されたｉ番目の受光エレメントの観測値、ｔ^j （λ）：ｊ番目の波長バンドパスフィルタの透過率分布特性、ｈ_i （ｒ，λ）：ｉ番目の受光エレメントにおける物体空間ｒおよび波長λの両方に対して定義される感度分布関数、ｆ（ｒ，λ）：被写体を表す物体空間ｒおよび波長λの両方に対して定義される原画像関数である。
【００２９】
なお、感度分布関数ｈ_i （ｒ，λ）は、前記した特開平６−９８１７０号公報および特開平７−１６０８４７号公報に開示されている感度分布関数を拡張して定義したもので、撮像素子のｉ番目の受光エレメントにおける物体空間および光の波長に対する光強度の感度分布を表す関数である。感度分布関数ｈ_i （ｒ，λ）は特開平６−９８１７０号公報の図１に示すような構成によって実際に計測しても良いし、光学結像系の解析に基づいて計算により求めても良い。
【００３０】
（１）式の定義において原画像ｆ（ｒ，λ）は連続系で定義され、観測値
ｇ_i ^j は離散値で定義される。そして両者を関係付けるために波長バンドフィルタの透過率分布特性ｔ^j （λ）および感度分布関数ｈ_i （ｒ，λ）が定義される。一方、原画像ｆ（ｒ，λ）をＲＧＢカラー画像に変換する過程は次式で表される。
【００３１】
【数２】

【００３２】
ただし、ｆ^k （ｒ）：ｋ番目の原色画像（ｋ＝Ｒ，Ｇ，Ｂ）、ｓ^k （λ）：ｋ番目の色の等色関数で、図４に示すような波長特性を有する。また現実の装置でカラー画像を表示するためには連続系で定義される原色画像ｆ^k （ｒ）を再サンプリングする必要があるので、その過程を次式で定義する。
【００３３】
【数３】

ただし、ｆ_i ^k ：ｋ番目の原色画像のｉ番目の画素値、ｐ_i （ｒ）：ｉ番目の画素値を求めるためのサンプリング関数である。（１）式で表される画像入力系により入力された複数の波長バンド画像から（２）式で表される各原色画像を推定する過程を、前記した特開平６−９８１７０号公報および特開平７−１６０８４７号公報に開示されているような連続−離散系モデルに対して一般的に拡張定義された擬似逆行列の考え方に従って定式化すると、以下のように表される。なお、繁雑さを避けるため、以下の説明では途中の導出に関する記述を省略し、最終的な結果のみを示す。
【００３４】
【数４】

【００３５】
【数５】

【００３６】
（６）式において、連続−離散変換線形オペレータ｛ｈ（ｒ，λ）｝は次式に示すように受光エレメント数と等しいＭ個の感度分布関数ｈ_i （ｒ，λ）を列方向に並べて構成されるオペレータである。
【００３７】
【数６】

【００３８】
結局、オペレータ｛ａ^k （ｒ）｝^t は行方向にＮ×Ｍ個の関数を並べて構成される。一方、行列Ｂ^a は行列Ｂの擬似逆行列であるが、行列Ｂは次のようにＮ×Ｎ個のブロック行列Ｂ_mn（ｍ，ｎ＝１，２，…，Ｎ）で構成される。
【００３９】
【数７】

【００４０】
（４）式に従って求められる連続空間で定義される推定原色画像は、次に（３）式に基づいて離散値に変換される。そこで離散原色画像をベクトルｆ^k ＝［ｆ₁ ^k ｆ₂ ^k …ｆ_M' ^k ］^t で定義すると、観測ベクトルｇからベクトルｆ^k を推定する過程は（１０）式にまとめられる。なお、Ｍ′は離散原色画像の画素数であり、実用的には受光エレメント数Ｍとほぼ同等に決めれば良い。
【００４１】
【数８】

【００４２】
また、（１０）式において行列Ａ^k はオペレータ｛ｐ（ｒ）｝とオペレータ｛ａ^k （ｒ）｝^t とを線形演算することにより改めて定義されるＭ′×ＭＮ行列であり、行列Ｃ^k は行列Ａ^k と行列Ｂ^a との行列積を計算することにより定義されるＭ′×ＭＮ行列である。つまり、ＲＯＭ２０５に予め記録される係数列は（１１）式で表される行列Ｃ^k の各要素であれば良く、ｋ＝Ｒ，Ｇ，Ｂの各々について定義されるそれらの係数列を全て記録しておけば良いことになる。
【００４３】
上記した第１実施形態によれば、多波長バンド光学結像を用いた連続−離散画像入力系の定義に基づいた復元処理を行なうことにより正確なカラー画像が復元可能であるため、レンズなどの光学結像系を簡便に設計することができ、小型軽量のカメラを用いながら解像度や色再現性に優れたカラー画像を得ることができる。
【００４４】
以下に本発明の第２実施形態を説明する。本発明の第２実施形態は、あらかじめ予測される被写体の統計的自己相関情報が記録されたカードを用いて、より高性能なカラー画像を復元する方法に関する。図５に第２実施形態における画像処理プロセッサ４００の構成を示す。なおその他のカメラとＴＶモニタは第１実施形態におけるカメラ１００およびＴＶモニタ３００と同様なので、図示および説明は省略する。
【００４５】
図５において、Ａ／Ｄ変換器４０１、画像メモリ４０２−１〜Ｎ、回転色フィルタドライバ４０３、行列演算器４０４、画像メモリ４０６−１〜３、Ｄ／Ａ変換器４０７、ＣＰＵ４１０は、それぞれ図１におけるＡ／Ｄ変換器２０１、画像メモリ２０２−１〜Ｎ、回転色フィルタドライバ２０３、行列演算器２０４、画像メモリ２０６−１〜３、Ｄ／Ａ変換器２０７、ＣＰＵ２１０と同様である。なお、図５では繁雑さを避けるためＣＰＵ４１０と他の構成要素との結線を省略してある。
【００４６】
第２実施形態では、これらの他に目的や用途に応じて着脱可能なカード４２０とカードリーダー４２１が用意される。第２の係数メモリとしてのカード４２０には半導体ＩＣ、光メモリなどの記録媒体が内蔵されており、対象とする被写体の自己相関行列に関する情報（統計的相関情報）が記録されている。この自己相関行列の係数値は、第１の係数メモリとしてのＲＯＭ４２２内に記録されている係数値と共に係数列演算手段としての行列演算器４２３に送られることにより所定の行列演算が行なわれ、算出された２種類の新たな係数値が係数メモリ４２４−１〜２に記録される。
【００４７】
行列演算手段としての行列演算器４０４では、画像メモリ４０２−１〜Ｎに記録されている多波長バンド画像と係数メモリ４２４−１〜２に記録されている係数値との間で所定の行列演算が行われ、復元されたＲＧＢカラー画像が画像メモリ４０６−１〜３に記録される。
【００４８】
以下に、上記した構成を有する第２実施形態の作用を説明する。第２実施形態は、以下に説明するように最小二乗規範に基づいて連続−離散系画像入力モデルを再定式化することにより被写体の原画像に対して最も誤差の少ない画像を推定復元する方法に関する。それには先験的に予測される原画像の統計的自己相関情報が必要となるため、図５の装置は被写体の種類、例えば人物、風景、医療、建造物画像などに応じて適当な自己相関情報が記録されたカード４２０を選択できるように構成される。
【００４９】
このような方法は、色収差の影響のため波長により結像状態が異なるような場合においても、波長バンド画像間の相関を利用して復元されるカラー画像の解像度や色再現性を向上させる作用を有する。例えば色収差の影響の大きい光学結像系では、緑の波長領域に対して最も焦点の合った状態で結像される際には赤や青の波長領域は焦点はずれの状態となる。従って各波長バンド画像ごとに復元を行なおうとすると、緑の画像に比べ赤や青の画像はぼけを回復しきれないため、結果的に復元されるカラー画像は色バンド画像間で解像度のバランスが悪く色再現性に劣ったものになってしまう。ところが第２実施形態の方法によれば、焦点の合った状態で入力できる緑の画像と赤あるいは青の画像との相関情報を活用することにより赤や青の画像における結像特性の劣化を補正し、緑と同等な画質にまで復元する作用が有る。
【００５０】
以下に最小二乗規範に基づいた画像入力系の再定式化を行ない、カード４２０に記録される自己相関情報の内容と具体的な計算方法について示す。なお、以下の説明では離散−連続変換オペレータによる偏微分などの計算式が現れるが、全ての関係式は線形であり行列とベクトルで構成される離散系における定義を拡張することにより定義される。つまり、連続系は要素数が無限個の離散系であると考えて扱えば良い。
【００５１】
【数９】

【００５２】
ただし、オペレータＥ［・］は集合平均を表す。平均二乗誤差ｅ^k を最小にするためのオペレータ｛ｍ^k （ｒ）｝^t は（１８）式を｛ｍ^k （ｒ）｝で偏微分した値が０になるような次の方程式により求められる。
【００５３】
【数１０】

【００５４】
（１４）式を解くことにより最終的に得られるオペレータ｛ｍ^k （ｒ）｝^t は次式のように表される。
【００５５】
【数１１】

【００５６】
ただし、離散−連続変換オペレータ｛ｄ^k （ｒ）｝^t は以下のように定義される。
【００５７】
【数１２】

【００５８】
ただし、ｊ番目の要素関数［ｄ^k （ｒ）］^j は次式で定義される。
【００５９】
【数１３】

【００６０】
ここで、Ｒ_ff（ｒ，λ：ｒ′，λ′）は原画像ｆ（ｒ，λ）の自己相関関数であり、次式で定義される。
【００６１】

（１６），（１７）式を（５），（６）式と比較すると、最小二乗規範を導入した第２実施形態において定義されるオペレータ｛ｄ^k （ｒ）｝^t の各要素関数［ｄ^k （ｒ）］^j は、第１実施形態で定義されたオペレータ｛ａ^k （ｒ）｝^t における各要素関数［ａ^k （ｒ）］^j の積分項に原画像ｆ（ｒ，λ）の自己相関関数Ｒ_ff（ｒ，λ：ｒ′，λ′）を挿入させたものであることが分かる。
【００６２】
一方、（１５）式における行列Ｑ^a は次のように定義される行列Ｑの擬似逆行列である。
【００６３】
【数１４】

【００６４】
（１９），（２０）式を（８），（９）式と比較すると、第２実施形態において定義される行列Ｑの各ブロック行列Ｑ_mnは、第１実施形態で定義された行列Ｂの各ブロック行列Ｂ_mnにおける積分項に原画像ｆ（ｒ，λ）の自己相関関数
Ｒ_ff（ｒ，λ：ｒ′，λ′）を挿入したものであることが分かる。最後に（１２）式に従って求められる連続空間で定義される推定原色画像は（３）式に基づいて離散変換されるが、
【数１５】

【００６５】
ただし、（２１）式において行列Ｄ^k はオペレータ｛ｐ（ｒ）｝とオペレータ｛ｄ^k （ｒ）｝^t とを線形演算することにより改めて定義されるＭ′×ＭＮ行列である。つまり、カード４２０には対象とする被写体の性質に最も近いと予測される自己相関Ｒ_ff（ｒ，λ：ｒ′，λ′）を記録しておけば良く、ＲＯＭ４２２には各波長バンドパスフィルタの透過率分布特性ｔ^j （λ）、物体空間ｒおよび波長λの両方に対して定義される感度分布関数ｈ_i （ｒ，λ）、ｋ番目の色の等色関数ｓ^k （λ）、サンプリング関数ｐ_i （ｒ）を記録しておけば良い。なお、実際の装置では自己相関Ｒ_ff（ｒ，λ；ｒ′，λ′）などの連続関数や感度分布関数ｈ_i （ｒ，λ）のような離散系と連続系を変換するオペレータを厳密に記録することは困難であるが、離散系のサンプリング数（ＭやＭ′）に比べて多いサンプリング数になるように細かくピッチを設定した擬似的な連続空間を定義すれば実用上問題はなく、本発明の主旨に沿った処理が実現できる。
【００６６】
カード４２０およびＲＯＭ４２２に記録されているデータは行列演算器４２３において（２１）式で定義された行列Ｄ^k ，Ｑ^a を算出するのに利用され、結果はそれぞれ係数メモリ４２４−１，２に記録される。そして行列演算器４０４では画像メモリ４０２−１〜Ｎに記録されている各波長バンド画像と係数メモリ４２４−１〜２に記録された行列係数との間で（２１）式に従った計算が行なわれ、ＲＧＢ原色画像が復元される。
【００６７】
なお、第２実施形態ではカード４２０に画像の自己相関情報を記録するように構成したが、画像入力系の条件があらかじめ固定されるような場合に対しては最終的に必要となる行列Ｄ^k ，Ｑ^a の係数をカード４２０に記録しても良い。その場合、ＲＯＭ４２２、行列演算器４２３および係数メモリ４２４−１〜２が省略され、構成が簡便になる。また、特定の被写体を対象とする場合や広い被写体の種類に対して共通に設定可能な自己相関関数が定義される場合には、第１実施形態と同様に、最終的に算出される行列Ｄ^k Ｑ^a の要素を１個のＲＯＭに記録するように構成しても良い。
【００６８】
上記した第２実施形態によれば、色収差などの影響による原色画像間の解像度のムラを補正できるため、レンズなどの光学結像系を簡便にしながらも、より解像度や色再現に優れたカラー画像を得ることができる。また適当な自己相関情報が記録されたカードを選択できることから、被写体の種類に対して適応的に復元画像の画質を向上させることができる。
【００６９】
以下に、本発明の第３実施形態を説明する。本発明の第３実施形態は、モザイク色フィルタが装着された単板カラーカメラを用いて入力された画像からＲＧＢカラー画像を復元する方法に関する。図６に第３実施形態におけるカメラ５００と画像処理プロセッサ６００の構成を示す。カメラ５００内にはレンズ５０１で構成される光学結像系が設けられており、このレンズ５０１を介して被写体の光像がＣＣＤ撮像素子５０２の受光面上に結像される。ＣＣＤ撮像素子５０２は単板型のカラー撮像素子であり、その撮像面上にモザイク色フィルタ５０５が装着されている。モザイク色フィルタ５０５は、例えば図７に示すような緑市松Ｒ／Ｂ線順次方式あるいはベイヤー方式と呼ばれる色配置に従って色フィルタが設けられており、ＣＣＤ撮像素子５０２では画素順次方式によりカラー画像が撮像される。
【００７０】
ＣＣＤ撮像素子５０２からの出力画像信号は画像処理プロセッサ６００内のＡ／Ｄ変換器６０１によりディジタル変換され、画像メモリ６０２に記録される。次に画像メモリ６０２に記録されたディジタル画像信号はＲＯＭ６０５に記憶されている係数列と共に行列演算器６０４に送られ、所定の行列演算が行なわれることによりＲＧＢカラー画像が復元され、その各原色画像は画像メモリ６０６−１〜３にそれぞれ記録される。画像メモリ６０６−１〜３に記録されたＲＧＢ画像はＤ／Ａ変換器６０７により適当なアナログビデオ信号に変換され、図示しないＴＶモニタに表示される。
【００７１】
以下に、上記した構成を有する第３実施形態の作用を説明する。第３実施形態は、単板カラー方式のカメラ５００で入力されたＲ，Ｇ，Ｂの３バンド波長領域の画像から、第１あるいは第２実施形態に示したような連続−離散系モデルに基づく画像復元の原理に従って解像度や色再現性に優れたカラー画像を得る作用を有する。モザイク色フィルタ５０５を介して入力された画素順次方式のカラー画像は、欠落画素を有する原色画像の集まりであると考えられる。つまり、図７に示すようなモザイク色配置によれば、例えばＲ画像に着目するとＧ，Ｂのフィルタが配置されている画素は欠落画素であると考えられる。第３実施形態はそのような欠落画素を含む原色画像から欠落の無い完全な原色画像を復元する作用を有する。復元処理はＲ，Ｇ，Ｂの３色バンド領域における連続−離散系画像入力モデルに基づいて定義されるが、結局は第１，２実施形態と同様に行列演算により実現される。
【００７２】
以下に第２実施形態と同様な最小二乗規範に基づいた具体的な計算方法について示す。まずＣＣＤ撮像素子５０２を構成するＭ個の受光エレメントの内の１つ（ｉ番目；ｉ＝１，…，Ｍ）に着目し、Ｒ，Ｇ，Ｂの色バンドの内の１つ（ｊ番目；ｊ＝Ｒ，Ｇ，Ｂ）で離散値ｇ_i ^j が観測される画像入力系を次のように定式化する。
【００７３】
【数１６】

【００７４】
ただし、ｈ_i ^j （ｒ）：ｉ番目の受光エレメントのｊ番目の色バンド領域において物体空間ｒに対して定義される感度分布関数、ｆ^j （ｒ）：ｊ番目の色バンド領域における原画像関数である。また、Ｒ，Ｇ，Ｂ各画素数をそれぞれＭ_R ，
Ｍ_G ，Ｍ_B とするとＭ_R ＋Ｍ_G ＋Ｍ_B ＝Ｍである。
【００７５】
【数１７】

【００７６】
【数１８】

【００７７】
（２５）式を解くことにより最終的に得られるオペレータ｛ｎ^k （ｒ）｝^t は次式のように表される。
【００７８】
【数１９】

【００７９】
ただし、離散−連続変換オペレータ｛ｕ^k （ｒ）｝^t は以下のように定義される。
【００８０】
【数２０】

【００８１】
（２７）式におけるｊ番目の要素関数［ｕ^k （ｒ）］^j は次式で定義される。
【００８２】
【数２１】

【００８３】
ただし、Ｒ_kj（ｒ；ｒ′）は原画像ｆ^k （ｒ）と原画像ｆ^j （ｒ′）の相互相関関数であり、次式で表される。
【００８４】
【数２２】

【００８５】
また、（２８）式における連続−離散変換オペレータ｛ｈ^j （ｒ）｝は次式に表されるように、ｊ番目の色のフィルタが装着されたＭ_i 個の各画素に対する感度分布関数を列方向に並べることにより定義される。
【００８６】
【数２３】

【００８７】
一方、（２６）式における行列Ｖ^a は次のように定義される行列Ｖの擬似逆行列である。
【００８８】
【数２４】

【００８９】
ただし、ブロック行列Ｖ_kjは次式で定義され、各々Ｍ_k ×Ｍ_j 個の要素を持つ。
【数２５】

【００９０】
最後に（２３）式に従って求められる連続空間で定義される推定原色画像は（３）式に基づいて離散変換されるが、
【数２６】

【００９１】
ただし、（３３）式において行列Ｕ^k はオペレータ｛ｐ（ｒ）｝とオペレータ｛ｕ^k （ｒ）｝^t とを線形演算することにより改めて定義されるＭ′×Ｍ行列である。また、行列Ｘ^k は行列Ｕ^k と行列Ｖ^a との行列積を計算することにより改めて定義されるＭ′×Ｍ行列である。
【００９２】
【数２７】

【００９３】
なお、第３実施形態においても前記した第２実施形態のように、原色画像間の相関情報を記録したカードを脱着可能にすることにより、被写体に対して適応的に復元処理を行なわせることができる。また、第３実施形態は、第２実施形態のみならず、前記した第１実施形態、さらには以下に述べる第４、第５実施形態に適用することも可能である。
【００９４】
上記した第３実施形態によれば、単板カメラを用いながら解像度や色再現性に優れたカラー画像を得ることができるため、コンパクトでしかも画像入力時間の短い実用上有用なカラー画像撮像装置を提供することができる。
【００９５】
以下に、本発明の第４実施形態を説明する。本発明の第４実施形態は、撮像素子のＳ／Ｎに関するデータをあらかじめカメラ内のメモリに記録しておき、そのデータに基づいて適応的にノイズが抑制されたカラー画像を復元する方法に関する。図８に第４実施形態におけるカメラ７００と画像処理プロセッサ８００の構成を示す。カメラ７００において、レンズ７０１、ＣＣＤ撮像素子７０２、回転色フィルタ７０３、モータ７０４はそれぞれ第１実施形態のカメラ１００におけるレンズ１０１、ＣＣＤ撮像素子１０２、回転色フィルタ１０３、モータ１０４と同様である。カメラ７００にはそれらの他に撮像素子７０２のＳ／Ｎに関するデータが記録された第２の係数メモリとしてのＲＯＭ７０６が内蔵されている。
【００９６】
また画像処理プロセッサ８００において、Ａ／Ｄ変換器８０１、画像メモリ８０２−１〜Ｎ、回転色フィルタドライバ８０３、行列演算器８０４、画像メモリ８０６−１〜３、Ｄ／Ａ変換器８０７、ＣＰＵ８１０は、それぞれ第１実施形態の画像処理プロセッサ２００におけるＡ／Ｄ変換器２０１、画像メモリ２０２−１〜Ｎ、回転色フィルタドライバ２０３、行列演算器２０４、画像メモリ２０６−１〜３、Ｄ／Ａ変換器２０７、ＣＰＵ２１０と同様である。
【００９７】
画像処理プロセッサ８００にはさらに第１の係数メモリとしてのＲＯＭ８３０が内蔵され、このＲＯＭ８３０に記録されている係数列はカメラ７００内のＲＯＭ７０６から送られてきたデータと共に係数列演算手段としての行列演算器８３１に送られ、所定の行列演算が行なわれて算出された係数列は係数メモリ８３２−１、２に記録される。そして、これらの係数列と画像メモリ８０２−１〜Ｎに記録されている多波長バンド画像との間で行列演算が行なわれることによりＲＧＢ原色画像が復元され、結果は画像メモリ８０６−１〜３に記録される。
【００９８】
以下に上記した構成を有する第４実施形態の作用を説明する。第４実施形態は、ＣＣＤ撮像素子７０２の熱雑音などにより入力画像に混入するノイズの影響を抑制しながらカラー画像を復元する作用を有する。カメラ７００内にカメラ７００のＳ／Ｎ特性を示すパラメータを記録したＲＯＭ７０６を設けることにより、カメラ７００に対して適応的なノイズ抑制効果をもたらす。
【００９９】
以下に、Ｓ／Ｎを考慮したカラー画像の復元処理について説明する。加法的離散ノイズの加わった観測ベクトルｇ′は次式で表される。
【０１００】
ｇ′＝ｇ＋ｎ （34）
ただし、ｇ：ノイズの加わっていない観測ベクトル、ｎ：ノイズベクトルである。この観測ベクトルｇ′から連続空間で定義される原色画像を推定する過程は、第２実施形態と同様な最小二乗規範に従った解析により、以下の（３５）式のように導出される。ただし、式の導出において原画像とノイズには相関が無いと仮定した。
【０１０１】
【数２８】

【０１０２】
（３５）式において離散−連続変換オペレータ｛ｄ^k （ｒ）｝^t は（１６）式に定義されたオペレータと同様であり、行列Ｑは（１９），（２０）式と同様に定義される。また行列Ｎはノイズの自己相関行列であり、ＣＣＤ撮像素子７０２の各受光エレメントに生じるノイズ同士は無相関で、発生確率は各受光エレメント間で等しいとすると次式で表すことができる。
【０１０３】
【数２９】

【０１０４】
ただし、α：ノイズの分散を表す定数パラメータ、Ｉ：ＭＮ×ＭＮの単位行列である。以上の解析により、
【数３０】

【０１０５】
ただし、行列Ｄ^k は（２１）式に定義した行列Ｄ^k と同様であり、行列Ｑは（１９），（２０）式に定義される行列Ｑと同様である。ＲＯＭ８３０内には感度分布関数ｈ_i （ｒ，λ），（ｉ＝１，…，Ｍ）と、波長バンドパスフィルタの透過率分布特性ｔ^j （λ），（ｊ＝１，…，Ｎ）と、原色画像の等色関数ｓ^k （λ），（ｋ＝Ｒ，Ｇ，Ｂ）および原画像ｆ（ｒ，λ）の自己相関関数Ｒ_ff（ｒ，λ；ｒ′，λ′）が記録される。またカメラ７００内のＲＯＭ７０６にはパラメータαが記録される。ＲＯＭ８３０，７０６に記録されているこれらのデータが行列演算器８３１に入力されることにより行列Ｄ^k および（Ｑ＋αＩ）^a が算出され、その要素がそれぞれ係数メモリ８３２−１，２に記録される。行列演算器８０４では、ＲＯＭ８３２−１，２内に記録されている行列Ｄ^k と（Ｑ＋αＩ）^a を用いて画像メモリ８０２−１〜Ｎに記録されている各波長バンド画像との間で（３７）式に基づく原色画像の復元処理が行なわれる。
【０１０６】
上記した第４実施形態によれば、入力画像に混入するノイズの影響が抑制された明瞭なカラー画像を出力することができる。またカメラ自体にカメラの性能に依存したＳ／Ｎパラメータを保持させるので、カメラを画像処理プロセッサに対して着脱可能な構成とした場合は、使用するカメラに適応させた最適なノイズ抑制処理が実現できる。
【０１０７】
以下に、本発明の第５実施形態を説明する。本発明の第５実施形態は、光学結像系の色収差を利用して波長ごとに異なる物体面に焦点の合った多波長バンド画像を入力し、それらを用いて任意の物体面に焦点の合ったカラー画像を復元する方法に関する。
【０１０８】
図９に第５実施形態におけるカメラ９００と画像処理プロセッサ１０００の構成を示す。カメラ９００において、レンズ９０１、ＣＣＤ撮像素子９０２、回転色フィルタ９０３、モータ９０４はそれぞれ第１実施形態のカメラ１００におけるレンズ１０１、ＣＣＤ撮像素子１０２、回転色フィルタ１０３、モータ１０４と同様である。ただし、第５実施形態のレンズ９０１は色収差が大きく、光学結像系は波長に応じて合焦範囲に広がりがあるとする。また画像処理プロセッサ１０００において、Ａ／Ｄ変換器１００１、画像メモリ１００２−１〜Ｎ、回転色フィルタドライバ１００３、行列演算器１００４、画像メモリ１００６−１〜３、Ｄ／Ａ変換器１００７、ＣＰＵ１０１０は、それぞれ第１実施形態の画像処理プロセッサ２００におけるＡ／Ｄ変換器２０１、画像メモリ２０２−１〜Ｎ、回転色フィルタドライバ２０３、行列演算器２０４、画像メモリ２０６−１〜３、Ｄ／Ａ変換器２０７、ＣＰＵ２１０と同様である。
【０１０９】
画像処理プロセッサ１０００にはさらにＲＯＭ１０４０が内蔵され、複数の被写体の種類に対応した自己相関情報が記録されている。またＲＯＭ１０４１には複数の物体面の位置に対応した感度分布関数が記録されている。これらのＲＯＭ１０４０，１０４１に記録されている複数の係数列の内の１つが選択されて行列演算器１０４３に入力されるが、どの係数列を選択するかはユーザインターフェース１０５０によりユーザから指定された条件に基づいてＣＰＵ１０１０からＲＯＭ１０４０，１０４１に送られる読み出しアドレス信号により制御される。
【０１１０】
また、ＲＯＭ１０４２には３次元物体空間に対する感度分布関数、回転色フィルタ９０３に設けられているＮ枚の色バンドフィルタの透過率分布特性およびＲＧＢ等色関数が記録されており、それらの係数列は行列演算器１０４３に入力される。行列演算器１０４３ではＲＯＭ１０４０，１０４１，１０４２から送られてきた係数列を用いて所定の行列演算が行なわれ、算出された２種類の新たな係数値が係数メモリ１０４４−１〜２に記録される。行列演算器１００４では、画像メモリ１００２−１〜Ｎに記録されている多波長バンド画像と係数メモリ１０４４−１〜２に記録されている係数値との間で所定の行列演算が行なわれ、復元されたＲＧＢカラー画像が画像メモリ１００６−１〜３に記録される。
【０１１１】
以下に上記した構成を有する第５実施形態の作用を説明する。第５実施形態は、色収差の大きい光学結像系を用いた場合は波長ごとに異なる物体面に焦点が合うことを利用して、レンズの位置を固定させながら波長と焦点位置とが共に異なる複数の画像を入力する。それらを用いて、全入力画像に対応した合焦範囲内における任意の物体面の位置に焦点の合ったカラー画像を復元する作用を有する。そのような処理を実現させるために、以下に説明するような３次元物体空間に対する多波長バンドの連続−離散系モデルに基づき、各波長バンド画像間の相関を利用したカラー画像復元法を用いる。
【０１１２】
以下に具体的な計算方法について示す。まず（１）式と同様に３次元物体空間に対する連続−離散画像入力系を次式で定義する。
【０１１３】
【数３１】

【０１１４】
上式において、空間座標Ｒ＝（ｘ，ｙ，ｚ）を３次元物体空間を表す直交座標系とし、感度分布関数ｈ_i （Ｒ，λ）と原画像ｆ（Ｒ，λ）とを３次元物体空間に対して再定義する。ただし、ｚを光学結像系の光軸と平行な軸に定義する。
【０１１５】
【数３２】

【０１１６】
ただし、離散−連続変換オペレータ｛ｗ^k （ｒ，ｚ₀ ）｝^t は以下のように定義される。
【０１１７】
【数３３】

【０１１８】
ただし、ｊ番目の要素関数［ｗ^k （ｒ，ｚ₀ ）］^j は次式で定義される。
【０１１９】
【数３４】

【０１２０】
ただし、｛ｈ（ｒ，ｚ₀ ；λ）｝はｚ＝ｚ₀ の２次元物体面における感度分布関数ｈ_i （ｒ，ｚ₀ ；λ），（ｉ＝１，…，Ｍ）を列方向に並べることにより構成される連続−離散系変換オペレータであり、Ｒ_ff（ｒ，λ；ｒ′，λ′；ｚ₀ ）はｚ＝ｚ₀ における２次元原画像ｆ（ｒ，λ；ｚ₀ ）の自己相関関数である。一方、（３９）式における行列Ｚ^a は行列Ｚの擬似逆行列であるが、行列ＺのＮ×Ｎ個のブロック要素行列Ｚ_mn，（ｍ，ｎ＝１，…，Ｎ）は次式で定義され、各々Ｍ×Ｍ個の要素を持つ。
【０１２１】
【数３５】

【０１２２】
ただし、Ｒ_ff（Ｒ，λ；Ｒ′，λ′）：３次元連続物体空間で定義された原画像ｆ（Ｒ，λ）の自己相関関数である。以上の解析により、
【数３６】

【０１２３】
ただし、（４３）式において行列Ｗ^k はオペレータ｛ｐ（ｒ）｝とオペレータ｛ｗ^k （ｒ）｝^t とを線形演算することにより改めて定義されるＭ′×ＭＮ行列である。ＲＯＭ１０４０には被写体の種類に対応した複数の３次元自己相関関数Ｒ_ff（Ｒ，λ；Ｒ′，λ′）、および複数の被写体の種類と複数の物体面の位置ｚ₀ に対応した２次元自己相関関数Ｒ_ff（Ｒ，λ；Ｒ′，λ′；ｚ₀ ）が記録される。またＲＯＭ１０４１には複数の物体面の位置ｚ₀ に対応した２次元感度分布関数ｈ_i （ｒ，ｚ₀ ；λ），（ｉ＝１，…，Ｍ）が記録される。
【０１２４】
ユーザは被写体の種類および結像させたい物体面の位置に応じて適当な条件を選択することにより、ＲＯＭ１０４０，１０４１に記録されている複数種類の上記係数列から適当なものが選択される。一方、ＲＯＭ１０４２には画像入力・表示系により固定された係数列である３次元感度分布関数ｈ_i （Ｒ，λ），（ｉ＝１，…，Ｍ）、色バンドパスフィルタの透過率分布特性ｔ^j （λ），（ｊ＝１，…，Ｎ）およびＲＧＢ等色関数ｓ^k （λ），（ｋ＝Ｒ，Ｇ，Ｂ）が記録されている。
【０１２５】
以上説明したようなＲＯＭ１０４０，１０４１，１０４２から送られてくる係数列を用いて行列演算器１０４３では所定の行列演算が行なわれ、係数メモリ１０４４−１，２にはそれぞれ行列Ｗ^k ，Ｚ^a の要素係数列が記録される。そして行列演算器１００４では画像メモリ１００２−１〜Ｎに記録されている各波長バンド画像と係数メモリ１０４４−１〜２に記録された行列との間で（４３）式に従った計算が行なわれ、ＲＧＢ離散原色画像が復元される。
【０１２６】
上記した第５実施形態によれば、所定の物体空間の範囲内での焦点の合った物体面の位置をコントロールできるような装置をレンズ駆動によるメカニカルな動作をせずに実現できる。従ってレンズ駆動部を省略できるために構成が簡便になる上に、モータによる振動の影響なども無い。しかも多波長バンド光学結像を用いた連続−離散画像入力系の定義に基づいた復元処理を行なうことにより解像度や色再現性に優れたカラー画像を得ることができる。
【０１２７】
なお、前記した具体的実施形態には以下の構成を有する発明が含まれており、各発明に対応する実施形態と効果は次の通りである。
【０１２８】
（構成１）
撮像光学系と、
この撮像光学系により結像される物体の光像をＮ種類の波長領域ごとに透過させるＮ種類の色フィルタで構成される多色フィルタと、
前記撮像光学系と前記多色フィルタとを介して入射される物体の光像を撮像するための撮像手段と、
前記多色フィルタにより透過されたＮ種類の波長領域の物体の光像を前記撮像手段により撮像して得られたＮ種類の多波長領域画像を記録するための画像メモリと、
この画像メモリに記録されたＮ種類の多波長領域画像との行列演算に用いられる係数列が記録された係数メモリと、
前記画像メモリに記録されたＮ種類の多波長領域画像と、前記係数メモリに記録された係数列との間で行列演算を行ない、Ｎと等しいかまたはＮより少ないＮ′種類の波長領域の画像を出力する行列演算手段と、
を具備することを特徴とする撮像装置。
【０１２９】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第１実施形態が対応する。また、前記行列演算手段による行列演算は、第１実施形態で述べた（１０）式を計算することに相当する。
【０１３０】
（効果）
Ｎ種類の多波長バンドの各々に対する光学結像特性を考慮しながらＮと等しいかまたはＮより少ないＮ′種類の波長バンド画像を復元するようにしたので、波長領域に対して冗長性を持たせると共に色収差の影響を波長単位で補正することが可能になる。これによって画像が各々正確に復元されることから結果的に色再現性に優れた画像が得られる。
【０１３１】
（構成１．１）
構成１に記載の撮像装置において、前記係数メモリに記録された係数列は、前記撮像手段における各受光素子に対して、前記撮像光学系により伝達される物体空間および波長に対する光強度分布を表す感度分布関数と、前記多色フィルタに設けられたＮ種類の色フィルタの透過率分布特性と、前記行列演算手段により出力されるＮ′種類の出力画像の各々の波長感度分布とを用いて、前記撮像光学系と、前記多色フィルタと、前記撮像手段とで決定される画像入力系を線形に関係付けたところの一般化行列に対する擬似逆行列である
ことを特徴とする。
【０１３２】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第１実施形態が対応する。
【０１３３】
（効果）
画像入力系が一般的な連続−離散系モデルに基づいて扱われることにより、従来の標本化定理に基づく解析が要求するような帯域／領域制限や伝達特性の位置不変性の条件を必要とせず、適用範囲が広く正確な解析が可能となる。従って光学結像系の収差の影響により位置によって伝達特性が変化するような画像入力系に対しても正確な画像復元ができる。これによって、レンズなどの光学結像系を簡便に設計することができ、小型軽量のカメラを用いながら解像度や色再現性に優れたカラー画像を得ることができる。
【０１３４】
（構成１．２）
構成１に記載の撮像装置において、前記行列演算手段により出力される画像の種類Ｎ′は３であり、それらの画像はカラー画像を構成する赤，緑，青の波長領域の画像である
ことを特徴とする。
【０１３５】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第１実施形態が対応する。
【０１３６】
（効果）
出力されるＮ′種類の波長領域画像をＲＧＢカラー画像とすることにより、一般的なカラー画像機器に親和性のあるカラー画像を出力できる。
【０１３７】
（構成２）
撮像光学系と、
この撮像光学系により結像される物体の光像をＮ種類の波長領域ごとに透過させるＮ種類の色フィルタで構成される多色フィルタと、
前記撮像光学系と前記多色フィルタとを介して入射される物体の光像を撮像するための撮像手段と、
前記多色フィルタにより透過されたＮ種類の波長領域の物体の光像を前記撮像手段により撮像して得られたＮ種類の多波長領域画像を記録するための画像メモリと、
前記撮像手段における各受光素子に対して、前記撮像光学系により伝達される物体空間および波長に対する光強度分布を表す感度分布関数と、前記多色フィルタに設けられたＮ種類の色フィルタの透過率分布特性と、Ｎと等しいかまたはＮより少ないＮ′種類の出力画像の各々の波長感度分布とを係数列として記録した第１の係数メモリと、
物体空間座標と波長との関数として定義される被写体画像の統計的相関情報を係数列として記録した第２の係数メモリと、
前記第１および第２の係数メモリに記録された係数列から、前記画像メモリに記録されたＮ種類の多波長領域画像との行列演算に用いられる新たな係数列を算出する係数列演算手段と、
前記画像メモリに記録されたＮ種類の多波長領域画像と、前記係数列演算手段により算出された新たな係数列との間で行列演算を行ない、Ｎ′種類の波長領域の画像を出力する行列演算手段と、
を具備することを特徴とする撮像装置。
【０１３８】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第２実施形態が対応する。また、前記行列演算手段による行列演算は、第２実施形態で述べた（２１）式を計算することに相当する。
【０１３９】
（効果）
色収差の影響によって波長ごとに結像状態が異なるような場合においても、波長バンド画像間の相関を利用して復元されるカラー画像の解像度や色再現性を向上させることができる。従って、レンズなどの光学結像系を簡便な構成とした場合でも、解像度や色再現により優れたカラー画像を得ることができる。
【０１４０】
（構成２．１）
構成２に記載の撮像装置において、前記第２の係数メモリは、複数の係数メモリとそれらの中の一つから係数列を読み込む手段とから構成される
ことを特徴とする。
【０１４１】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第２実施形態が対応する。
【０１４２】
（効果）
被写体の種類、例えば人物、風景、医療、建造物画像などに応じて適当な統計的相関情報が記録されたメモリを選択することができ、より正確な画像の復元が可能になる。
【０１４３】
（構成２．１．１）
構成２．１に記載の撮像装置において、前記第２の係数メモリにおける複数の係数メモリは複数のカード媒体に内蔵され、それらの中の一つから係数列を読み込む手段はカードリーダーである
ことを特徴とする。
【０１４４】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第２実施形態が対応する。
【０１４５】
（効果）
適当な自己相関情報が記録されたカードを選択できることから、被写体の種類に対して適応的に復元画像の画質を向上させることができる。
【０１４６】
（構成３）
撮像光学系と、
この撮像光学系により結像される物体の光像をＮ種類の波長領域ごとに透過させるＮ種類の色フィルタで構成される多色フィルタと、
前記撮像光学系と前記多色フィルタとを介して入射される物体の光像を撮像するための撮像手段と、
前記多色フィルタにより透過されたＮ種類の波長領域の物体の光像を前記撮像手段により撮像して得られたＮ種類の多波長領域画像を記録するための画像メモリと、
前記撮像手段における各受光素子に対して、前記撮像光学系により伝達される物体空間および波長に対する光強度分布を表す感度分布関数と、前記多色フィルタに設けられたＮ種類の色フィルタの透過率分布特性と、Ｎと等しいかまたはＮより少ないＮ′種類の出力画像の各々の波長感度分布と、物体空間座標と波長との関数として定義される被写体画像の統計的相関情報とを係数列として記録した第１の係数メモリと、
前記撮像手段のＳ／Ｎを表すパラメータを記録した第２の係数メモリと、
前記第１の係数メモリに記録された係数列と、前記第２の係数メモリに記録されたパラメータとから、前記画像メモリに記録されたＮ種類の多波長領域画像との行列演算に用いられる新たな係数列を算出する係数列演算手段と、
前記画像メモリに記録されたＮ種類の多波長領域画像と、前記係数列演算手段により算出された新たな係数列との間で行列演算を行ない、Ｎ′種類の波長領域の画像を出力する行列演算手段と、
を具備することを特徴とする撮像装置。
【０１４７】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第４実施形態が対応する。また、前記行列演算手段による行列演算は、第４実施形態で述べた（３７）式を計算することに相当する。
【０１４８】
（効果）
入力画像に混入するノイズの影響が抑制された明瞭なカラー画像を出力することができる。
【０１４９】
（構成３．１）
構成３に記載の撮像装置において、前記第２の係数メモリに記録されるパラメータは、前記撮像手段からの出力信号に加わるノイズの統計量である
ことを特徴とする。
【０１５０】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第４実施形態が対応する。
【０１５１】
（効果）
撮像手段において発生する入力画像と相関の無いノイズを定量的に扱うために撮像手段の性能に依存したノイズの統計量を記録しておくことにより、最適な条件でノイズを抑制できる画像復元を行なうことができる。
【０１５２】
（構成３．２）
構成３に記載の撮像装置において、前記第２の係数メモリは、前記撮像手段に内蔵される
ことを特徴とする。
【０１５３】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第４実施形態が対応する。
【０１５４】
（効果）
カメラ自体にカメラの性能に依存したＳ／Ｎパラメータを保持させるので、カメラを画像処理プロセッサに対して着脱可能に構成した場合でも使用するカメラに適応させた最適なノイズ抑制処理を実現することができる。
【０１５５】
（構成４）
構成２または３に記載の撮像装置において、
前記係数列演算手段は、前記行列演算手段により出力されたＮ′種類の波長領域の画像と、物体空間座標と波長との関数として定義される被写体画像に、前記第１の係数メモリに記録されているＮ′種類の出力画像の各々の波長感度分布を作用することにより得られるところのＮ′種類の波長領域の被写体画像との誤差が最小になるような係数を算出する
ことを特徴とする。
【０１５６】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第２，３実施形態が対応する。
【０１５７】
（効果）
被写体の原画像から得られることが予測される理想的な出力画像に対して最も誤差の少ない画像を推定復元することができる。
【０１５８】
（構成５）
波長ごとに異なる物体面に焦点が合う撮像光学系と、
この撮像光学系により結像される物体の光像をＮ種類の波長領域ごとに透過させるＮ種類の色フィルタで構成される多色フィルタと、
前記撮像光学系と前記多色フィルタとを介して入射される物体の光像を撮像するための撮像手段と、
前記多色フィルタにより透過されたＮ種類の波長領域の物体の光像を前記撮像手段により撮像することにより得られたＮ種類の多波長領域画像を記録するための画像メモリと、
前記撮像手段における各受光素子に対して、前記撮像光学系により伝達される３次元物体空間および波長に対する光強度分布を表す３次元感度分布関数と、前記撮像手段における各受光素子に対して、前記撮像光学系により伝達される３次元物体空間内の所定の２次元物体面および波長に対する光強度分布を表す２次元感度分布関数と、３次元物体空間座標と波長との関数として定義される被写体画像の統計的３次元相関情報と、３次元物体空間内の所定の２次元物体面座標と波長との関数として定義される被写体画像の統計的２次元相関情報と、前記多色フィルタに設けられたＮ種類の色フィルタの透過率分布特性と、Ｎと等しいかまたはＮより少ないＮ′種類の出力画像の各々の波長感度分布と、を係数列として記録した係数メモリと、
この係数メモリに記録されている係数列から前記画像メモリに記録されたＮ種類の多波長領域画像との行列演算に用いられる新たな係数列を算出する係数列演算手段と、
前記画像メモリに記録されたＮ種類の多波長領域画像と、前記係数列演算手段により算出された係数列との間で行列演算を行ない、Ｎ′種類の波長領域の画像を出力する行列演算手段と、
を具備することを特徴とする撮像装置。
【０１５９】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第５実施形態が対応する。また、前記行列演算手段による行列演算は、第５実施形態で述べた（４３）式を計算することに相当する。
【０１６０】
（効果）
色収差の大きい光学結像系を用いる場合は波長ごとに異なる物体面に焦点が合うことを利用して、レンズの位置を固定させながら波長と焦点位置とが共に異なる複数の画像を入力し、全入力画像に対応した合焦範囲内における任意の物体面の位置に焦点の合った画像を復元するようにしたので、所定の物体空間の範囲内で焦点の合った物体面の位置をコントロールできるような装置をレンズ駆動によるメカニカルな動作をせずに実現できる。したがって、レンズ駆動部を省略でき構成が簡便になる上にモータによる振動の影響なども無くなる。
【０１６１】
（構成５．１）
構成５に記載の撮像装置において、前記係数列演算手段は、前記係数メモリに記録されている３次元感度分布関数と、被写体画像の統計的３次元相関情報と、前記多色フィルタに設けられたＮ種類の色フィルタの透過率分布特性との線形結合により定義される行列の擬似逆行列と、２次元感度分布関数と被写体画像の統計的２次元相関情報とＮ′種類の出力画像の各々の波長感度分布との線形結合により定義される行列と、を算出する
ことを特徴とする。
【０１６２】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第５実施形態が対応する。
【０１６３】
（効果）
多波長バンド光学結像を用いた連続−離散画像入力系の定義に基づいた復元処理を行なうことができ、解像度や色再現性に優れた画像を出力することができる。
【０１６４】
（構成５．２）
構成５に記載の撮像装置において、前記係数メモリには、前記撮像手段における各受光素子に対して、前記撮像光学系により伝達される３次元物体空間内の所定の２次元物体面および波長に対する光強度分布を表す２次元感度分布関数と、３次元物体空間内の所定の２次元物体面座標と波長との関数として定義される被写体画像の統計的２次元相関情報とが異なる物体面に対応させて複数記録され、それらの内の１つづつを選択する手段を有する
ことを特徴とする。
【０１６５】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第５実施形態が対応する。
【０１６６】
（効果）
焦点の合った物体面の位置を容易に選択可能な装置を構成できる。
【０１６７】
（構成６）
構成１、２、３、５に記載の撮像装置において、前記多色フィルタはＮ種類の色フィルタが同一平面上に空間的に配置されたモザイク色フィルタである
ことを特徴とする。
【０１６８】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第３実施形態が対応する。
【０１６９】
（効果）
単板方式で多波長バンド画像を入力するカメラを用いながら、連続−離散系モデルに基づく画像復元の原理に従って解像度や色再現性に優れたカラー画像を得ることができる。従って、コンパクトでしかも画像入力時間の短い実用上有用なカラー画像撮像装置を提供することができる。
【０１７０】
（構成６．１）
構成６に記載の撮像装置において、前記モザイク色フィルタに設けられる色フィルタの種類Ｎおよび前記行列演算手段により出力される画像の数Ｎ′は、共に３である
ことを特徴とする。
【０１７１】
（対応する発明の実施形態）
この発明に関する実施の形態には、前記した第３実施形態が対応する。
【０１７２】
（効果）
一般にカラー画像は３つの原色画像により構成されるため、一般的なカラー画像機器に親和性のある処理を実現できる。
【０１７３】
（構成６．１．１）
構成６．１に記載の撮像装置において、前記モザイク色フィルタに設けられる３種類の色フィルタは赤，緑，青の波長領域を透過させる色フィルタであり、前記行列演算手段により出力されるＮ′種類の画像はカラー画像を構成する赤，緑，青の波長領域の画像である
ことを特徴とする。
【０１７４】
（対応する発明の実施形態）
この発明に係る実施形態には、前記した第３実施形態が対応する。
【０１７５】
（効果）
一般的に広く使われる単板カラーカメラやカラー画像機器を使用しながら本発明の効果を得ることができる。
【０１７６】
【発明の効果】
本発明によれば、連続−離散系モデルに基づいた画像復元方法を用いて、色再現性に優れた正確なカラー画像を復元することができる。
【図面の簡単な説明】
【図１】本発明の第１実施形態に係る撮像装置の構成を示す図である。
【図２】回転色フィルタの断面構造を示す図である。
【図３】色フィルタの透過率分布特性を示す図である。
【図４】等色関数の波長特性を示す図である。
【図５】本発明の第２実施形態に係る画像処理プロセッサの構成を示す図である。
【図６】本発明の第３実施形態におけるカメラと画像処理プロセッサの構成を示す図である。
【図７】モザイク色フィルタの構成を示す図である。
【図８】本発明の第４実施形態におけるカメラと画像処理プロセッサの構成を示す図である。
【図９】本発明の第５実施形態におけるカメラと画像処理プロセッサの構成を示す図である。
【符号の説明】
１００…カメラ、１０１…レンズ、１０２…ＣＣＤ撮像素子、１０３…回転色フィルタ、１０４…モータ、２０１…Ａ／Ｄ変換器、２０２−１〜Ｎ…画像メモリ、２０３…回転色フィルタドライバ、２０４…行列演算器、２０５…ＲＯＭ、２０６−１〜３…画像メモリ、２０７…Ｄ／Ａ変換器、２１０…ＣＰＵ、３００…ＴＶモニタ。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging apparatus that captures a color image by a camera that converts an optically imaged object image into an electronic image signal, and more particularly to an imaging apparatus configured by combining an imaging system and an image restoration processing system. .
[0002]
[Prior art]
2. Description of the Related Art Conventionally, a technique for restoring an image captured using an optical imaging system to a clear image by digital image processing is known. In such a restoration technique, in order to solve the problem defined by the continuous-continuous model in principle by the discrete-discrete model and solve it by digital processing, the two models are related based on the sampling theorem. The image restoration algorithm defined by is conventionally used.
[0003]
The sampling theorem for 2D image data is disclosed in JW Goodman, Introduction to Fourier Optics, pp. 21-25, (McGraw-Hill; San Francisco; 1968), and is defined in a continuous-continuous system model. A method for realizing a restored image using an algorithm based on a discrete-discrete system model is disclosed in WK Pratt, Digital, Image Processing, pp. 378-425, (John Wiley &Sons; New York; 1978). . However, the above-described conventional restoration technique has a limit in restoring an accurate image for the following reason.
[0004]
In order to apply the 1.2-dimensional sampling theorem, it is necessary that the continuous signal is band-limited to a rectangle in the frequency space. When applying restoration filtering using discrete Fourier transform, discretization in Fourier space is also required, and the object space is required to be limited to a rectangular region. In addition, the sampling interval depends on the length of one side of both restricted rectangular areas. However, imposing such a constraint condition is an obstacle to completely expressing the image input system.
[0005]
2. Sampling theorem holds when the transfer function of a continuous signal is position invariant. Therefore, the present invention cannot be applied when the transfer characteristics differ depending on the position due to the aberration of the optical imaging system.
[0006]
Japanese Patent Laid-Open Nos. 6-98170 and 7-160847 continuously describe the relationship between an object space defined by a continuous system and an image sensor defined by a discrete system composed of a plurality of light receiving elements. -A method of redefining with a discrete system model and accurately restoring an image according to the generalized inverse matrix concept is disclosed. However, JP-A-6-98170 discloses a restored image by performing a restoration process for each color band image of red (Red; R), green (Green; G), and blue (Blue; B). Such a method has a limit in realizing accurate color reproducibility, and no method for overcoming this is disclosed. Further, when the optical imaging system has chromatic aberration, the transfer characteristics differ for each color band image, but no mention is made of how this affects the finally obtained restored image.
[0007]
[Problems to be solved by the invention]
As described above, a conventional image restoration method using a model obtained by approximating an imaging relationship defined by a continuous-continuous system by a discrete-discrete system is used for an image input system that does not comply with the conditions required by the sampling theorem. In many cases, the image cannot be restored because it cannot be applied in principle.
[0008]
The conventional image restoration technique based on the continuous-discrete system model does not disclose any color image restoration method for realizing accurate color reproducibility.
[0009]
The imaging apparatus of the present invention has been made paying attention to such a problem, and the object is to use an image restoration method based on a continuous-discrete system model, and to achieve an accurate color reproducibility. An object of the present invention is to provide an imaging apparatus capable of restoring a color image.
[0010]
[Means for Solving the Problems]
  In order to achieve the above object, an imaging apparatus according to a first aspect of the present invention includes an imaging optical system and N types of light that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions. A multicolor filter composed of color filters, an image pickup means for picking up an optical image of an object incident through the image pickup optical system and the multicolor filter, and N types transmitted by the multicolor filter An image memory for recording N types of multi-wavelength region images obtained by imaging an optical image of an object in the wavelength region of the image by the imaging unit, and N types of multi-wavelength region images recorded in the image memory; A matrix operation is performed between the coefficient memory in which the coefficient sequence used in the matrix operation is recorded, the N types of multi-wavelength region images recorded in the image memory, and the coefficient sequence recorded in the coefficient memory, Equal to or greater than N And a matrix operation means for outputting an image free N 'type wavelength regionThe coefficient sequence recorded in the coefficient memory includes a sensitivity distribution function representing a light intensity distribution with respect to an object space and a wavelength transmitted by the imaging optical system for each light receiving element in the imaging unit, and the multicolor. Using the transmittance distribution characteristics of N types of color filters provided in the filter and the wavelength sensitivity distribution of each of N ′ types of output images output by the matrix calculation means, the imaging optical system, This is a pseudo inverse matrix for a generalized matrix obtained by linearly relating an image input system determined by a multicolor filter and the imaging means..
[0011]
  An imaging apparatus according to a second aspect of the invention is an imaging optical system and a multiplicity of N color filters configured to transmit an optical image of an object imaged by the imaging optical system for each of N wavelength regions. Color filters, imaging means for capturing a light image of an object incident through the imaging optical system and the multicolor filter, and light of an object in N types of wavelength regions transmitted by the multicolor filter An image memory for recording N types of multi-wavelength region images obtained by imaging an image by the imaging means;A coefficient memory in which a coefficient sequence used for matrix operation with N types of multi-wavelength area images recorded in the image memory is recorded, N types of multi-wavelength area images recorded in the image memory, and the coefficient memory Matrix calculation means for performing matrix calculation with the recorded coefficient sequence and outputting images of three types of wavelength regions of red, green, and blue constituting a color image, less than N;It has.
[0012]
  An image pickup apparatus according to a third aspect of the present invention is an image pickup optical system and a multiplicity of N color filters configured to transmit an optical image of an object formed by the image pickup optical system for each of N kinds of wavelength regions. Color filters, imaging means for capturing a light image of an object incident through the imaging optical system and the multicolor filter, and light of an object in N types of wavelength regions transmitted by the multicolor filter An image memory for recording N types of multi-wavelength region images obtained by imaging an image by the imaging unit, an object space transmitted by the imaging optical system to each light receiving element in the imaging unit, and Sensitivity distribution function representing light intensity distribution with respect to wavelength, transmittance distribution characteristics of N kinds of color filters provided in the multicolor filter, and wavelengths of N ′ kinds of output images equal to or less than N SensitivityAndA first coefficient memory recorded as a coefficient sequence;Statistical correlation information of subject images defined as a function of object space coordinates and wavelength as a coefficient sequenceA recorded second coefficient memory; and the first coefficient memoryAnd from the coefficient sequence recorded in the second coefficient memory,Coefficient sequence calculation means for calculating a new coefficient sequence used for matrix calculation with N types of multi-wavelength region images recorded in the image memory; N types of multi-wavelength region images recorded in the image memory; Matrix calculation means for performing matrix calculation with the new coefficient string calculated by the coefficient string calculation means and outputting images of N ′ types of wavelength regions.
  An imaging device according to a fourth aspect of the invention isAn imaging optical system, a multicolor filter including N types of color filters that transmit an optical image of an object imaged by the imaging optical system for each of N types of wavelength regions, the imaging optical system, and the multicolor An imaging unit for capturing an optical image of an object incident through a filter and an optical image of an object in N types of wavelength regions transmitted by the multicolor filter are obtained by the imaging unit. An image memory for recording N types of multi-wavelength area images, and a sensitivity distribution function representing a light intensity distribution with respect to the object space and wavelength transmitted by the imaging optical system for each light receiving element in the imaging unit; Functions of transmittance distribution characteristics of N color filters provided in the multicolor filter, wavelength sensitivity distributions of N ′ type output images equal to or less than N, and object space coordinates and wavelengths When A first coefficient memory that records statistical correlation information of the subject image defined as a coefficient sequence, a second coefficient memory that records a parameter representing the S / N of the imaging means, and the first coefficient From the coefficient sequence recorded in the memory and the parameter recorded in the second coefficient memory, a new coefficient sequence used for matrix operation with N types of multi-wavelength area images recorded in the image memory is calculated. Performing a matrix operation between the coefficient sequence calculating means to perform, the N types of multi-wavelength region images recorded in the image memory and the new coefficient sequence calculated by the coefficient sequence calculating means, and N ′ types of wavelength regions Matrix calculating means for outputting the above image.
  An imaging apparatus according to a fifth aspect of the present invention transmits an imaging optical system that focuses on a different object surface for each wavelength and an optical image of an object formed by the imaging optical system for each of N types of wavelength regions. A multicolor filter composed of N types of color filters, an imaging means for capturing a light image of an object incident through the imaging optical system and the multicolor filter, and transmitted by the multicolor filter. An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions by the imaging unit, and for each light receiving element in the imaging unit, A three-dimensional sensitivity distribution function representing a light intensity distribution with respect to a three-dimensional object space and wavelength transmitted by the imaging optical system, and a three-dimensional transmission transmitted by the imaging optical system to each light receiving element in the imaging means. A two-dimensional sensitivity distribution function representing a light intensity distribution with respect to a predetermined two-dimensional object plane and wavelength in body space, and statistical three-dimensional correlation information of a subject image defined as a function of three-dimensional object space coordinates and wavelength; Statistical two-dimensional correlation information of a subject image defined as a function of predetermined two-dimensional object plane coordinates and wavelength in a three-dimensional object space, and transmittance distribution of N types of color filters provided in the multicolor filter A coefficient memory in which the characteristics and the wavelength sensitivity distribution of each of N ′ types of output images equal to or less than N are recorded as a coefficient sequence, and the coefficient sequence recorded in the coefficient memory is stored in the image memory. Coefficient sequence calculating means for calculating a new coefficient sequence used for matrix operation with the recorded N types of multi-wavelength region images, N types of multi-wavelength region images recorded in the image memory, and the coefficient sequence calculation Performs matrix operation between the calculated coefficient sequence by stages, and a matrix operation means for outputting an image of N 'type wavelength region To do.
[0013]
  That is, the imaging apparatus according to the first invention is configured by an imaging optical system and N color filters that transmit an optical image of an object imaged by the imaging optical system for each of N wavelength regions. A light image of an object incident through the multi-color filter is picked up by the image pickup means, and N types of multi-wavelength area images obtained by the image pickup are recorded in the image memory. Next, matrix calculation is performed by the matrix calculation means between the N types of multi-wavelength region images recorded in the image memory and the coefficient sequence recorded in the coefficient memory, and N ′ equal to or less than N ′. Outputs images of various wavelength regions.Here, the coefficient sequence recorded in the coefficient memory is provided in a multi-color filter and a sensitivity distribution function representing the light intensity distribution with respect to the object space and wavelength transmitted by the imaging optical system for each light receiving element in the imaging means. An image pickup optical system, a multicolor filter, and an image pickup using the transmittance distribution characteristics of the N kinds of color filters obtained and the wavelength sensitivity distribution of each of the N ′ kinds of output images output by the matrix calculation means This is a pseudo-inverse matrix for the generalized matrix that linearly relates the image input system determined by the means.
[0014]
  An image pickup apparatus according to the second invention is first configured with an image pickup optical system and N types of color filters that transmit an optical image of an object formed by the image pickup optical system for each of N types of wavelength regions. A light image of an object incident through the multi-color filter is picked up by the image pickup means, and N types of multi-wavelength area images obtained by the image pickup are recorded in the image memory. AndA matrix operation is performed between the N types of multi-wavelength area images recorded in the image memory and the coefficient sequence recorded in the coefficient memory by a matrix operation means, and less than N, red, green, Outputs three types of blue wavelength range images.
[0015]
  An imaging apparatus according to a third aspect of the invention is first configured with an imaging optical system and N types of color filters that transmit an optical image of an object imaged by the imaging optical system for each of N types of wavelength regions. A light image of an object incident through the multi-color filter is picked up by the image pickup means, and N types of multi-wavelength area images obtained by the image pickup are recorded in the image memory. A sensitivity distribution function representing a light intensity distribution with respect to an object space and a wavelength, which is recorded in the first coefficient memory and transmitted by the imaging optical system to each light receiving element in the imaging unit, and the multicolor filter A coefficient sequence consisting of the transmittance distribution characteristics of the provided N color filters and the wavelength sensitivity distribution of each of the N ′ type output images equal to or less than N, and recorded in the second coefficient memory ,From a coefficient sequence consisting of statistical correlation information of the subject image defined as a function of object space coordinates and wavelength,A new coefficient sequence used for matrix calculation with N types of multi-wavelength area images recorded in the image memory is calculated by a coefficient sequence calculation means, and N types of multi-wavelength area images recorded in the image memory; A matrix operation is performed by the matrix operation unit with the new coefficient sequence calculated by the coefficient sequence operation unit, and images of N ′ types of wavelength regions are output.
  An imaging apparatus according to a fourth aspect of the invention first includesLight of an object incident through an imaging optical system and a multicolor filter including N color filters that transmit a light image of the object imaged by the imaging optical system for each of N wavelength regions An image is picked up by the image pickup means, and N types of multi-wavelength area images obtained in the image pickup are recorded in the image memory. A sensitivity distribution function representing a light intensity distribution with respect to an object space and a wavelength, which is recorded in the first coefficient memory and transmitted by the imaging optical system to each light receiving element in the imaging unit, and the multicolor filter Object defined as a function of transmittance distribution characteristics of N color filters provided, wavelength sensitivity distribution of each of N ′ type output images equal to or less than N, and object space coordinates and wavelength N types of multi-wavelength region images recorded in the image memory from a coefficient sequence composed of statistical correlation information of the image and a parameter recorded in the second coefficient memory and representing the S / N of the imaging means A new coefficient sequence used in the matrix calculation is calculated by the coefficient sequence calculation means, and the N types of multi-wavelength area images recorded in the image memory and the new coefficient calculated by the coefficient sequence calculation means are calculated. Performs matrix operation by the matrix operation means to and from the column, and outputs an image of N 'type of wavelength regions.
  An imaging apparatus according to a fifth aspect of the invention first includes an imaging optical system that focuses on a different object plane for each wavelength, and an optical image of an object formed by the imaging optical system for each of N types of wavelength regions. A light image of an object incident through a multi-color filter composed of N kinds of color filters to be transmitted is picked up by an image pickup means, and the N kinds of multi-wavelength region images obtained by the image pickup are recorded in an image memory. To do. And for each light receiving element in the imaging means, a three-dimensional sensitivity distribution function representing a light intensity distribution with respect to a three-dimensional object space and wavelength transmitted by the imaging optical system, and for each light receiving element in the imaging means , Defined as a function of a two-dimensional sensitivity distribution function representing a light intensity distribution with respect to a predetermined two-dimensional object plane and wavelength in the three-dimensional object space transmitted by the imaging optical system, and a function of three-dimensional object space coordinates and wavelength. Statistical multi-dimensional correlation information of a subject image, statistical two-dimensional correlation information of a subject image defined as a function of a predetermined two-dimensional object plane coordinate and wavelength in a three-dimensional object space, and the multicolor filter The transmittance distribution characteristics of the N kinds of color filters obtained and the wavelength sensitivity distribution of each of the N ′ kinds of output images equal to or less than N are recorded in the coefficient memory as a coefficient sequence. To. Next, a new coefficient sequence used for matrix calculation with the N types of multi-wavelength region images recorded in the image memory is calculated by the coefficient sequence calculation means from the coefficient sequence recorded in the coefficient memory, and the image A matrix operation is performed by the matrix operation unit between the N types of multi-wavelength region images recorded in the memory and the coefficient sequence calculated by the coefficient sequence operation unit, and images of N ′ types of wavelength regions are output.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0017]
First, a first embodiment of the present invention will be described. 1st Embodiment of this invention is related with the imaging device which inputs a multi-wavelength band image (multi-wavelength area | region image), and decompress | restores an RGB color image using them. FIG. 1 shows the configuration of the first embodiment of the present invention. The configuration is largely divided into a camera 100, an image processor 200, and a TV monitor 300. An optical imaging system (imaging optical system) including a lens 101 is provided in the camera 100, and an optical image of a subject is placed on the light receiving surface of a CCD image sensor 102 as an imaging unit via the lens 101. Imaged. In place of the CCD image sensor 102, an image sensor of another type such as MOS or CMD may be used.
[0018]
A rotating color filter (multicolor filter) 103 is provided between the lens 101 and the CCD image sensor 102. As shown in FIG. 2, the rotary color filter 103 has a cross-sectional structure in which N color filters 103-1 to 10-3 are arranged in the rotation direction, and a predetermined color filter is selected by the motor 104 for optical imaging. It is installed in the optical path of the system. As shown in FIG. 3, each of the color filters 103-1 to 103-N has a transmittance distribution characteristic in a predetermined wavelength band within the visible light wavelength range.
[0019]
Wavelength band images corresponding to the color filters 103-1 to 103 -N picked up by such a configuration are digitally converted by the A / D converter 201 in the image processor 200 and stored in the N image memories 202-1 to 202 -N. Are recorded on a predetermined one of them. The above operation is repeated when the rotation color filter 103 is controlled to rotate in the camera 100 and one of the color filters 103-1 to N is selected one after another, and a total of N wavelength band images are stored in the image memory 202-. 1 to N.
[0020]
A rotation color filter driver 203 is provided in the image processor 200 and controls the operation of the motor 104 in the camera 100.
[0021]
Next, each wavelength band image recorded in the image memories 202-1 to 202-N and a coefficient sequence recorded in the ROM 205 as a coefficient memory are input to the matrix calculator 204, and a predetermined matrix operation, that is, a product-sum operation is performed. Is executed and the result is recorded in one of the image memories 206-1 to 206-3. Here, the image memories 206-1 to 206-3 are used to record R, G, and B images constituting each color image. That is, when an R image is formed, a coefficient sequence for the R image is read out from the data recorded in the ROM 205, a predetermined matrix operation is performed, and the result is recorded in the image memory 206-1. The same operation is repeated for the G and B images.
[0022]
The color images recorded in the image memories 206-1 to 206-3 in this way are converted into predetermined analog video signals by the D / A converter 207 and displayed on the TV monitor 300. The operation control in the above configuration is performed by the CPU 210 in the image processing processor 200.
[0023]
In the first embodiment described above, the configuration in which the camera 100 and the image processor 200 are connected online is shown, but an offline configuration may be used. That is, a configuration is provided in which image data output from the CCD image sensor 102 in the camera 100 is recorded as an analog signal or digitally converted and then recorded on a medium such as a semiconductor memory card, floppy disk, or optical memory card. May be configured to realize similar processing by providing a reader device that reads image signals from these recording media. In such a case, the user can take advantage of the off-line system, such as being able to go to the shooting location with only the camera.
[0024]
The operation of the first embodiment having the above configuration will be described below. In the first embodiment, an RGB color image is restored using N multi-wavelength band images input using N color filters 103-1 to 103 -N whose transmission characteristics are set over the visible light range. Have At that time, the generalized inverse matrix method based on the continuous-discrete system model as described below optimizes the difference in imaging characteristics for each wavelength due to the distortion of the peripheral part due to the aberration of the optical imaging system and the influence of chromatic aberration. To restore the correct image. In addition, by obtaining RGB images from N (N> 3) multi-wavelength band images, it has the effect of realizing accurate color reproducibility.
[0025]
In other words, according to the following analysis, the image input system is handled based on a general continuous-discrete system model. Therefore, the band / region limitation and the position of the transfer characteristic are not as required by the analysis based on the conventional sampling theorem. A denaturation condition is not required, and a wide range of application and accurate analysis are possible. Accordingly, accurate image restoration can be performed even for an image input system in which the transfer characteristic changes depending on the position due to the influence of aberration of the optical imaging system. Moreover, it is possible to correct the influence of chromatic aberration in units of wavelengths by restoring the color image while taking into consideration the optical imaging characteristics for each of the multi-wavelength bands that are finely set. As a result, a color image having excellent color reproducibility is obtained.
[0026]
The image restoration process is realized by a product-sum operation performed in the matrix computing unit 204 using the multi-wavelength band images recorded in the image memories 202-1 to 202-N and the coefficient sequence recorded in the ROM 205. The content of specific arithmetic processing will be described. First, an image input system using an optical imaging system is defined by a continuous-discrete system model that associates an object space defined by a continuous system and an imaging unit defined by a discrete system. Focusing on one of the M light receiving elements constituting the CCD image sensor 102 (i-th; i = 1,..., M), one of the N wavelength band filters (j-th; j = 1,..., N) Consider a system in which an optical image formed through light is received, and formulate it as follows.
[0027]
[Expression 1]

[0028]
Where g_i ^j: Observation value of the i th light receiving element input using the j th wavelength bandpass filter, t^j(Λ): transmittance distribution characteristic of j-th wavelength bandpass filter, h_i(R, λ): sensitivity distribution function defined for both object space r and wavelength λ in the i-th light receiving element, f (r, λ): for both object space r representing object and wavelength λ Is an original image function defined by
[0029]
Sensitivity distribution function h_i(R, λ) is defined by expanding the sensitivity distribution function disclosed in the above-mentioned JP-A-6-98170 and JP-A-7-160847. It is a function showing the sensitivity distribution of the light intensity with respect to the object space and the wavelength of light. Sensitivity distribution function h_i(R, λ) may be actually measured by a configuration as shown in FIG. 1 of JP-A-6-98170, or may be obtained by calculation based on an analysis of an optical imaging system.
[0030]
In the definition of equation (1), the original image f (r, λ) is defined as a continuous system, and the observed value
g_i ^jIs defined as a discrete value. In order to relate the two, the transmittance distribution characteristic t of the wavelength band filter^j(Λ) and sensitivity distribution function h_i(R, λ) is defined. On the other hand, the process of converting the original image f (r, λ) into an RGB color image is expressed by the following equation.
[0031]
[Expression 2]

[0032]
Where f^k(R): kth primary color image (k = R, G, B), s^k(Λ): A color matching function of the k-th color and having a wavelength characteristic as shown in FIG. Further, in order to display a color image with an actual device, a primary color image f defined by a continuous system^kSince it is necessary to resample (r), the process is defined by the following equation.
[0033]
[Equation 3]

Where f_i ^k: I-th pixel value of the k-th primary color image, p_i(R): A sampling function for obtaining the i-th pixel value. The process of estimating each primary color image represented by equation (2) from a plurality of wavelength band images input by the image input system represented by equation (1) is described in JP-A-6-98170 and JP-A-Hei. Formulating in accordance with the idea of a pseudo inverse matrix generally extended and defined for a continuous-discrete system model as disclosed in Japanese Patent Laid-Open No. 7-160847, the following expression is obtained. In order to avoid complications, the following description omits descriptions related to derivation in the middle, and shows only the final results.
[0034]
[Expression 4]

[0035]
[Equation 5]

[0036]
In the equation (6), the continuous-discrete conversion linear operator {h (r, λ)} is M sensitivity distribution functions h equal to the number of light receiving elements as shown in the following equation._iIt is an operator configured by arranging (r, λ) in the column direction.
[0037]
[Formula 6]

[0038]
After all, the operator {a^k(R)}^tIs configured by arranging N × M functions in the row direction. Meanwhile, matrix B^aIs a pseudo-inverse of the matrix B. The matrix B is an N × N block matrix B as follows:_mn(M, n = 1, 2,..., N).
[0039]
[Expression 7]

[0040]
The estimated primary color image defined in the continuous space obtained according to the equation (4) is then converted into a discrete value based on the equation (3). Therefore, a discrete primary color image is represented by a vector f.^k= [F₁ ^kf₂ ^k... f_{M '}  ^k]^t, The observation vector g to vector f^kThe process of estimating is summarized in equation (10). Note that M ′ is the number of pixels of the discrete primary color image, and may be determined practically equal to the number M of light receiving elements.
[0041]
[Equation 8]

[0042]
Further, in the equation (10), the matrix A^kAre operator {p (r)} and operator {a^k(R)}^tIs a M ′ × MN matrix redefined by performing a linear operation on^kIs matrix A^kAnd matrix B^aIs an M ′ × MN matrix defined by calculating the matrix product of That is, the coefficient sequence recorded in advance in the ROM 205 is a matrix C expressed by equation (11).^kIt is sufficient to record all the coefficient sequences defined for each of k = R, G, and B.
[0043]
According to the first embodiment described above, an accurate color image can be restored by performing restoration processing based on the definition of a continuous-discrete image input system using multi-wavelength band optical imaging. The optical imaging system can be easily designed, and a color image excellent in resolution and color reproducibility can be obtained while using a small and light camera.
[0044]
The second embodiment of the present invention will be described below. The second embodiment of the present invention relates to a method of restoring a higher-performance color image using a card in which statistical autocorrelation information of a subject predicted in advance is recorded. FIG. 5 shows the configuration of the image processor 400 in the second embodiment. Since the other cameras and the TV monitor are the same as those of the camera 100 and the TV monitor 300 in the first embodiment, illustration and description thereof are omitted.
[0045]
In FIG. 5, an A / D converter 401, image memories 402-1 to 40-N, a rotation color filter driver 403, a matrix calculator 404, image memories 406-1 to 40-3, a D / A converter 407, and a CPU 410 are respectively illustrated. 1 is the same as the A / D converter 201, the image memories 202-1 to 202-N, the rotation color filter driver 203, the matrix calculator 204, the image memories 206-1 to 206, the D / A converter 207, and the CPU 210. In FIG. 5, the connection between the CPU 410 and other components is omitted in order to avoid complexity.
[0046]
In the second embodiment, in addition to these, a removable card 420 and a card reader 421 are prepared according to the purpose and application. The card 420 as the second coefficient memory incorporates a recording medium such as a semiconductor IC or an optical memory, and records information (statistical correlation information) about the autocorrelation matrix of the subject. The coefficient value of the autocorrelation matrix is sent to a matrix calculator 423 as a coefficient string calculation means together with the coefficient value recorded in the ROM 422 as the first coefficient memory, whereby a predetermined matrix calculation is performed and calculated. The two kinds of new coefficient values thus recorded are recorded in the coefficient memories 424-1 and 424-2.
[0047]
In the matrix calculator 404 as a matrix calculation means, a predetermined matrix calculation is performed between the multi-wavelength band images recorded in the image memories 402-1 to 402-N and the coefficient values recorded in the coefficient memories 424-1 to 4-2. And the restored RGB color image is recorded in the image memories 406-1 to 406-1.
[0048]
The operation of the second embodiment having the above configuration will be described below. The second embodiment relates to a method for estimating and restoring an image with the least error with respect to an original image of a subject by reformulating a continuous-discrete image input model based on a least-square criterion as described below. . This requires statistical autocorrelation information of the original image predicted a priori, so that the apparatus of FIG. 5 is suitable for the type of subject, for example, person, landscape, medical care, building image, etc. The card 420 on which information is recorded can be selected.
[0049]
Such a method has the effect of improving the resolution and color reproducibility of a color image restored using the correlation between wavelength band images even when the imaging state varies depending on the wavelength due to the influence of chromatic aberration. Have. For example, in an optical imaging system that is greatly affected by chromatic aberration, red and blue wavelength regions are out of focus when an image is formed in the most focused state with respect to the green wavelength region. Therefore, if restoration is performed for each wavelength band image, the red and blue images cannot fully recover the blur compared to the green image, so the resulting color image has a resolution balance between the color band images. However, the color reproducibility is poor. However, according to the method of the second embodiment, the deterioration of the imaging characteristics in the red or blue image is corrected by utilizing the correlation information between the green image and the red or blue image that can be input in a focused state. However, it has the effect of restoring the image quality equivalent to that of green.
[0050]
In the following, the re-formulation of the image input system based on the least square criterion will be described, and the contents of the autocorrelation information recorded on the card 420 and a specific calculation method will be described. In the following description, calculation formulas such as partial differentiation by a discrete-continuous conversion operator appear, but all relational expressions are linear and are defined by extending the definition in a discrete system composed of matrices and vectors. That is, the continuous system may be treated as if it is a discrete system having an infinite number of elements.
[0051]
[Equation 9]

[0052]
However, the operator E [•] represents a set average. Mean square error e^kTo minimize the operator {m^k(R)}^tIs the expression (18)^k(R)} is obtained by the following equation such that the value obtained by partial differentiation becomes zero.
[0053]
[Expression 10]

[0054]
An operator {m finally obtained by solving the equation (14)^k(R)}^tIs expressed as:
[0055]
## EQU11 ##

[0056]
However, the discrete-continuous conversion operator {d^k(R)}^tIs defined as follows:
[0057]
[Expression 12]

[0058]
However, the j-th element function [d^k(R)]^jIs defined as:
[0059]
[Formula 13]

[0060]
Where R_ff(R, λ: r ′, λ ′) is an autocorrelation function of the original image f (r, λ), and is defined by the following equation.
[0061]

Comparing the equations (16) and (17) with the equations (5) and (6), the operator {d defined in the second embodiment introducing the least square criterion^k(R)}^tEach element function [d^k(R)]^jIs the operator {a defined in the first embodiment.^k(R)}^tEach element function in [a^k(R)]^jThe autocorrelation function R of the original image f (r, λ)_ffIt can be seen that (r, λ: r ′, λ ′) is inserted.
[0062]
On the other hand, the matrix Q in equation (15)^aIs a pseudo inverse of the matrix Q defined as follows.
[0063]
[Expression 14]

[0064]
Comparing the equations (19) and (20) with the equations (8) and (9), each block matrix Q of the matrix Q defined in the second embodiment._mnIs a block matrix B of the matrix B defined in the first embodiment._mnAutocorrelation function of the original image f (r, λ)
R_ffIt can be seen that (r, λ: r ′, λ ′) is inserted. Finally, the estimated primary color image defined in the continuous space obtained according to the equation (12) is discretely transformed based on the equation (3).
[Expression 15]

[0065]
However, the matrix D in equation (21)^kAre operator {p (r)} and operator {d^k(R)}^tThis is an M ′ × MN matrix that is defined again by performing a linear operation. That is, the auto-correlation R predicted to be closest to the property of the subject of interest on the card 420._ff(R, λ: r ′, λ ′) may be recorded, and the transmittance distribution characteristic t of each wavelength bandpass filter is stored in the ROM 422.^j(Λ), sensitivity distribution function h defined for both object space r and wavelength λ_i(R, λ), k-th color matching function s^k(Λ), sampling function p_i(R) may be recorded. In an actual apparatus, autocorrelation R_ffContinuous function such as (r, λ; r ′, λ ′) and sensitivity distribution function h_iAlthough it is difficult to strictly record an operator that converts a discrete system and a continuous system such as (r, λ), the number of samplings is larger than the sampling number (M or M ′) of the discrete system. If a pseudo continuous space with fine pitches is defined, there is no practical problem, and processing in accordance with the gist of the present invention can be realized.
[0066]
Data recorded in the card 420 and the ROM 422 is converted into a matrix D defined by the equation (21) in the matrix calculator 423.^k, Q^aAnd the results are recorded in the coefficient memories 424-1 and 424-2, respectively. Then, the matrix calculator 404 performs a calculation according to the equation (21) between each wavelength band image recorded in the image memories 402-1 to 402-N and the matrix coefficient recorded in the coefficient memories 424-1 to 4-2. Thus, the RGB primary color image is restored.
[0067]
In the second embodiment, the auto-correlation information of the image is recorded on the card 420. However, in the case where the conditions of the image input system are fixed in advance, the matrix D finally required^k, Q^aMay be recorded on the card 420. In that case, the ROM 422, the matrix calculator 423, and the coefficient memories 424-1 and 4-2 are omitted, and the configuration becomes simple. When an autocorrelation function that can be set in common for a wide range of subject types is defined when targeting a specific subject, the matrix D finally calculated is the same as in the first embodiment.^kQ^aThese elements may be recorded in one ROM.
[0068]
According to the second embodiment described above, since it is possible to correct unevenness in resolution between primary color images due to the influence of chromatic aberration and the like, a color image that is superior in resolution and color reproduction while simplifying an optical imaging system such as a lens. Can be obtained. Further, since a card on which appropriate autocorrelation information is recorded can be selected, the quality of the restored image can be improved adaptively with respect to the type of subject.
[0069]
The third embodiment of the present invention will be described below. The third embodiment of the present invention relates to a method for restoring an RGB color image from an image input using a single-plate color camera equipped with a mosaic color filter. FIG. 6 shows the configuration of the camera 500 and the image processor 600 in the third embodiment. An optical imaging system including a lens 501 is provided in the camera 500, and an optical image of a subject is formed on the light receiving surface of the CCD image sensor 502 via the lens 501. The CCD image pickup device 502 is a single-plate color image pickup device, and a mosaic color filter 505 is mounted on the image pickup surface. The mosaic color filter 505 is provided with a color filter according to a color arrangement called a green checkered R / B line sequential method or a Bayer method as shown in FIG. 7, for example, and the CCD image pickup device 502 captures a color image by a pixel sequential method. Is done.
[0070]
An output image signal from the CCD image sensor 502 is digitally converted by an A / D converter 601 in the image processor 600 and recorded in the image memory 602. Next, the digital image signal recorded in the image memory 602 is sent to the matrix calculator 604 together with the coefficient sequence stored in the ROM 605, and an RGB color image is restored by performing a predetermined matrix calculation, and each primary color image is restored. Are recorded in the image memories 606-1 to 606-3, respectively. The RGB images recorded in the image memories 606-1 to 606-3 are converted into appropriate analog video signals by the D / A converter 607 and displayed on a TV monitor (not shown).
[0071]
The operation of the third embodiment having the above configuration will be described below. The third embodiment is based on a continuous-discrete system model as shown in the first or second embodiment from an image in a three-band wavelength region of R, G, B input by a single-plate color camera 500. According to the principle of image restoration, it has an effect of obtaining a color image having excellent resolution and color reproducibility. A pixel sequential color image input via the mosaic color filter 505 is considered to be a collection of primary color images having missing pixels. That is, according to the mosaic color arrangement as shown in FIG. 7, for example, when attention is paid to the R image, the pixels in which the G and B filters are arranged are considered to be missing pixels. The third embodiment has an effect of restoring a complete primary color image without any missing from the primary color image including such missing pixels. The restoration process is defined based on a continuous-discrete image input model in the three-color band region of R, G, and B, but is eventually realized by matrix calculation as in the first and second embodiments.
[0072]
A specific calculation method based on the least square criterion similar to that of the second embodiment will be described below. First, paying attention to one of the M light receiving elements constituting the CCD image sensor 502 (i-th; i = 1,..., M), one of the R, G, and B color bands (j-th). J = R, G, B) and discrete value g_i ^jThe image input system in which is observed is formulated as follows.
[0073]
[Expression 16]

[0074]
However, h_i ^j(R): sensitivity distribution function defined for the object space r in the j-th color band region of the i-th light receiving element, f^j(R): Original image function in the jth color band region. Also, the number of R, G, and B pixels is set to M._R,
M_G, M_BM_R+ M_G+ M_B= M.
[0075]
[Expression 17]

[0076]
[Formula 18]

[0077]
The operator {n finally obtained by solving the equation (25)^k(R)}^tIs expressed as:
[0078]
[Equation 19]

[0079]
However, the discrete-continuous conversion operator {u^k(R)}^tIs defined as follows:
[0080]
[Expression 20]

[0081]
The j-th element function [u] in equation (27)^k(R)]^jIs defined as:
[0082]
[Expression 21]

[0083]
However, R_kj(R; r ′) is the original image f^k(R) and original image f^jThis is a cross-correlation function of (r ′) and is expressed by the following equation.
[0084]
[Expression 22]

[0085]
Further, the continuous-discrete conversion operator {h^j(R)} is M with the jth color filter attached as shown in the following equation._iIt is defined by arranging the sensitivity distribution function for each pixel in the column direction.
[0086]
[Expression 23]

[0087]
On the other hand, the matrix V in equation (26)^aIs a pseudo-inverse of the matrix V defined as follows.
[0088]
[Expression 24]

[0089]
However, the block matrix V_kjIs defined as:_k× M_jHas elements.
[Expression 25]

[0090]
Finally, the estimated primary color image defined in the continuous space obtained according to the equation (23) is discretely transformed based on the equation (3).
[Equation 26]

[0091]
However, in the equation (33), the matrix U^kAre the operator {p (r)} and the operator {u^k(R)}^tIs an M ′ × M matrix that is newly defined by performing a linear operation. Matrix X^kIs the matrix U^kAnd matrix V^aThis is an M ′ × M matrix that is defined again by calculating the matrix product.
[0092]
[Expression 27]

[0093]
In the third embodiment as well, as in the second embodiment described above, by making it possible to insert and remove the card on which the correlation information between the primary color images is removable, the subject can be adaptively restored. it can. The third embodiment can be applied not only to the second embodiment but also to the first embodiment described above, and further to the fourth and fifth embodiments described below.
[0094]
According to the third embodiment described above, since a color image excellent in resolution and color reproducibility can be obtained while using a single-plate camera, a practically useful color image imaging device that is compact and has a short image input time. Can be provided.
[0095]
The fourth embodiment of the present invention will be described below. The fourth embodiment of the present invention relates to a method of previously recording data relating to S / N of an image sensor in a memory in a camera and restoring a color image in which noise is adaptively suppressed based on the data. FIG. 8 shows the configuration of the camera 700 and the image processor 800 in the fourth embodiment. In the camera 700, the lens 701, the CCD image sensor 702, the rotation color filter 703, and the motor 704 are the same as the lens 101, the CCD image sensor 102, the rotation color filter 103, and the motor 104, respectively, in the camera 100 of the first embodiment. In addition to these, the camera 700 includes a ROM 706 as a second coefficient memory in which data relating to the S / N of the image sensor 702 is recorded.
[0096]
In the image processor 800, the A / D converter 801, the image memories 802-1 to 802-1, the rotation color filter driver 803, the matrix calculator 804, the image memories 806-1 to 806, the D / A converter 807, and the CPU 810 are A / D converter 201, image memories 202-1 to 202-N, rotation color filter driver 203, matrix calculator 204, image memories 206-1 to 206, D / A conversion, respectively, in image processor 200 of the first embodiment. The same as the device 207 and the CPU 210.
[0097]
The image processor 800 further includes a ROM 830 as a first coefficient memory, and the coefficient sequence recorded in the ROM 830 is a matrix calculator as a coefficient sequence calculation means together with data sent from the ROM 706 in the camera 700. The coefficient sequence calculated by performing a predetermined matrix operation is recorded in the coefficient memories 832-1 and 832-1. An RGB primary color image is restored by performing a matrix operation between these coefficient sequences and the multiwavelength band images recorded in the image memories 802-1 to 802-1, and the result is the image memory 806-1 to 806-1. To be recorded.
[0098]
The operation of the fourth embodiment having the above-described configuration will be described below. The fourth embodiment has an effect of restoring a color image while suppressing the influence of noise mixed in the input image due to thermal noise of the CCD image sensor 702 or the like. By providing a ROM 706 in which a parameter indicating the S / N characteristic of the camera 700 is recorded in the camera 700, an adaptive noise suppression effect is brought about for the camera 700.
[0099]
A color image restoration process taking S / N into consideration will be described below. An observation vector g ′ added with additive discrete noise is expressed by the following equation.
[0100]
g ′ = g + n (34)
Where g is an observation vector to which no noise is added, and n is a noise vector. The process of estimating the primary color image defined in the continuous space from this observation vector g ′ is derived as shown in the following equation (35) by the analysis according to the least square criterion similar to the second embodiment. However, in the derivation of the equation, it was assumed that there was no correlation between the original image and noise.
[0101]
[Expression 28]

[0102]
In equation (35), a discrete-continuous conversion operator {d^k(R)}^tIs the same as the operator defined in Equation (16), and the matrix Q is defined in the same manner as Equations (19) and (20). Further, the matrix N is an autocorrelation matrix of noise, and noise generated in each light receiving element of the CCD image sensor 702 is uncorrelated and can be expressed by the following equation when the occurrence probability is equal between the light receiving elements.
[0103]
[Expression 29]

[0104]
Where α is a constant parameter representing noise variance, and I is a unit matrix of MN × MN. Based on the above analysis,
[30]

[0105]
However, matrix D^kIs the matrix D defined in equation (21)^kThe matrix Q is the same as the matrix Q defined in the equations (19) and (20). Sensitivity distribution function h is stored in ROM830._i(R, λ), (i = 1,..., M) and the transmittance distribution characteristic t of the wavelength bandpass filter^j(Λ), (j = 1,..., N) and the color matching function s of the primary color image^k(Λ), (k = R, G, B) and autocorrelation function R of the original image f (r, λ)_ff(R, λ; r ′, λ ′) is recorded. A parameter α is recorded in the ROM 706 in the camera 700. These data recorded in the ROMs 830 and 706 are input to the matrix calculator 831 so that the matrix D^kAnd (Q + αI)^aAre calculated and the elements are recorded in the coefficient memories 832-1 and 832-1, respectively. In the matrix calculator 804, the matrix D recorded in the ROM 832-1, 2 is stored.^kAnd (Q + αI)^aIs used to restore the primary color image based on the equation (37) with each wavelength band image recorded in the image memory 802-1 to 80-N.
[0106]
According to the fourth embodiment described above, a clear color image in which the influence of noise mixed in the input image is suppressed can be output. In addition, since the S / N parameters depending on the camera performance are held in the camera itself, when the camera is configured to be detachable from the image processor, optimum noise suppression processing adapted to the camera used is realized. it can.
[0107]
The fifth embodiment of the present invention will be described below. The fifth embodiment of the present invention inputs a multi-wavelength band image focused on a different object surface for each wavelength using the chromatic aberration of the optical imaging system, and uses them to focus on an arbitrary object surface. The present invention relates to a method for restoring a color image.
[0108]
FIG. 9 shows the configuration of the camera 900 and the image processor 1000 in the fifth embodiment. In the camera 900, the lens 901, the CCD image sensor 902, the rotation color filter 903, and the motor 904 are the same as the lens 101, the CCD image sensor 102, the rotation color filter 103, and the motor 104 in the camera 100 of the first embodiment, respectively. However, it is assumed that the lens 901 of the fifth embodiment has large chromatic aberration, and the optical imaging system has a wide in-focus range depending on the wavelength. In the image processor 1000, the A / D converter 1001, the image memories 1002-1 to N, the rotation color filter driver 1003, the matrix calculator 1004, the image memories 1006-1 to 3, the D / A converter 1007, and the CPU 1010 are A / D converter 201, image memories 202-1 to 202-N, rotation color filter driver 203, matrix calculator 204, image memories 206-1 to 206, D / A conversion, respectively, in image processor 200 of the first embodiment. The same as the device 207 and the CPU 210.
[0109]
The image processor 1000 further includes a ROM 1040 in which autocorrelation information corresponding to a plurality of types of subjects is recorded. The ROM 1041 stores sensitivity distribution functions corresponding to a plurality of object plane positions. One of a plurality of coefficient sequences recorded in the ROMs 1040 and 1041 is selected and input to the matrix calculator 1043. Which coefficient sequence is selected is a condition specified by the user through the user interface 1050. Is controlled by a read address signal sent from the CPU 1010 to the ROMs 1040 and 1041 based on the above.
[0110]
The ROM 1042 stores sensitivity distribution functions for a three-dimensional object space, transmittance distribution characteristics of N color band filters provided in the rotating color filter 903, and RGB color matching functions. This is input to the matrix calculator 1043. The matrix calculator 1043 performs a predetermined matrix calculation using the coefficient sequence sent from the ROMs 1040, 1041, and 1042, and records the two new calculated coefficient values in the coefficient memories 1044-1 and 104-2. . In the matrix calculator 1004, a predetermined matrix calculation is performed between the multi-wavelength band images recorded in the image memories 1002-1 to 100-N and the coefficient values recorded in the coefficient memories 1044-1 to 102-2, and the restoration is performed. The RGB color images thus recorded are recorded in the image memories 1006-1 to 1006-1.
[0111]
The operation of the fifth embodiment having the above-described configuration will be described below. In the fifth embodiment, when an optical imaging system with large chromatic aberration is used, a plurality of different wavelengths and focal positions are different while fixing the position of the lens by utilizing the fact that the object surface is different for each wavelength. Enter the image. Using them, the color image focused on the position of an arbitrary object plane within the focusing range corresponding to all input images is restored. In order to realize such processing, a color image restoration method using a correlation between wavelength band images is used based on a continuous-discrete system model of a multi-wavelength band with respect to a three-dimensional object space as described below.
[0112]
A specific calculation method is shown below. First, the continuous-discrete image input system for the three-dimensional object space is defined by the following equation as in the equation (1).
[0113]
[31]

[0114]
In the above equation, the spatial coordinate R = (x, y, z) is an orthogonal coordinate system representing the three-dimensional object space, and the sensitivity distribution function h_i(R, λ) and the original image f (R, λ) are redefined with respect to the three-dimensional object space. However, z is defined as an axis parallel to the optical axis of the optical imaging system.
[0115]
[Expression 32]

[0116]
However, the discrete-continuous conversion operator {w^k(R, z₀)}^tIs defined as follows:
[0117]
[Expression 33]

[0118]
However, the j-th element function [w^k(R, z₀]]^jIs defined as:
[0119]
[Expression 34]

[0120]
Where {h (r, z₀; Λ)} is z = z₀Sensitivity distribution function h on the two-dimensional object surface_i(R, z₀Λ), (i = 1,..., M) arranged in the column direction, a continuous-discrete system conversion operator, R_ff(R, λ; r ′, λ ′; z₀) Is z = z₀Two-dimensional original image f (r, λ; z₀) Autocorrelation function. On the other hand, the matrix Z in equation (39)^aIs a pseudo inverse of the matrix Z, but N × N block element matrices Z of the matrix Z_mn, (M, n = 1,..., N) are defined by the following equations, each having M × M elements.
[0121]
[Expression 35]

[0122]
However, R_ff(R, λ; R ′, λ ′): An autocorrelation function of the original image f (R, λ) defined in the three-dimensional continuous object space. Based on the above analysis,
[Expression 36]

[0123]
However, the matrix W in the equation (43)^kAre operator {p (r)} and operator {w^k(R)}^tThis is an M ′ × MN matrix that is defined again by performing a linear operation. The ROM 1040 has a plurality of three-dimensional autocorrelation functions R corresponding to the type of subject._ff(R, λ; R ′, λ ′), and types of objects and positions z of object surfaces₀2D autocorrelation function R corresponding to_ff(R, λ; R ′, λ ′; z₀) Is recorded. The ROM 1041 has a plurality of object plane positions z.₀2D sensitivity distribution function h corresponding to_i(R, z₀; Λ), (i = 1,..., M) are recorded.
[0124]
The user selects an appropriate condition from a plurality of types of coefficient sequences recorded in the ROMs 1040 and 1041 by selecting an appropriate condition according to the type of subject and the position of the object surface to be imaged. On the other hand, the ROM 1042 has a three-dimensional sensitivity distribution function h that is a coefficient sequence fixed by an image input / display system._i(R, λ), (i = 1,..., M), transmittance distribution characteristic t of the color bandpass filter^j(Λ), (j = 1,..., N) and RGB color matching function s^k(Λ) and (k = R, G, B) are recorded.
[0125]
The matrix calculator 1043 performs a predetermined matrix calculation using the coefficient sequences sent from the ROMs 1040, 1041, and 1042 as described above, and the coefficient memories 1044-1 and 1044 and 2 respectively store the matrix W.^k, Z^aThe element coefficient sequence is recorded. The matrix calculator 1004 performs a calculation according to the equation (43) between each wavelength band image recorded in the image memories 1002-1 to 100-N and the matrix recorded in the coefficient memories 1044-1 to 4-2. , RGB discrete primary color images are restored.
[0126]
According to the fifth embodiment described above, it is possible to realize an apparatus capable of controlling the position of a focused object surface within a predetermined object space without mechanical operation by lens driving. Therefore, since the lens driving unit can be omitted, the configuration is simple, and there is no influence of vibration by the motor. Moreover, a color image excellent in resolution and color reproducibility can be obtained by performing restoration processing based on the definition of a continuous-discrete image input system using multi-wavelength band optical imaging.
[0127]
The specific embodiments described above include inventions having the following configurations, and embodiments and effects corresponding to the respective inventions are as follows.
[0128]
(Configuration 1)
An imaging optical system;
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter by the imaging unit;
A coefficient memory in which a coefficient sequence used for matrix operation with N types of multi-wavelength area images recorded in the image memory is recorded;
A matrix operation is performed between the N types of multi-wavelength region images recorded in the image memory and the coefficient sequence recorded in the coefficient memory, and N ′ types of wavelength region images equal to or less than N Matrix computing means for outputting
An imaging apparatus comprising:
[0129]
(Corresponding Embodiment of the Invention)
The first embodiment described above corresponds to the embodiment according to the present invention. Further, the matrix calculation by the matrix calculation means is equivalent to calculating the expression (10) described in the first embodiment.
[0130]
(effect)
Considering optical imaging characteristics for each of N types of multi-wavelength bands, N ′ types of wavelength band images that are equal to or less than N are restored, so that redundancy is provided for the wavelength region. At the same time, the influence of chromatic aberration can be corrected in wavelength units. As a result, each image is accurately restored, and as a result, an image having excellent color reproducibility can be obtained.
[0131]
(Configuration 1.1)
In the imaging apparatus according to Configuration 1, the coefficient sequence recorded in the coefficient memory is a sensitivity representing a light intensity distribution with respect to an object space and a wavelength transmitted by the imaging optical system to each light receiving element in the imaging unit. Using the distribution function, the transmittance distribution characteristics of N types of color filters provided in the multicolor filter, and the wavelength sensitivity distribution of each of the N ′ types of output images output by the matrix calculation means, It is a pseudo inverse matrix for a generalized matrix in which an image input system determined by an imaging optical system, the multicolor filter, and the imaging means is linearly related.
It is characterized by that.
[0132]
(Corresponding Embodiment of the Invention)
The first embodiment described above corresponds to the embodiment according to the present invention.
[0133]
(effect)
Since the image input system is handled based on a general continuous-discrete system model, there is no need for the band / region limitation and the condition of position invariance of transfer characteristics required by the analysis based on the conventional sampling theorem. Wide application range and accurate analysis becomes possible. Accordingly, accurate image restoration can be performed even for an image input system in which the transfer characteristic changes depending on the position due to the influence of aberration of the optical imaging system. As a result, an optical imaging system such as a lens can be easily designed, and a color image excellent in resolution and color reproducibility can be obtained while using a small and light camera.
[0134]
(Configuration 1.2)
In the imaging apparatus according to Configuration 1, the image type N ′ output by the matrix calculation means is 3, and these images are images in the wavelength regions of red, green, and blue constituting a color image.
It is characterized by that.
[0135]
(Corresponding Embodiment of the Invention)
The first embodiment described above corresponds to the embodiment according to the present invention.
[0136]
(effect)
By making the output N′-type wavelength region images RGB color images, it is possible to output color images having affinity for general color image equipment.
[0137]
(Configuration 2)
An imaging optical system;
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter by the imaging unit;
A sensitivity distribution function representing the light intensity distribution with respect to the object space and wavelength transmitted by the imaging optical system for each light receiving element in the imaging means, and the transmittance of N types of color filters provided in the multicolor filter A first coefficient memory in which distribution characteristics and wavelength sensitivity distributions of N ′ types of output images equal to or less than N are recorded as a coefficient sequence;
A second coefficient memory in which statistical correlation information of a subject image defined as a function of object space coordinates and wavelength is recorded as a coefficient sequence;
Coefficient sequence calculation means for calculating a new coefficient sequence to be used for matrix calculation with N types of multi-wavelength area images recorded in the image memory from the coefficient sequences recorded in the first and second coefficient memories; ,
A matrix that performs matrix calculation between the N types of multi-wavelength region images recorded in the image memory and the new coefficient sequence calculated by the coefficient sequence calculation means, and outputs N ′ type wavelength region images. Computing means;
An imaging apparatus comprising:
[0138]
(Corresponding Embodiment of the Invention)
The second embodiment described above corresponds to the embodiment according to the present invention. The matrix calculation by the matrix calculation means corresponds to the calculation of equation (21) described in the second embodiment.
[0139]
(effect)
Even in the case where the imaging state differs for each wavelength due to the influence of chromatic aberration, the resolution and color reproducibility of the color image restored using the correlation between the wavelength band images can be improved. Therefore, even when the optical imaging system such as a lens has a simple configuration, an excellent color image can be obtained by resolution and color reproduction.
[0140]
(Configuration 2.1)
In the imaging device according to Configuration 2, the second coefficient memory includes a plurality of coefficient memories and means for reading a coefficient sequence from one of them.
It is characterized by that.
[0141]
(Corresponding Embodiment of the Invention)
The second embodiment described above corresponds to the embodiment according to the present invention.
[0142]
(effect)
A memory in which appropriate statistical correlation information is recorded can be selected in accordance with the type of subject, for example, a person, landscape, medical care, a building image, and the like, and a more accurate image can be restored.
[0143]
(Configuration 2.1.1)
In the imaging device according to Configuration 2.1, a plurality of coefficient memories in the second coefficient memory are built in a plurality of card media, and a means for reading a coefficient sequence from one of them is a card reader.
It is characterized by that.
[0144]
(Corresponding Embodiment of the Invention)
The second embodiment described above corresponds to the embodiment according to the present invention.
[0145]
(effect)
Since a card in which appropriate autocorrelation information is recorded can be selected, the quality of the restored image can be improved adaptively with respect to the type of subject.
[0146]
(Configuration 3)
An imaging optical system;
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter by the imaging unit;
A sensitivity distribution function representing the light intensity distribution with respect to the object space and wavelength transmitted by the imaging optical system for each light receiving element in the imaging means, and the transmittance of N types of color filters provided in the multicolor filter Coefficient sequence of distribution characteristics, wavelength sensitivity distribution of each of N ′ types of output images equal to or less than N, and statistical correlation information of a subject image defined as a function of object space coordinates and wavelength A recorded first coefficient memory;
A second coefficient memory that records a parameter representing the S / N of the imaging means;
A new value used for matrix calculation of N types of multi-wavelength area images recorded in the image memory from the coefficient sequence recorded in the first coefficient memory and the parameters recorded in the second coefficient memory. Coefficient sequence calculation means for calculating a correct coefficient sequence;
A matrix that performs matrix calculation between the N types of multi-wavelength region images recorded in the image memory and the new coefficient sequence calculated by the coefficient sequence calculation means, and outputs N ′ type wavelength region images. Computing means;
An imaging apparatus comprising:
[0147]
(Corresponding Embodiment of the Invention)
The fourth embodiment described above corresponds to the embodiment according to the present invention. Further, the matrix calculation by the matrix calculation means is equivalent to calculating the equation (37) described in the fourth embodiment.
[0148]
(effect)
A clear color image in which the influence of noise mixed in the input image is suppressed can be output.
[0149]
(Configuration 3.1)
In the imaging device according to Configuration 3, the parameter recorded in the second coefficient memory is a statistical amount of noise added to an output signal from the imaging means.
It is characterized by that.
[0150]
(Corresponding Embodiment of the Invention)
The fourth embodiment described above corresponds to the embodiment according to the present invention.
[0151]
(effect)
By recording noise statistics that depend on the performance of the imaging means in order to quantitatively handle noise that has no correlation with the input image generated in the imaging means, image restoration that can suppress noise under optimal conditions is performed. be able to.
[0152]
(Configuration 3.2)
In the imaging device according to Configuration 3, the second coefficient memory is built in the imaging unit.
It is characterized by that.
[0153]
(Corresponding Embodiment of the Invention)
The fourth embodiment described above corresponds to the embodiment according to the present invention.
[0154]
(effect)
Since the S / N parameter depending on the performance of the camera is held in the camera itself, even when the camera is configured to be detachable from the image processor, it is possible to realize an optimum noise suppression process adapted to the camera to be used. it can.
[0155]
(Configuration 4)
In the imaging device according to Configuration 2 or 3,
The coefficient sequence calculation means is recorded in the first coefficient memory on an image of N ′ types of wavelength regions output by the matrix calculation means and a subject image defined as a function of object space coordinates and wavelength. A coefficient is calculated such that an error from the subject image in the N ′ type wavelength region obtained by acting the wavelength sensitivity distribution of each of the N ′ type output images is minimized.
It is characterized by that.
[0156]
(Corresponding Embodiment of the Invention)
The second and third embodiments described above correspond to the embodiment according to the present invention.
[0157]
(effect)
It is possible to estimate and restore an image with the least error with respect to an ideal output image predicted to be obtained from the original image of the subject.
[0158]
(Configuration 5)
An imaging optical system that focuses on a different object surface for each wavelength; and
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter with the imaging unit;
For each light receiving element in the imaging means, a three-dimensional sensitivity distribution function representing a light intensity distribution with respect to a three-dimensional object space and wavelength transmitted by the imaging optical system, and for each light receiving element in the imaging means, A subject image defined as a function of a two-dimensional sensitivity distribution function representing a light intensity distribution with respect to a predetermined two-dimensional object plane and wavelength in a three-dimensional object space transmitted by the imaging optical system, and a function of three-dimensional object space coordinates and wavelength Provided in the multi-color filter, the statistical two-dimensional correlation information of the subject image defined as a function of a predetermined two-dimensional object plane coordinate in the three-dimensional object space and the wavelength, and A coefficient memory in which transmittance distribution characteristics of N types of color filters and wavelength sensitivity distributions of N ′ types of output images equal to or less than N are recorded as a coefficient sequence;
Coefficient sequence calculation means for calculating a new coefficient sequence used for matrix calculation with N types of multi-wavelength area images recorded in the image memory from the coefficient sequence recorded in the coefficient memory;
Matrix calculation means for performing matrix calculation between the N types of multi-wavelength region images recorded in the image memory and the coefficient sequence calculated by the coefficient sequence calculation means, and outputting N ′ type wavelength region images. When,
An imaging apparatus comprising:
[0159]
(Corresponding Embodiment of the Invention)
The fifth embodiment described above corresponds to the embodiment of the present invention. The matrix calculation by the matrix calculation means is equivalent to calculating the equation (43) described in the fifth embodiment.
[0160]
(effect)
When using an optical imaging system with large chromatic aberration, by utilizing the fact that the object surface is different for each wavelength, multiple images with different wavelengths and focal positions are input while fixing the lens position. Since an image focused on an arbitrary object plane within the focus range corresponding to the input image is restored, the position of the focused object plane can be controlled within a predetermined object space. Can be realized without mechanical operation by lens driving. Therefore, the lens driving unit can be omitted, the configuration is simplified, and the influence of vibration by the motor is eliminated.
[0161]
(Configuration 5.1)
In the imaging device according to Configuration 5, the coefficient string calculation unit is provided in the three-dimensional sensitivity distribution function recorded in the coefficient memory, statistical three-dimensional correlation information of the subject image, and the multicolor filter. Each of the pseudo inverse matrix of a matrix defined by linear combination with the transmittance distribution characteristics of N kinds of color filters, the two-dimensional sensitivity distribution function, the statistical two-dimensional correlation information of the subject image, and the N ′ kinds of output images And a matrix defined by a linear combination with the wavelength sensitivity distribution
It is characterized by that.
[0162]
(Corresponding Embodiment of the Invention)
The fifth embodiment described above corresponds to the embodiment of the present invention.
[0163]
(effect)
Restoration processing based on the definition of a continuous-discrete image input system using multiwavelength band optical imaging can be performed, and an image excellent in resolution and color reproducibility can be output.
[0164]
(Configuration 5.2)
In the imaging device according to Configuration 5, in the coefficient memory, light for a predetermined two-dimensional object plane and wavelength in a three-dimensional object space transmitted by the imaging optical system to each light receiving element in the imaging unit. The two-dimensional sensitivity distribution function representing the intensity distribution and the statistical two-dimensional correlation information of the subject image defined as a function of predetermined two-dimensional object plane coordinates and wavelength in the three-dimensional object space are made to correspond to different object planes. And a means for selecting one of them.
It is characterized by that.
[0165]
(Corresponding Embodiment of the Invention)
The fifth embodiment described above corresponds to the embodiment of the present invention.
[0166]
(effect)
It is possible to configure a device that can easily select the position of the object surface in focus.
[0167]
(Configuration 6)
In the imaging device according to any one of configurations 1, 2, 3, and 5, the multicolor filter is a mosaic color filter in which N types of color filters are spatially arranged on the same plane.
It is characterized by that.
[0168]
(Corresponding Embodiment of the Invention)
The third embodiment described above corresponds to the embodiment according to the present invention.
[0169]
(effect)
A color image excellent in resolution and color reproducibility can be obtained in accordance with the principle of image restoration based on a continuous-discrete system model while using a camera that inputs a multi-wavelength band image by a single plate method. Accordingly, it is possible to provide a practically useful color image capturing apparatus that is compact and has a short image input time.
[0170]
(Configuration 6.1)
In the imaging device according to Configuration 6, the number N of color filters provided in the mosaic color filter and the number N ′ of images output by the matrix calculation means are both 3.
It is characterized by that.
[0171]
(Corresponding Embodiment of the Invention)
The third embodiment described above corresponds to the embodiment relating to the present invention.
[0172]
(effect)
In general, since a color image is composed of three primary color images, it is possible to realize processing that is compatible with general color image equipment.
[0173]
(Configuration 6.1.1)
In the imaging device according to Configuration 6.1, the three types of color filters provided in the mosaic color filter are color filters that transmit red, green, and blue wavelength regions, and N ′ output from the matrix calculation unit. Types of images are images in the red, green, and blue wavelength regions that make up color images
It is characterized by that.
[0174]
(Corresponding Embodiment of the Invention)
The third embodiment described above corresponds to the embodiment according to the present invention.
[0175]
(effect)
The effects of the present invention can be obtained while using a generally used single-panel color camera or color image equipment.
[0176]
【The invention's effect】
According to the present invention, an accurate color image with excellent color reproducibility can be restored using an image restoration method based on a continuous-discrete system model.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention.
FIG. 2 is a diagram showing a cross-sectional structure of a rotating color filter.
FIG. 3 is a diagram illustrating a transmittance distribution characteristic of a color filter.
FIG. 4 is a diagram illustrating wavelength characteristics of a color matching function.
FIG. 5 is a diagram illustrating a configuration of an image processor according to a second embodiment of the present invention.
FIG. 6 is a diagram illustrating a configuration of a camera and an image processor in a third embodiment of the present invention.
FIG. 7 is a diagram illustrating a configuration of a mosaic color filter.
FIG. 8 is a diagram illustrating a configuration of a camera and an image processor in a fourth embodiment of the present invention.
FIG. 9 is a diagram illustrating a configuration of a camera and an image processing processor according to a fifth embodiment of the present invention.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 100 ... Camera, 101 ... Lens, 102 ... CCD image sensor, 103 ... Rotation color filter, 104 ... Motor, 201 ... A / D converter, 202-1-N ... Image memory, 203 ... Rotation color filter driver, 204 ... Matrix calculator, 205... ROM, 206-1 to 3... Image memory, 207... D / A converter, 210.

Claims (12)

An imaging optical system;
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter by the imaging unit;
A coefficient memory in which a coefficient sequence used for matrix operation with N types of multi-wavelength area images recorded in the image memory is recorded;
A matrix operation is performed between the N types of multi-wavelength region images recorded in the image memory and the coefficient sequence recorded in the coefficient memory, and N ′ types of wavelength region images equal to or less than N Matrix computing means for outputting
Equipped with,
The coefficient sequence recorded in the coefficient memory is:
A sensitivity distribution function representing the light intensity distribution with respect to the object space and wavelength transmitted by the imaging optical system for each light receiving element in the imaging means;
Transmittance distribution characteristics of N types of color filters provided in the multicolor filter;
The wavelength sensitivity distribution of each of the N ′ types of output images output by the matrix calculation means;
An imaging matrix characterized by being a pseudo inverse matrix for a generalized matrix obtained by linearly relating an image input system determined by the imaging optical system, the multicolor filter, and the imaging means. apparatus. An imaging optical system;
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter by the imaging unit;
A coefficient memory in which a coefficient sequence used for matrix operation with N types of multi-wavelength area images recorded in the image memory is recorded;
A matrix operation is performed between the N types of multi-wavelength region images recorded in the image memory and the coefficient sequence recorded in the coefficient memory. Matrix computing means for outputting an image in the wavelength region of
An imaging apparatus comprising: An imaging optical system;
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter by the imaging unit;
A sensitivity distribution function representing the light intensity distribution with respect to the object space and wavelength transmitted by the imaging optical system for each light receiving element in the imaging means, and the transmittance of N types of color filters provided in the multicolor filter and distribution characteristic, the first coefficient memory which records and wavelength sensitivity distribution of each small N 'type of output image from equal N or N as the coefficient sequence,
A second coefficient memory in which statistical correlation information of a subject image defined as a function of object space coordinates and wavelength is recorded as a coefficient sequence ;
Coefficient sequence calculation means for calculating a new coefficient sequence to be used for matrix calculation with N types of multi-wavelength area images recorded in the image memory from the coefficient sequences recorded in the first and second coefficient memories ; ,
A matrix that performs matrix calculation between the N types of multi-wavelength region images recorded in the image memory and the new coefficient sequence calculated by the coefficient sequence calculation means, and outputs N ′ type wavelength region images. Computing means;
An imaging apparatus comprising: 4. The imaging apparatus according to claim 3, wherein the second coefficient memory includes a plurality of coefficient memories and means for reading a coefficient sequence from one of them. 5. The imaging apparatus according to claim 4, wherein the means for reading the coefficient sequence selects a coefficient memory in which statistical correlation information appropriate for the type of the subject is recorded according to the type of the subject. apparatus. 5. The imaging apparatus according to claim 4, wherein the plurality of coefficient memories in the second coefficient memory are built in a plurality of card media, and the means for reading a coefficient sequence from one of them is a card reader. An imaging device. An imaging optical system;
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter by the imaging unit;
A sensitivity distribution function representing the light intensity distribution with respect to the object space and wavelength transmitted by the imaging optical system for each light receiving element in the imaging means, and the transmittance of N types of color filters provided in the multicolor filter Coefficient sequence of distribution characteristics, wavelength sensitivity distribution of each of N ′ types of output images equal to or less than N, and statistical correlation information of a subject image defined as a function of object space coordinates and wavelength A recorded first coefficient memory;
A second coefficient memory that records a parameter representing the S / N of the imaging means;
A new value used for matrix calculation of N types of multi-wavelength area images recorded in the image memory from the coefficient sequence recorded in the first coefficient memory and the parameters recorded in the second coefficient memory. Coefficient sequence calculation means for calculating a correct coefficient sequence;
Matrix calculation is performed between the N types of multi-wavelength region images recorded in the image memory and the new coefficient sequence calculated by the coefficient sequence calculation means, and an N ′ type wavelength region image is output. Means,
An imaging apparatus comprising: 8. The imaging apparatus according to claim 7, wherein the parameter recorded in the second coefficient memory is a statistical amount of noise added to an output signal from the imaging means. An imaging optical system that focuses on a different object surface for each wavelength; and
A multicolor filter composed of N types of color filters that transmit an optical image of an object formed by the imaging optical system for each of N types of wavelength regions;
Imaging means for imaging a light image of an object incident through the imaging optical system and the multicolor filter;
An image memory for recording N types of multi-wavelength region images obtained by capturing an optical image of an object of N types of wavelength regions transmitted by the multi-color filter with the imaging unit;
For each light receiving element in the imaging means, a three-dimensional sensitivity distribution function representing a light intensity distribution with respect to a three-dimensional object space and wavelength transmitted by the imaging optical system, and for each light receiving element in the imaging means, A subject image defined as a function of a two-dimensional sensitivity distribution function representing a light intensity distribution with respect to a predetermined two-dimensional object plane and wavelength in a three-dimensional object space transmitted by the imaging optical system, and a function of three-dimensional object space coordinates and wavelength Statistical second-order of the subject image defined as a function of the statistical three-dimensional correlation information and the predetermined two-dimensional object plane coordinates and wavelength in the three-dimensional object space A coefficient sequence including original correlation information, transmittance distribution characteristics of N types of color filters provided in the multicolor filter, and wavelength sensitivity distributions of N ′ types of output images equal to or less than N Coefficient memory recorded as
Coefficient sequence calculation means for calculating a new coefficient sequence used for matrix calculation with N types of multi-wavelength area images recorded in the image memory from the coefficient sequence recorded in the coefficient memory;
Matrix calculation means for performing matrix calculation between the N types of multi-wavelength region images recorded in the image memory and the coefficient sequence calculated by the coefficient sequence calculation means, and outputting N ′ type wavelength region images; ,
An imaging apparatus comprising: 10. The imaging apparatus according to claim 9, wherein the coefficient sequence calculation means is provided in a three-dimensional sensitivity distribution function recorded in the coefficient memory, statistical three-dimensional correlation information of a subject image, and the multicolor filter. Pseudo inverse matrix of a matrix defined by linear combination with transmittance distribution characteristics of N kinds of color filters, two-dimensional sensitivity distribution function, statistical two-dimensional correlation information of subject image, and N ′ kinds of output images And a matrix defined by linear combination with the wavelength sensitivity distribution of the imaging device. 10. The imaging device according to claim 9, wherein the coefficient memory has a predetermined two-dimensional object plane and wavelength in a three-dimensional object space transmitted by the imaging optical system for each light receiving element in the imaging unit. 2D sensitivity distribution function representing light intensity distribution and object surface statistical 2D correlation information defined as a function of predetermined 2D object surface coordinates and wavelength in 3D object space And an image pickup apparatus having means for selecting one of the plurality of images recorded therein. 10. The imaging apparatus according to claim 1, claim 3, claim 7, or claim 9, wherein the types of images output by the matrix calculation means are N 'less than N and 3, and these images are color. An imaging apparatus characterized by being an image in the red, green, and blue wavelength regions constituting the image.

Images