JP4175832B2 - Wide-angle image generator - Google Patents

Description

ここで角度αは、内部焦点４と任意点Ｐを結ぶ直線とＸＹ平面のなす角度である。角度βは、主点７（双曲面反射鏡１の外部焦点と略々一致）と撮像面５上の点ｐを結ぶ直線とＸＹ平面のなす角度である。
【０１４０】
（式９）よりＸ_P，Ｙ_P，Ｚ_Pが決まればαが決まる。また（式１０）よりαとｂ、ｃが決まればβが決まる。
全方位カメラ１００では、次に（式１１）を用いることによって、Ｆの値が決まれば（ｘ²＋ｙ²）^0.5、すなわち、広角画像上の原点から点ｐまでの距離がわかった。しかし（実施の形態５）では、撮像面５がγだけ傾いているために（式１１）では正しく（ｘ²＋ｙ²）^0.5の値を求めることはできない。従ってこのままでは正しく展開処理が行えない。
【０１４１】
そこで（実施の形態５）では（式１３）を用いる。
（ｘ²＋ｙ²）^0.5 ＝ Ｆ・cosβ／cosγ／sin（β＋γ） （式１３）
図１７を用いて（式１３）の導出法について説明する。
【０１４２】
図１７は図１６広角撮像装置８の撮像面５付近を拡大したものである。図１７において、ＹＺ平面内にあり、Ｙ軸に平行で、撮像面５上の原点ｏを通る直線と任意点Ｐからの光線の交点をｑとする。この時、∠ｐｑｏ＝β、∠ｐｏｑ＝γとなる。また線分ｏｑの長さ|ｏｑ|は（式１４）で示される。
【０１４３】
|ｏｑ| ＝ Ｆ・tan(90°−β)／cosγ ＝ Ｆ／tanβ／cosγ （式１４）
従って、△ｏｐｑにおける正弦定理により|ｏｐ|即ち（ｘ²＋ｙ²）^0.5は（式１３）で求められる。

ここでＦは結像レンズ２と撮像素子３からなる撮像装置の焦点距離である。
【０１４４】
（式９）（式１０）（式１３）よりＸ_P，Ｙ_P，Ｚ_Pおよびｂ、ｃ、Ｆが決まれば（ｘ_p ²＋ｙ_p ²）^0.5、即ち撮像面５上の原点ｏから点ｐまでの距離が決まる。以上の議論はＺ軸を含むＹＺ平面以外の平面でも成立するが、Ｚ軸を含むＹＺ平面以外のどの平面をとるかに応じて撮像面５の傾きγは変化するという点に注意しなければならない。この点について図１８を用いて説明する。
【０１４５】
図１８はＺ軸方向から見た時の双曲面反射鏡１と撮像面５を示している。
３５は仮想投影面３４上の任意の点Ｐ（図１８では図示せず）から発せられ、双曲面反射鏡１で反射して撮像面５上の点ｐに到達する光線を示している。図１８中に示したように角度θを、Ｘ軸の−方向を０度とし、左周りに＋として定義する。例えば光線３５がＹＺ平面内にある場合、θ＝９０度となる。この時のθと撮像面５の傾きγ（θ）の関係は（式１５）のようになる。
【０１４６】
tanγ（θ） ＝ tanγ（９０°）・sinθ （式１５）
ここでγ（９０°）とはθ＝９０°の時のγを意味する。また（式１３）は（式１３ａ）のようになる。
【０１４７】

以上の議論で、Ｚ軸を含む各平面での原点ｏから点ｐまでの距離が得ることができた。次にＸＹ平面での原点Ｏから点Ｐへの方向θと、ｘｙ平面、すなわち撮像面５上での原点ｏから点ｐへの方向φとの関係について考える。ここでφは図１８に示すようにＸ軸の＋方向を０度とする撮像面５上の点の方向を規定するための角度パラメータである。
【０１４８】
全方位カメラ１００の場合はθ＝φであり、ＸＹ平面およびｘｙ平面における点Ｐ、点ｐの方向は一致していた。しかし広角撮像装置８の場合はＸＹ平面に対して撮像面５がγ（９０°）だけ傾いているため、θとφの関係は（式１６）のようなものとなる。
【０１４９】
cosφ ＝ cosθ／（１＋tan²γ（９０°）・sin²θ）^0.5 （式１６）
図１９を用いて（式１６）の導出法について説明する。
図１９のＸａ，Ｙａ平面はＸＹ平面をＺ軸を中心として１８０度回転し、Ｘａ軸がｘ軸と一致するように平行移動したものである。ここでθはＸａ軸の−方向を０度とし、Ｘａ，Ｙａ平面上で時計回りを＋方向としており、ＸＹ平面のθと等価なパラメータとなる。このため、ここではパラメータ名を変更せず、θのまま用いる。またφはｘ軸の−方向を０度とし、ｘｙ平面上で時計回りを＋方向としている。このように設定し、θとφとの関係を導出する。円弧ＣＡＤは単位円の半分である。またＣＢＤは、Ｘａ，Ｙａ平面との交線が半円ＣＡＤとなり、Ｘａ，Ｙａ平面と垂直であるような半円柱とｘｙ平面との交線である。この時ベクトルｏＡ，ｏＢは（式１７）（式１８）のようになる。
【０１５０】
ｏＡ ＝（−cosθ、sinθ、０） （式１７）
ｏＢ ＝（−cosθ、sinθ、tanγ（９０°）・sinθ） （式１８）
従って、−ｘ方向の単位ベクトル−Ｘとの内積を用いることにより（式１６）が求まる。
【０１５１】

以上より、β、γ（＝γ（９０°））、Ｆがわかれば、（式１３ａ）（式１５）（式１６）を用いることによってＰ(Ｘ_P，Ｙ_P，Ｚ_P)に対応する撮像面５上の点（即ち広角画像上の点）ｐ（ｘ_p，ｙ_p）を一意に決定することができる。Ｘ_P，Ｙ_P，Ｚ_Pの値は、仮想投影面３４の半径をＭ、展開画像の横方向(長手方向)画素数をISX、縦方向(短手方向)画素数をISYとすると以下のように表せる。
【０１５２】
Ｘ_P ＝ Ｍ・cosθ
Ｙ_P ＝ Ｍ・sinθ
Ｚ_P ＝ ２π・Ｍ・(ｊ−ｊ0)／ISX （式１９）
但し、θ＝180°・ｉ／ISX、ｉは展開画像上のX座標値で０≦ｉ＜ISXの整数、ｊは展開画像上のＹ座標値で０≦ｊ＜ISYの整数、またｊ₀はＺ_P＝ｃの時(視線方向が水平方向の場合)の展開画像上のＹ座標値である。（式１９）を用いることにより展開画像上の点と仮想投影面３４の点Ｐ（Ｘ_P，Ｙ_P，Ｚ_P）を対応づけることができる。
【０１５３】
従って、展開画像の投影面（仮想投影面３４）を決め、投影面から広角画像への写像を考えることによって、展開画像の各画素に対する広角画像上の理想的な対応点を決定することができる。
【０１５４】
次に図２０を用いて図１５のステップＳ３における結像レンズ２および撮像素子３からなる撮像装置の光学的な歪の影響を補正する方法について詳しく説明する。
【０１５５】
図２０は光学的な歪の影響を模式的に示したものである。光学的な歪がなく理想的な画像が得られるとすれば、画角が１８０度の場合、図２０の（ａ）に示すような半円状の像が得られる。しかし、実際には光学系の中に光学的な歪が含まれており、例えば図２０の（ｂ）に示すように半円状の像が歪んで得られてしまう。このため、この光学歪を考慮に入れずに展開処理を行うと得られる展開画像は正しいものにならない。
【０１５６】
そこで（実施の形態５）では、予め使用する光学系の歪量を把握し、その影響をキャンセルするようにステップＳ２で作成した展開テーブルを補正する。
まず、歪量の把握方法について簡単に説明する。
【０１５７】
光学的な歪を数式で記述するための一般的な方法として高次多項式がよく用いられる。ここでは（式２０）に示すような多項式を用いた場合を考える。
Δｘ＝ｋｘ（ｘ²＋ｙ²）
Δｙ＝ｋｙ（ｘ²＋ｙ²） （式２０）
ここで、Δｘ，Δｙはそれぞれｘ，ｙ方向の歪量。ｋは歪量を調整するパラメータである。（式２０）は歪量を示す式としては非常に単純なものであり、設定するパラメータはｋのみである。実際の光学系に適用した場合、（式２０）では光学歪量を記述するのに不十分な場合もある。その場合は適宜、高次項を考慮したものを使用すればよい。
【０１５８】
結像レンズ２および撮像素子３からなる撮像装置で基準となる（即ち、歪量を測定するために、距離が既知であるようなパターンが含まれている）テストパターン３６を撮像する。テストパターン３６としては例えば図２１の（ａ）に示すような２次元格子の描かれたものを用いても良い。ここで例えば格子ピッチの正確な値が既知であればよい。その結果得られたテストパターン画像３７は、図２１の（ｂ）に示すように光学歪の影響により、元のテストパターン３６とは異なったパターンとなる。格子ピッチと撮像の条件から算出できる理想的な格子点の座標値と、テストパターン画像３７から得られた、対応する格子点の座標値を複数個（ｎ個）比較することにより、歪量のサンプルをとる。ｎ個とったサンプルのうちのｉ番目のデータの理想の座標値をｘ（ｉ），ｙ（ｉ）、その時のｘ方向の実測歪量をΔｘ（ｉ）、ｙ方向の実測歪量をΔｙ（ｉ）とした時に（式２１)の２つの誤差関数が最小となるようなｋを最小自乗法により求める。
【０１５９】
Ｅx ＝ Σ（Δｘ(ｉ)−ｋｘ(ｉ) (ｘ(ｉ)²＋ｙ(ｉ)²)）
Ｅy ＝ Σ（Δｙ(ｉ)−ｋｙ(ｉ) (ｘ(ｉ)²＋ｙ(ｉ)²)） （式２１）
以上の処理により、撮像装置の歪量を示す（式２０）が決定する。次にこれを用いて、ステップＳ２で求めた理想的な展開テーブルを補正する。展開テーブルのｘ方向、ｙ方向それぞれの各画素（各要素）には、光学歪を考慮しない場合の各画素に対応する広角画像の画素値が含まれている。これを（ｘ₀（Ｘ，Ｙ），ｙ₀（Ｘ，Ｙ））とする。ここで（Ｘ，Ｙ）は今対象としている展開テーブルの画素（要素）の座標値である。この値を（式２０）に代入し、補正後の値（ｘ₁（Ｘ，Ｙ）、ｙ₁（Ｘ，Ｙ））を算出する。具体的には（式２２）を各画素（要素）に対して行う。
【０１６０】

この処理を展開テーブルの全ての画素（要素）に行うことにより撮像装置の光学的な歪の影響を補正した展開テーブルを得ることができる。従ってこのような展開テーブルを用いて展開処理を行えば、撮像装置の光学歪の影響を排除した展開画像を得ることができる。
【０１６１】
また、このような光学歪補正処理は一旦展開画像を作成した後にその展開画像に対して行うことも可能であるが、その場合展開画像を作成するたびに（式２２）に示すような複雑な演算処理を行わなければならない。しかるに（実施の形態５）では、光学歪の補正処理を展開テーブルに盛り込むことにより、（式２２）のような複雑な演算処理を、展開テーブル生成部２３における初期の展開テーブル作成時のみ行うだけでよく、展開画像作成時は通常の展開処理を行うだけでよいことから、演算コストの面からも非常にメリットがある。
【０１６２】
次に展開画像生成部２４について説明する。
展開画像生成部２４では展開テーブル生成部２３で生成された展開テーブルを参照して、広角撮像装置８から得られる広角画像を展開画像に変換する。この際、座標値が整数値である展開画像の各画素に対応する広角画像の画素位置は、一般にその座標値が整数値とならない。しかし、広角画像は整数の座標位置のデータしか存在しない。この対策として、画質よりも処理速度を優先する場合には、展開テーブルの座標値データを四捨五入などの手段により、整数化することにより、本来必要な画素の近傍の画素の輝度値で代用してもよい。処理速度よりも画質を優先する場合には、本来必要な画素の近傍４画素の輝度データから補間演算により、本来必要な画素の輝度データを作り出してもよい。
【０１６３】
次に画質補正処理部２５について説明する。
画質補正処理部２５では、展開画像生成部２４で作成された展開画像に対して、画質補正処理を行う。以下にこの画質補正処理について説明する。
【０１６４】
展開画像生成部２４で生成された展開画像は、画像内の上部と下部とで解像度が異なってしまうという問題点がある。すなわち、図２２に示すように、広角画像３８とそれに展開処理を施すことによって得られる展開画像３９との対応関係において、展開画像３９の上端下端の関係が、広角画像３８の外周内周の関係となるため、情報量の差から解像度に差が生じる。そのため、展開画像３９において、外周部に対応した部分よりも内周部に対応した部分の方がピントがボケたように見えてしまう。また他の要因として、結像レンズ２と撮像素子３からなる撮像装置には被写界深度があり、わずかではあるが、半円形状の広角画像３８の半径方向にピントのムラ（例えば、広角画像３８の内周部分ではピントが良く合っているが、外周部分ではピントがややあまくなるといったようなムラ）が生じる場合もある。展開画像３９として最終的に得られる画像としては、上記の２種類の要因を重ね合わせた結果のボケが含まれたものとなる。
【０１６５】
このような展開画像３９に対して画質補正処理部２５では、展開画像３９の上下方向（即ち元画像である広角画像３８の半径方向に対応）に沿って、その場所ごとに相対的な鮮鋭化係数を決め、その係数を用いて画像の高周波数成分の強調による鮮鋭化処理を行う。
【０１６６】
鮮鋭化処理としては、例えばアンシャープマスキングと呼ばれる手法を用いる。これは（式２３）のように表現される。
Ｂ´(ｉ,ｊ) ＝ Ｂ(ｉ,ｊ) ＋Ｃ（ Ｂ(ｉ,ｊ)−Ｂ￣ (ｉ,ｊ) ） （式２３）
ここで、Ｂ(ｉ,ｊ)は座標(ｉ,ｊ)の画素の輝度値とする。Ｂ￣(ｉ,ｊ)は座標(ｉ,ｊ)の画素の近傍の局所平均輝度値（例えば近傍３×３の範囲の平均輝度値）とし、低周波成分からなる輝度値となる。従って（ Ｂ(ｉ,ｊ)−Ｂ￣(ｉ,ｊ) ）は元の画像から低周波成分を差し引くことで高周波成分を取り出す操作となる。Ｃは鮮鋭化係数とする。高周波成分をＣ倍したものを元画像に足し合わせることにより、高周波数成分を強調した鮮鋭化画像が得られる。
【０１６７】
鮮鋭化係数Ｃを展開画像３９の上下方向の位置に応じて変化させ、鮮鋭化処理を行う。まず撮像装置で双曲面反射鏡１上の像を理想的にボケなく撮像できる場合について説明する。
【０１６８】
この場合に得られる展開画像３９は図２３（ａ）に示すように下端部が、元の広角画像の内周部と外周部の情報量の差に起因するボケ領域４０となる。ここでボケ領域４０は一様にボケているわけではなく、展開画像３９の中央付近から下端にいくにつれてボケていく。この時、図２３（ａ）にグラフで示したように鮮鋭化係数Ｃをボケ具合に合わせて０から変化させることにより、ボケ具合に応じた補正処理（鮮鋭化処理）を行う。
【０１６９】
なお、ここでは展開画像３９の上下方向のほぼ中央付近から鮮鋭化係数Ｃの値を増やしていくような場合のグラフを示したが、これに限定されるものではない。本発明の主旨は下端に近づくにつれて悪化していくボケ具合に応じて鮮鋭化係数Ｃの値を増やしていくことにあり、例えば鮮鋭化係数Ｃを０よりも大きくする場所は展開画像３９の上端であっても構わない。また鮮鋭化係数Ｃの増やし方もボケ具合や撮影する対象などに応じて調整可能であり、例えば鮮鋭化係数Ｃを０よりも大きくする場所からの距離の３乗に比例させて増やしてもよい。
【０１７０】
次に、撮像装置で双曲面反射鏡１上の像を撮像する際に、場所によってピントの合い具合に差が生じてしまうような場合について図２３（ｂ）を用いて説明する。
【０１７１】
この場合は撮像装置のピントを調整することにより、得られる展開画像３９のボケ領域を調整する。鮮鋭化処理は画像の高周波成分を強調する処理であるため、対象となる画像がある程度以上ボケてしまうと処理を行っても有意な効果を発揮できなくなる。そこで出来るだけ最もボケ状態の悪い（ボケのひどい）部分をできるだけ改善するようにピントを調整する。具体的には、広角画像３８の内周部に光学的なピントのボケ部分が来るようにピントを調整すると、内周部分にボケ状態を悪くする条件が重なってしまい、補正しきれなくなる。それを避けるために、広角画像３８の半径方向中央よりもやや内周部分にピントのピークが来るように調整し、最終的に展開画像３９において上下端のボケ領域４０が図２３（ｂ）のようにほぼバランスがとれるようにピントを調整する。この時もボケ領域４０は一様にボケているわけではなく、展開画像３９の中央付近から上下端にいくにつれてボケていく。この時、図２３（ｂ）にグラフで示したように鮮鋭化係数Ｃをボケ具合に合わせて、展開画像３９の中央付近から上下端にいくにつれて０から変化させることにより、ボケ具合に応じた補正処理（鮮鋭化処理）を行う。
【０１７２】
次に画像マスク処理部２６について説明する。
画質補正処理部２５で生成された画質補正後の展開画像に対して画像マスク処理を行う画像マスク処理部２６は、次のように構成されている。
【０１７３】
図３０（ａ）はマスク処理を行わない場合の展開画像を、図３０（ｂ）はマスク処理を行った場合の展開画像を示している。展開画像中の４１は現金自動支払機（ＡＴＭ）端末、４２は操作者、４７はＡＴＭ端末４１の操作パネルである。マスク処理前の図３０（ａ）では全視野が見えており、操作者４２の操作や入力結果などが映ってしまうため、暗証番号などのプライベートな情報が外部に漏れてしまう危険性がある。そこで画像マスク処理部２６ではＲＯＭに記憶したマスク領域４８を例えば輝度値０で塗りつぶしてしまうような処理を行うことにより、図３０（ｂ）に示すように操作者４２の操作や入力結果などが見えないようにする。
【０１７４】
なお、ここでは輝度値０に塗りつぶすこととしたが、これに限定されるものではない。マスク領域４８内の画像を本来の画像と異なるもので埋めることができればよく、０以外の輝度値、あるいはランダムパターン画像を描いてもよい。
【０１７５】
以上のような処理を施して生成された展開画像は図示しない画像形式変換部でＮＴＳＣのようなビデオ信号に変換され、画像処理装置２２から出力される。モニタ２７はその信号を受け取り、展開画像を表示する。
【０１７６】
このように（実施の形態５）によれば、広角撮像装置８の光学系を撮像レンズ２および撮像素子３からなる撮像装置の光軸２８を回転中心軸６に対して傾けた構成としても、展開画像の各画素に対する広角画像上の理想的な対応点を決定することができ、さらに撮像装置の光学的な歪の影響を考慮にいれた展開テーブルの補正を行っているため、正しい展開処理が可能となる。また画質補正処理を行うことにより、展開画像の場所による解像度の差を軽減することができる。
【０１７７】
また、画像マスク処理部２６によって、展開画像の一部にマスク処理を施し、見えないようにしてから出力することにより、操作者の操作内容・入力内容が写らなくなり、プライバシー情報の漏洩を防ぐことが可能になる。
【０１７８】
なお、この（実施の形態５）では、回転対称体の回転中心軸に対して垂直な任意の方向に基準光軸をとり、前記基準光軸の方向を０度として略±９０度もしくは±９０度以上の範囲の凸面形状の反射面を有する反射鏡として、回転中心軸上に内部焦点を持つ双曲面形状であり、前記双曲面形状の外部焦点の位置と前記撮像装置の主点の位置が略々一致するものであったが、この反射鏡の形状は双曲面でなくても成立する。
【０１７９】
また、（実施の形態５）では、広角撮像装置８が回転中心軸周り略±９０度分の範囲を撮像するような場合を示したが、広角撮像装置８として図３４の（ａ）で示したような回転中心軸周り略３６０度分の範囲を撮像する全方位カメラ１００を用い、画像処理装置２２で全方位カメラ１００に対応した展開処理を行うような場合でも、これまでに説明した光学歪の影響補正や画質補正処理、画像マスク処理は、（実施の形態５）の場合と同様の効果を有する。
【０１８０】
（実施の形態６）
図２４〜図３１は本発明の（実施の形態６）を示す。
図２４は本発明の（実施の形態６）における広角画像生成装置を示す。
【０１８１】
座標軸を、紙面の上下方向にＸ軸、紙面から手前方向にＹ軸、紙面の左右方向にＺ軸とする。またＺ軸は水平方向とする。（実施の形態６）と（実施の形態５）との構成上の相違点は、広角撮像装置８をＺ軸が水平方向と一致するように設置する点である。処理上の相違点は画像処理装置２２の展開テーブル生成部２３において、展開テーブル生成の際に用いる仮想投影面３４の形状に特徴を持たせる点と、画像マスク処理の具体的な処理が異なる点と、展開画像生成部２４で（実施の形態５）の場合に対して９０度傾け、画像内の天地が正しくなるような展開画像を生成する点である。以下にこれらの相違点について説明する。
【０１８２】
まず結像のしくみについて説明する。
図２４において光線ａ，ｂ，ｃは対象物からの任意の光線を示している。そこで各光線の対象物を対象物ａ，ｂ，ｃと呼ぶことにする。対象物ａ，ｂ，ｃは、図２５（ａ）に示すようなスクリーン３３上に描かれた「ａ」「ｂ」「ｃ」の文字とし、このスクリーン３３を、図２５（ｂ）に示すようにＺ軸周りにＺ軸からスクリーン表面までの距離が一定となるように、対象物ａ、ｂ、ｃが描かれた面を双曲面反射鏡１から見えるように配設した場合を考える。この時に撮像素子３で得られる広角画像を図２６に示す。対象物ａ、ｂ、ｃが全て含まれることが分かる。
【０１８３】
このように広角撮像装置８を回転中心軸６が水平方向と一致するように設置する場合の用途例として、ＡＴＭ端末や自動販売機などの監視がある。通常の広角光学系では実現困難な鉛直方向１８０度程度の視野が得られることから、本装置のそばに立った人の顔から手元までを一枚の画像として得ることが出来るため、ＡＴＭ端末や自動販売機などの監視用途への適用が可能である。
【０１８４】
次に画像処理装置２２での処理について主に（実施の形態５）との相違点について説明する。
まず（実施の形態５）の場合と同様に、展開処理をはじめる前の初期段階で展開テーブル生成部２３にて展開処理に用いる展開テーブルの算出を行う。展開テーブル生成部２３で行う処理のフローを、図２７を用いて説明する。図２７で示したフローで行われる処理の中で（実施の形態５）の場合と異なる２点のうちの１点は、広角画像と投影面との間の理想的な対応点を決定するステップＳ２において、仮定する仮想投影面３４の形状を変更する点である。この点を図２８を用いて詳しく説明する。
【０１８５】
図２８はＡＴＭ端末４１を操作する操作者４２と広角撮像装置８および仮想投影面３４との関係を示している。３４は、その回転中心軸が広角撮像装置８の双曲面反射鏡１の内部焦点４を通り、紙面に垂直であるような円筒状の仮想投影面であり、従来はこのような投影面に投影した画像の再生を考えていた。ＡＴＭ端末４１の操作者４２はＡＴＭ端末４１を使用する時に通常１５〜３０度程度前方に傾斜角４３をとる。広角撮像装置８の視点となる内部焦点４からの距離は胸部に比べて、顔部の方が遠くなるため、得られる展開画像内で胸部付近は大きく、また顔部にいくほど小さく写り、見た目に違和感を生じさせてしまう。
【０１８６】
そこで（実施の形態６）では、図２８に示すように、得られる視野の中心方向となる基準光軸方向４６と反対の方向にその中心軸４５を設定した新しい仮想投影面４４を設定する。この新しい仮想投影面４４への投影を考えることは、広角撮像装置８の視点となる内部焦点４を、仮想的に基準光軸方向４６と反対方向に中心軸４５の位置まで移動することに相当する。そのため、操作者４２がＡＴＭ端末４１を使用している際の、広角撮像装置８の視点に対する顔部の位置と胸部の位置の距離の差が緩和される。
【０１８７】
具体的な仮想投影面形状の変更方法について図２９を用いて説明する。
図２９は図２８と同様にＺ軸方向から見た従来の仮想投影面３４と形状を変更した仮想投影面４４を示している。θは仮想投影面４４上の点Ｐから広角撮像装置８の図示しない内部焦点４への光線のＸＹ平面上における方位角をＸ軸方向を０度として設定したものである。αは仮想投影面４４上の点Ｐから仮想投影面４４の中心軸４５上の図示しない仮想の内部焦点（即ち内部焦点４を通る基準光軸と中心軸４５の交点）への光線のＸＹ平面上における方位角をＸ軸方向を０度として設定したものである。また図２９上で内部焦点４の位置をＯ₁、中心軸４５の位置をＯ₂とする。
【０１８８】
（実施の形態５）と同様に（式９）（式１０）（式１３ａ）（式１５）（式１６）を用いて仮想投影面上の点Ｐ（Ｘ_P，Ｙ_P，Ｚ_P）に対応する広角画像の点ｐ（ｘ_p，ｙ_p）を決定するのであるが、この時の（実施の形態５）との相違点は（式１６）でθの与え方である。（実施の形態５）では、特に展開画像の横方向(長手方向)画素数をISX、横方向の視野角を１８０度とした時にθ＝180°・ｉ／ISX（但しｉは整数で０≦ｉ＜ISX）のようにθを等間隔に取っていけばよい。（実施の形態６）ではαを等間隔で取っていった場合と等価となるようにθを決定する。△ＰＯ₁Ｏ₂において正弦定理より（式２４)が成り立つ。
【０１８９】
Ｒ／sin(９０°＋θ)＝ｒ／sin(９０°−α) （式２４）
（式２４）を整理すると（式２５）が得られる。
θ＝cos^-1（Ｒ・cosα／ｒ） （式２５）
但し、α₁≦α＜α₂
よって、（式２５）のαにα＝(α₂−α₁)・ｉ／ISX（但しｎは整数で０≦ｉ＜ISX）を代入してθを求め、そのθを用いて展開テーブルを作成すれば、仮想投影面形状を変更することができる。
【０１９０】
なお、ここでは仮想投影面３４として円筒を考えたが、球面など他の形状であっても構わない。
次に図２７で示したフローで行われる処理の中で（実施の形態５）の場合と異なる２点のうちのもう１点である、光学歪の影響補正を行うステップＳ３の次に行うマスク処理（ステップＳ４）について図３０を用いて説明する。
【０１９１】
図３０（ａ）はマスク処理を行わない場合の展開画像を、図３０（ｂ）はマスク処理を行った場合の展開画像を示している。展開画像中の４１はＡＴＭ端末、４２は操作者、４７はＡＴＭ端末４１の操作パネルである。マスク処理前の図３０（ａ）では全視野が見えており、操作者４２の操作や入力結果などが写ってしまうため、暗証番号などのプライベートな情報が外部に漏れてしまう危険性がある。ここではマスク領域４８に相当する部分の展開テーブルの座標値データを、本来のものと異なるもの、例えばマスク領域４８に相当する部分の全ての展開テーブル要素を、広角画像中で双曲面反射鏡１からの像が結像されない左上隅などを指定するように置き換えてしまうことによりマスク処理を実現する。
【０１９２】
以上のように展開テーブルにマスク処理の情報を埋め込むことにより、図２４の展開画像生成部２４にて展開処理をする際に同時に、演算負荷を増やすことなくマスク処理を行うことができる。
【０１９３】
次に展開画像生成部２４では、以上のような処理を行うことにより、展開テーブル生成部２３で生成された展開テーブルを用いて展開画像の生成を行う。この際の（実施の形態５）との相違点は、既に図３０で示したように展開画像の天地が正しくなるように、縦方向が画角１８０度となるように展開処理を行うことである。
【０１９４】
次に画質補正処理部２５では、（実施の形態５）の場合と同様の処理を行い、図示しない画像形式変換部でＮＴＳＣのようなビデオ信号に変換してモニタ２７に展開画像を表示する。
【０１９５】
このように（実施の形態６）によれば、仮想投影面４４の形状を変更することにより仮想的に広角撮像装置８の視点を変更することができるため、広角撮像装置８の内部焦点４からの距離の差に起因する展開画像の違和感を軽減することが可能になる。また、展開画像の一部にマスク処理を施し、見えないようにしてから出力することにより、操作者の操作内容・入力内容が写らなくなり、プライバシー情報の漏洩を防ぐことが可能になる。
【０１９６】
なお、ここでは、マスク処理の情報を展開テーブルに埋め込む方法を示したが、画像マスク処理を、展開画像を生成した後で行っても構わない。例えば図３１に示すように、画質補正処理部２５の直後に画像マスク処理部４９を設け、ここでマスク領域４８内部を輝度値０に塗りつぶすような処理を行っても構わない。
【０１９７】
なお、（実施の形態６）では、双曲面反射鏡１を±９０度分の反射面を有するものとしたが、これに限定されるものではない。±１８０度以下の範囲であればよい。
【０１９８】
また、（実施の形態６）では、広角撮像装置８が回転中心軸周り略±９０度分の範囲を撮像するような場合を示したが、広角撮像装置８として図３４の（ａ）で示したような回転中心軸周り略３６０度分の範囲を撮像する全方位カメラ１００を用い、画像処理装置２２で全方位カメラ１００に対応した展開処理を行うような場合でも、これまでに説明した仮想投影面形状の変更やマスク処理は、（実施の形態６）の場合と同様の効果を有する。また全方位カメラで仮想投影面形状の変更を行った場合は、１枚の広角画像を用いて視点の移動を表現可能であることから、撮像場所を少しづつずらした複数枚の広角画像を使って滑らかな視点移動が可能なウォークスルーソフトを実現できる。
【０１９９】
（実施の形態７）
図３２は本発明の（実施の形態７）を示す。
（実施の形態６）との相違部分は、画像処理装置２２内の処理単位の一つであった展開テーブル生成部２３を分離独立させ、新たに展開テーブル生成用演算処理装置５１を設けた点と、画像処理装置２２内に展開テーブル生成部２３の代わりに展開テーブル記憶部５０を設けた点である。
【０２００】
この図３２の動作を、（実施の形態６）との相違部分を中心に説明する。
広角撮像装置８は（実施の形態６）で用いたものと同様のものである。
展開テーブル生成用演算処理装置５１は、具体的にはマイクロコンピュータやパーソナルコンピュータ（ＰＣ）などの演算装置であり、ここで（実施の形態６）の展開テーブル生成部２３で行っていた展開テーブルの生成を行う。生成された展開テーブルは画像処理装置２２に送られ、展開テーブル記憶部５０に記憶される。
【０２０１】
展開テーブルの転送方法は、例えばＳＤカード（Secure Digitalカード 登録商標）などのメモリーカード５２を用いる。展開テーブル生成用演算処理装置５１と画像処理装置２２の双方に図示しないＳＤカードスロットを設けておく。展開テーブル生成用演算処理装置５１にて、生成した展開テーブルをＳＤカードに記録する。そのＳＤカードを画像処理装置２２のＳＤカードスロットに挿入し、ＳＤカード内に記録された展開テーブルを読み取り、展開テーブル記憶部５０に記憶する。
【０２０２】
次に、展開画像生成部２４は展開テーブル記憶部５０に記憶された展開テーブルを読み込み展開処理を行う。以降の処理は（実施の形態６）と同様である。
このように（実施の形態７）によれば、画像処理装置２２での展開テーブル生成処理の必要がなくなるため、装置起動時の展開テーブル生成処理のための起動時間をなくすことができる。
【０２０３】
なお、（実施の形態７）では、（実施の形態６）の画像処理装置２２内の処理単位の一つであった展開テーブル生成部２３を分離独立させ、新たに展開テーブル生成用演算処理装置５１を設け、画像処理装置２２内に展開テーブル生成部２３の代わりに展開テーブル記憶部５０を設けた場合を示した。同様の変更を（実施の形態５）の場合に行った場合でも（実施の形態７）の場合と同様の効果を得ることができる。また、どちらの場合でも、広角撮像装置８として図３４の（ａ）で示したような回転中心軸周り略３６０度分の範囲を撮像する全方位カメラ１００を用い、画像処理装置２２および展開テーブル生成用演算処理装置５１で全方位カメラ１００に対応した処理を行うような変更を行っても、（実施の形態７）と同様の効果を得ることができる。
【０２０４】
【発明の効果】
以上のように、（実施の形態１）〜（実施の形態４）を例に挙げて説明した本発明によれば、広角撮像装置をその回転中心軸が基準光軸方向に垂直な面内で鉛直方向に対して傾けた状態としても、広角画像生成装置としての出力は画像内の水平方向がＴＶモニタの水平方向と一致した画像となり、見る人に違和感を与えないものとすることが可能となる。
【０２０５】
また、広角撮像装置を、その回転中心軸と基準光軸を含む面内で、ある基準状態に対して傾けた状態としても、その傾き角を計測し、その値を利用して展開画像の投影面となる仮想円筒面の中心軸を常に基準状態の回転中心軸と一致させることにより、展開画像における視線中心の位置を変えることなく視野範囲を変更することが可能となる。さらに、視野範囲全体を使って、視線中心が展開画像の短手方向の中央にあるような展開画像を得ることができる。この為、展開画像中の任意の点の視線中心からの最大距離を最小とすることができ、展開画像中の視線中心から遠い部分は、他の部分よりも斜めに見ることになるために、他の部分に比べて歪んで見えてしまうという問題を軽減できる。
【０２０６】
また、（実施の形態５）〜（実施の形態７）を例に挙げて説明した本発明によれば、広角撮像装置の光学系として、撮像レンズおよび撮像素子からなる撮像装置の光軸を回転中心軸に対して傾けた構成としても、展開画像の各画素に対する広角画像上の理想的な対応点を決定することができ、さらに撮像装置の光学的な歪の影響を考慮にいれた展開テーブルの補正を行っているため、正しい展開処理が可能となる。また画質補正処理を行うことにより、展開画像の場所による解像度の差を軽減することができる。
【０２０７】
また、仮想投影面の形状を変更することにより仮想的に広角撮像装置の視点を変更することができるため、広角撮像装置の内部焦点からの距離の差に起因する展開画像の違和感を軽減することが可能になる。また、展開画像の一部にマスク処理を施し、見えないようにしてから出力することにより、操作者の操作内容・入力内容が写らなくなり、プライバシー情報の漏洩を防ぐことが可能になる。
【０２０８】
さらに、画像処理装置とは別個に展開テーブル生成用演算処理装置を用いる構成とすれば、画像処理装置での展開テーブル生成処理の必要がなくなるため、装置起動時の展開テーブル生成処理のための起動時間をなくすことができる。
【図面の簡単な説明】
【図１】本発明の（実施の形態１）の広角画像生成装置の構成図
【図２】同実施の形態の対象物ａ，ｂ，ｃの説明図
【図３】同実施の形態の広角画像生成装置において撮像面上に結ばれる像の模式図
【図４】同実施の形態の広角画像生成装置により撮像された画像に対して展開処理を施した結果の模式図
【図５】同実施の形態のＴＶモニタに出力される出力画像の模式図
【図６】広角撮像装置の回転とその時の出力画像の模式図
【図７】本発明の（実施の形態２）の広角画像生成装置の構成図
【図８】本発明の（実施の形態３）の広角画像生成装置をＸ軸方向（鉛直上方）から見た構成図
【図９】本発明の（実施の形態４）の広角画像生成装置をＸ軸方向（鉛直上方）から見た構成図
【図１０】本発明の（実施の形態５）の広角画像生成装置の構成図
【図１１】同実施の形態における広角撮像装置の説明図
【図１２】同実施の形態における広角撮像装置の撮像面上に結ばれる像の模式図
【図１３】同実施の形態における対象物ａ，ｂ，ｃの説明図
【図１４】同実施の形態にて得られる広角画像の説明図
【図１５】同実施の形態における展開テーブル作成フロー図
【図１６】同実施の形態の広角撮像装置の構成図
【図１７】同実施の形態における撮像面付近の拡大図
【図１８】同実施の形態においてＺ軸方向から見た場合の双曲面反射鏡と撮像面の配置図
【図１９】Ｘａ，Ｙａ面上の角度θとｘｙ面上の角度φの関係の説明図
【図２０】撮像装置の光学的な歪の影響の説明図
【図２１】テストパターン画像の説明図
【図２２】展開画像の左右の解像度差の説明図
【図２３】展開画像上の位置と鮮鋭化係数の関係の説明図
【図２４】本発明の（実施の形態６）の広角画像生成装置の構成図
【図２５】同実施の形態における対象物ａ，ｂ，ｃの説明図
【図２６】同実施の形態における撮像面上に結ばれる像の模式図
【図２７】同実施の形態における展開テーブル作成フロー図
【図２８】仮想投影面と操作者との関係を示す説明図
【図２９】仮想投影面形状の変更法についての説明図
【図３０】ＡＴＭ操作パネルをマスクした場合の模式図
【図３１】同実施の形態における広角画像生成装置の構成図
【図３２】本発明の（実施の形態７）の広角画像生成装置の構成図
【図３３】展開処理の説明図
【図３４】全方位カメラの説明図
【図３５】全方位カメラにおける展開処理の説明図
【符号の説明】
１ 双曲面反射鏡
２ 結像レンズ
３ 撮像素子
４ 双曲面反射鏡１の内部焦点
５ 撮像素子３の撮像面
６ 双曲面反射鏡１の回転中心軸
７ 結像レンズ２の主点
８ 広角撮像装置
９ 展開画像生成部
１０ 出力画像生成部
１１ 演算処理装置
１２ Ｙ軸周り角度センサ（回転中心軸面内傾斜方向検出部）
１３ ＴＶモニタ
１４ スクリーン
１５ 回転中心軸面内傾斜方向入力部
１６ Ｘ軸周り角度センサ
１７ 回転中心軸前方傾斜方向入力部
１８ 回転中心軸を傾けた時の広角撮像装置８の視野限界
１９ 回転中心軸を傾けた時の広角撮像装置８の視野限界
２０ 回転中心軸を傾けた時の広角撮像装置８の視野範囲
１８´ 基準状態の広角撮像装置８の視野限界
１９´ 基準状態の広角撮像装置８の視野限界
２０´ 基準状態の広角撮像装置８の視野範囲
２１ 仮想円筒面
２２ 画像処理装置
２３ 展開テーブル生成部
２４ 展開画像生成部
２５ 画質補正処理部
２６ 画像マスク処理部
２７ モニタ
２８ 光軸
２９ａ，２９ｂ 光線
３０ａ，３０ｂ 入射点
３１ａ，３１ｂ，３１ａ₀，３１ｂ₀ 結像点
３２，３２₀ 半円状の像
３３ スクリーン
３４ 仮想投影面
３５ 光線
３６ テストパターン
３７ テストパターン画像
３８ 広角画像
３９ 展開画像
４０ ボケ領域
４１ ＡＴＭ端末
４２ 操作者
４３ 操作者の上半身の傾斜角
４４ （実施の形態６）の仮想投影面
４５ （実施の形態６）の仮想投影面の中心軸
４６ 基準光軸方向
４７ ＡＴＭ端末の操作パネル
４８ マスク領域
４９ 画像マスク処理部
５０ 展開テーブル記憶部
５１ 展開テーブル生成用演算処理装置
５２ メモリーカード[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a wide-angle image generation apparatus that captures reflected light from a reflecting surface and generates a wide-angle image.
[0002]
[Prior art]
As a conventional example of a wide-angle imaging device using a reflecting surface, there is a so-called omnidirectional camera described in Japanese Patent No. 2939087. In this case, an image in the direction of 360 degrees around the rotation axis can be formed on the imaging surface at a time by the reflection surface having a rotationally symmetric shape around the optical axis of the camera. Also, a so-called panoramic image is obtained by performing image conversion processing on an omnidirectional image obtained by imaging.
[0003]
The omnidirectional camera is effective when the monitoring area is 360 degrees around. However, when the monitoring area may be approximately 180 degrees, the omnidirectional camera captures an unnecessary area corresponding to approximately 180 degrees, which is inefficient as an image sensor.
[0004]
Therefore, consider a wide-angle imaging device that captures only about ± 90 degrees of a reflection image formed by a reflection surface with an arbitrary direction perpendicular to the rotation center axis of the reflection surface as a reference direction.
[0005]
As with the omnidirectional camera disclosed in Japanese Patent No. 2939087, an image obtained by such a wide-angle imaging device (hereinafter referred to as a wide-angle image) is subjected to arithmetic processing to obtain a normal image. It is possible to obtain a developed image that is an image obtained by stitching together images taken by a corner camera.
[0006]
The expansion process will be described below.
FIG. 33 schematically shows the wide-angle imaging device 8 for explaining the expansion process. The coordinate axes are assumed to be the X axis in the forward direction from the page, the Y axis in the left and right direction of the page, and the Z axis in the vertical direction of the page. The Z axis coincides with the vertical direction.
[0007]
The wide-angle imaging device 8 includes a hyperboloidal reflecting mirror 1, an imaging lens 2, and an imaging element 3 having a shape that is axially symmetric. The imaging device includes an imaging lens 2 and an imaging element 3.
[0008]
The hyperboloid reflecting mirror 1 has a reflecting surface for ± 90 degrees around the rotation center axis of the hyperboloid. The direction of 0 degree is set as a reference optical axis direction, and here, it coincides with the Y-axis direction. Reference numeral 4 denotes an internal focal point of the hyperboloidal reflecting mirror 1. Reference numeral 6 denotes a rotation center axis of the hyperboloid reflector 1, which coincides with the Z axis.
[0009]
A light beam emitted from an arbitrary point P (X, Y, Z) in the space and incident on the internal focal point 4 is reflected by the reflecting surface of the hyperboloidal reflecting mirror 1 and is externally focused on the rotation center axis 6 (see FIG. (Not shown). The position of the external focus and the position of the principal point 7 of the imaging lens 2 are arranged so as to substantially coincide with each other, and a light beam forms an image on the imaging surface 5 of the imaging device 3 via the imaging lens 2.
[0010]
Reference numeral 21 denotes a virtual cylindrical surface that is a virtual projection surface when considering mapping from a wide-angle image to a developed image. The rotation center axis of the virtual cylindrical surface 21 coincides with the rotation center axis 6 of the hyperboloid reflector 1.
[0011]
Consider a point (namely, a point on a wide-angle image) p (x, y) on the imaging surface 5 corresponding to an arbitrary point P (X, Y, Z) in space. The hyperboloid shape of the hyperboloid reflector 1 is expressed by the following (formula 1).
[0012]
(X²+ Y²/ A²-Z²/ B²  = -1 (Formula 1)
Here, a and b are constants that determine the shape of the hyperboloid. Also
c = (a²+ b²)^0.5                                (Formula 2)
It is. Moreover, Formula 3-Formula 5 are materialized.
[0013]
Z = (X²+ Y²)^0.5・ Tanβ + c (Formula 3)
tanβ = {(b²+ c²) · Sin α-2bc} / {(b²-C²) · Cosα} (Formula 4)
tanα = F / (x²+ y²)^0.5                        (Formula 5)
Here, F is a focal length of an image pickup apparatus including the imaging lens 2 and the image pickup device 3. See FIG. 33 for angles α and β.
[0014]
If X, Y, Z and b, c, F are determined from (Expression 3) to (Expression 5), (x²+ y²)^0.5That is, the distance from the intersection with the rotation center axis 6 on the imaging surface 5 to the point p is determined.
In addition, the directions of the points P and p on the XY plane and the xy plane coincide with each other due to the property of the hyperboloid that the light beam traveling toward the internal focal point 4 is reflected by the hyperboloidal reflecting mirror 1 and travels toward the external focal point. Therefore,
Y / X = y / x (Formula 6)
If X and Y are obtained from (Expression 6), the direction of the point p is also determined, and accordingly, a point on the imaging surface 5 corresponding to P (X, Y, Z) (that is, a point on the wide-angle image) p (x , Y) can be uniquely determined. That is, even in a wide-angle imaging device that captures a field of view of approximately ± 90 degrees, as in the case of an omnidirectional camera, the projection plane (virtual cylindrical surface 21) of the developed image is determined, and the wide-angle image is projected onto the projection plane. By considering the mapping, the corresponding point in the wide-angle image of each pixel of the developed image is determined, and the developed image can be generated by determining the luminance values of all the pixels of the developed image accordingly.
[0015]
However, if the central axis of rotation of the reflecting surface is tilted from the vertical direction in a plane perpendicular to the reference direction, this tilt is ignored and arithmetic processing is performed on the image obtained from the wide-angle imaging device. When a wide-angle image of approximately ± 90 degrees is converted into a developed image, the outer frame of the developed image is the same as when the rotation center axis of the reflecting surface is not inclined. On the other hand, the contents of the developed image are inclined by the inclination of the rotation center axis of the reflecting surface, which causes a problem that the viewer feels uncomfortable.
[0016]
Further, the field of view in the direction of the rotation axis of such a wide-angle imaging device is generally | β | <| γ | when the elevation angle is β and the depression angle is γ (see FIG. 33). Therefore, the position where the elevation angle in the developed image is 0 (hereinafter referred to as the line-of-sight center) is shifted to one of the short sides of the developed image. At this time, a portion far from the line-of-sight center in the developed image is viewed obliquely as compared with other portions, so that there is a problem that the portion appears to be distorted as compared with other portions.
[0017]
The omnidirectional camera will be described with reference to FIG.
FIG. 34A shows the basic configuration of an omnidirectional camera. The omnidirectional camera 100 includes a hyperboloid reflecting mirror 101 as a reflecting surface having a rotationally symmetric shape, an imaging lens 102, and an image sensor 103 such as a CCD.
[0018]
Reference numeral 104 denotes an internal focal point of the hyperboloid reflecting mirror 101. Reference numeral 105 denotes an imaging surface of the image sensor 103. Reference numeral 106 denotes a rotation center axis of the hyperboloidal reflecting mirror 101, and here it is assumed to coincide with the Z axis. Further, an arbitrary direction perpendicular to the rotation center axis of the hyperboloid reflecting mirror 101 is defined as a Y-axis direction. Reference numeral 107 denotes a principal point of the imaging lens 102. A point that bisects a straight line connecting the internal focal point 104 and the principal point 107 is defined as an origin O of the coordinate system.
[0019]
Reference numerals 108a and 108b denote light rays incident on the hyperboloidal reflecting mirror 101, and reference numerals 109a and 109b denote incident points of the light rays 108a and 108b with respect to the hyperboloidal reflecting mirror 101. 110a and 110b are imaging points on the imaging surface 105 of the light beams 108a and 108b reflected by the incident points 109a and 109b.
[0020]
Rays 108 a and 108 b that are emitted from a certain point in the YZ plane, enter the internal focal point 104, and reach the incident points 109 a and 109 b on the hyperboloid reflector 101 are reflected by the reflecting surface of the hyperboloid reflector 101. Then, the light is condensed to an external focal point (not shown) on the rotation center axis 106. The position of the external focus and the position of the principal point 107 of the imaging lens 102 are arranged so as to substantially coincide with each other, and the light rays 108 a and 108 b are connected to the imaging surface 105 of the imaging element 103 via the imaging lens 102. An image is formed on the image points 110a and 110b.
[0021]
FIG. 34B shows a state of image formation on the imaging surface 105. As shown in the drawing, the upward direction on the paper is the Y-axis direction. The direction perpendicular to the paper surface is the Z-axis direction. A light beam (not shown) including the light rays 108a and 108b and incident on the hyperboloidal reflector 101 from the 360-degree rotation center axis 106 of the omnidirectional camera 100 toward the internal focal point 104 is the same as the light rays 108a and 108b. In addition, a circular image 111 is formed on the imaging surface 105 via the imaging lens 102 through an external focal point (not shown) of the hyperboloidal reflecting mirror 101. Further, both ends in the Y direction of the circular image 111 become image forming points 110a and 110b of the light beams 108a and 108b.
[0022]
An expansion process for converting the circular omnidirectional image obtained as described above into a panoramic image will be described below.
FIG. 35 schematically shows the omnidirectional camera 100 for explaining the development process. The hyperboloid reflecting mirror 101, the imaging lens 102, the image sensor 103, the internal focal point 104, the imaging surface 105, the rotation center axis 106, and the principal point 107 are the same as those described with reference to FIG.
[0023]
Reference numeral 112 denotes a virtual cylindrical surface which is a virtual projection surface when considering mapping from a circular omnidirectional image to a developed image. The rotation center axis of the virtual cylindrical surface 112 coincides with the rotation center axis 106 of the hyperboloid reflector 101. The coordinate axes are the X axis in the front direction from the plane of the paper, the Y axis in the horizontal direction of the plane, and the Z axis in the vertical direction of the plane. In addition, the Z axis coincides with the rotation center axis 106.
[0024]
As already explained, an arbitrary point P (X_P, Y_P, Z_P) And incident on the internal focal point 104 are reflected by the reflecting surface of the hyperboloidal reflecting mirror 101 and are collected to an external focal point (not shown) on the rotation center axis 106. The position of the external focus and the position of the principal point 107 of the imaging lens 102 are arranged so as to substantially coincide with each other, and a light beam forms an image on the imaging surface 105 of the imaging element 103 via the imaging lens 102.
[0025]
Arbitrary point P (X_P, Y_P, Z_P) Corresponding to a point on the imaging surface 105, that is, a point on the circular omnidirectional image p (x_p, Y_p)think of. The hyperboloid shape of the hyperboloid reflector 101 is expressed by (Equation 7).
[0026]
(X²+ Y²) / A²-Z²/ B²= -1 (Formula 7)
Here, a and b are constants that determine the shape of the hyperboloid. Also
c = (a²+ b²)^0.5                                  (Formula 8)
It is. Further, (Equation 9) to (Equation 11) are established.
[0027]
Z = (X²+ Y²)^0.5・ Tanα + c (Formula 9)
tanα = {(b²+ c²) · Sinβ-2bc} / {(b²-C²) · Cosβ} (Formula 10)
tanβ = F / (x²+ y²)^0.5                        (Formula 11)
Here, F is a focal length of an image pickup apparatus including the imaging lens 102 and the image pickup element 103. The angle α is an angle formed by a straight line connecting the internal focal point 104 and the arbitrary point P and the XY plane. The angle β is an angle formed between the principal point 107 (substantially coincident with the external focal point of the hyperboloid reflector 101) and a straight line connecting the point p on the imaging surface 105 and the XY plane. From (Formula 9) to (Formula 11) X_P, Y_P, Z_PAnd b, c, and F are determined (x_p ²+ Y_p ²)^0.5That is, the distance from the intersection with the rotation center axis 106 on the imaging surface 105 to the point p is determined.
[0028]
In addition, the directions of the points P and p on the XY plane and the xy plane coincide with each other because the ray toward the internal focal point 104 is reflected by the hyperbolic reflecting mirror 101 and travels toward the external focal point. Therefore
Y / X = y / x (Formula 12)
X from (Equation 12)_P, Y_PIs obtained, the direction of the point p is also determined.
[0029]
From the above, P (X_P, Y_P, Z_P) Corresponding to a point on the imaging surface 5 (that is, a point on a circular omnidirectional image) p (x_p, Y_p) Can be determined uniquely. Therefore, by determining the projection plane (virtual cylindrical surface 112) of the developed image and considering the mapping from the circular omnidirectional image to the projection plane, the corresponding points in the circular omnidirectional image for each pixel of the developed image are determined. Then, the developed image can be generated by determining the luminance values of all the pixels of the developed image accordingly.
[0030]
The omnidirectional camera is effective when the monitoring area is 360 degrees around. However, when the monitoring area may be approximately 180 degrees, the omnidirectional camera captures an unnecessary area corresponding to approximately 180 degrees, which is inefficient as an image sensor.
[0031]
[Problems to be solved by the invention]
In the present invention, even when the wide-angle imaging device is tilted with respect to the vertical direction in the plane whose rotation center axis is perpendicular to the reference optical axis direction, the output as the wide-angle image generating device is that the horizontal direction in the image is the TV monitor. It is an object of the present invention to provide a wide-angle image generating apparatus that matches the horizontal direction of the image and does not give a strange feeling to the viewer.
[0032]
Further, the present invention contemplates a wide-angle image generation apparatus that captures only about ± 90 degrees of a reflection image formed by a reflection surface with an arbitrary direction perpendicular to the rotation center axis of the reflection surface as a reference direction. In this case, if a developed image is obtained by applying the same processing as in the case of the omnidirectional camera to a wide-angle image, the image is obtained due to the deformation of the image due to the optical axis inclination or optical aberration of the imaging device. There is a problem that the developed image is not correct, and an object of the present invention is to provide a wide-angle image generation apparatus that can solve this problem.
[0045]
[Means for Solving the Problems]
  Claim 1 of the present inventionThe wide-angle image generation apparatus takes a reference optical axis in an arbitrary direction perpendicular to the rotation center axis of the rotationally symmetric body, and takes the direction of the reference optical axis as 0 degree and at least an angle θ range (θ1 ≦ θ ≦ θ2). , −180 ° ≦ θ1 ≦ 180 °, −180 ° ≦ θ2 ≦ 180 °, θ1 <θ2), a reflecting mirror having a principal point on the rotation center axis of the reflecting mirror, When the field angle is 2α, the optical axis is inclined in the direction of the reference optical axis by an angle of α or less with respect to the rotation center axis, and a reflected image of the reflecting mirror is formed to rotate the reflecting mirror. An imaging device that captures an object in a range of at least an angle θ around a central axis, a developed image generation unit that generates a developed image viewed from the rotation central axis from a wide-angle image obtained from the imaging device, and the developed image The wide-angle image and the pixel of the developed image when the generation unit develops the wide-angle image Characterized by providing the expansion table generating unit that generates a deployment table containing information response relationship.
[0046]
  Claims of the invention2The described wide-angle image generation apparatus has a reference optical axis in an arbitrary direction perpendicular to the rotation center axis of the rotationally symmetric body, and the direction of the reference optical axis is 0 degree, and at least an angle θ range (θ1 ≦ θ ≦ a reflecting mirror having a convex reflecting surface of θ2, −180 ° ≦ θ1 ≦ 180 °, −180 ° ≦ θ2 ≦ 180 °, θ1 <θ2), and a principal point on the rotation center axis of the reflecting mirror When the angle of view is 2α, the optical axis is inclined in the direction of the reference optical axis by an angle of α or less with respect to the rotation center axis, and a reflected image of the reflecting mirror is formed. An imaging device that captures an object in the range of at least an angle θ around the rotation center axis, a development image generation unit that generates a development image viewed from the rotation center axis from a wide-angle image obtained from the imaging device, and the development When the image generation unit develops the wide-angle image, the wide-angle image and the developed image An expansion table generating arithmetic processing device that generates an expansion table including information on elementary correspondences, and an expansion that holds the expansion table generated by the expansion table generation arithmetic processing device and is connected to the expanded image generation unit A table storage unit is provided.
[0049]
  Of the present inventionClaim 3The wide-angle image generating apparatus described in claim 1.Or claim 2The reflecting mirror has a hyperboloid shape having an internal focal point on a rotation center axis, and the position of the external focal point of the hyperboloid shape and the position of the principal point of the imaging device are substantially the same. .
[0053]
  Of the present inventionClaim 4The wide-angle image generating apparatus described in claim 1.Or claim 2In this case, the expansion table generation unit or the expansion table generation processing processing unit is used to set the correspondence between the wide-angle image including distortion due to the optical aberration of the imaging device and the wide-angle image after distortion correction, and the wide-angle image and the developed image after distortion correction. A development table including pixel correspondence information obtained by combining the correspondence relationships is generated.
[0054]
  Of the present inventionClaim 5The wide-angle image generating apparatus described in claim 1.Or claim 2In the expansion table generation unit or the expansion table generation processing processing device, the pixel correspondence between the wide-angle image and the developed image is changed by changing the shape of the virtual projection plane when considering the pixel correspondence relationship between the wide-angle image and the developed image. The present invention is characterized in that an expanded table having a changed relationship is generated.
[0055]
  Of the present inventionClaim 6The wide-angle image generating apparatus described in claim 1.Or claim 2In the above, the expansion table generation unit or the expansion table generation processing unit arbitrarily changes the pixel correspondence relationship between the wide-angle image and the expansion image in an arbitrary mask area provided on the expansion image, and the expansion image in the mask area It is characterized by generating a development table that masks.
[0056]
  The wide-angle image generating device according to claim 7 of the present invention is the post-processing of the developed image generating unit, the post-processing of the developed image generating unit according to claim 1 or claim 2 according to the position on the developed image. An image quality correction processing unit for performing high frequency component emphasis processing with different values is added.
  According to an eighth aspect of the present invention, there is provided a wide-angle image generating apparatus having a reflecting mirror having a convex reflecting surface and a principal point on a rotation center axis of the reflecting mirror, and forming a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;PartA development table including information on correspondence between a wide-angle image including distortion due to optical aberration of the imaging apparatus and a wide-angle image after distortion correction, and pixel correspondence information obtained by synthesizing the correspondence between the wide-angle image after distortion correction and the development image It is characterized by having produced | generated.
  According to a ninth aspect of the present invention, there is provided a wide-angle image generating apparatus having a reflecting mirror having a convex reflecting surface and a principal point on the rotation center axis of the reflecting mirror, and forming a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;PartThe configuration is such that, by changing the shape of the virtual projection plane when considering the pixel correspondence between the wide-angle image and the developed image, a development table in which the pixel correspondence between the wide-angle image and the developed image is changed is generated. And
  The wide-angle image generating apparatus according to claim 10 of the present invention has a reflecting mirror having a convex reflecting surface, a principal point on the rotation center axis of the reflecting mirror, and forms a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;PartThe pixel correspondence relationship between the wide-angle image and the developed image in an arbitrary mask area provided on the developed image is arbitrarily changed, and a development table that masks the developed image in the mask area is generated. Features.
  The wide-angle image generation device according to claim 11 of the present invention has a reflecting mirror having a convex reflecting surface, a principal point on the rotation center axis of the reflecting mirror, and forms a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation unit that generates a development table that includes information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image, and a subsequent stage of the development image generation unit Further, as a post-processing of the developed image generation unit, an image quality correction processing unit that performs a high frequency component enhancement process in which the degree of enhancement is changed according to the position on the developed image is added.
  The wide-angle image generating apparatus according to claim 12 of the present invention has a reflecting mirror having a convex reflecting surface, a principal point on the rotation center axis of the reflecting mirror, and forms a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation arithmetic processing unit that generates a development table including information on a pixel correspondence relationship between the wide angle image and the development image when the development image generation unit develops the wide angle image; A development table storage unit that holds the development table generated by the processing device and is connected to the development image generation unit, andExhibitionPixels obtained by synthesizing the correspondence between the wide-angle image including distortion due to optical aberration of the imaging device and the wide-angle image after distortion correction, and the correspondence between the wide-angle image after distortion correction and the developed image The present invention is characterized in that a development table including correspondence information is generated.
  The wide-angle image generating device according to claim 13 of the present invention has a reflecting mirror having a convex reflecting surface and a principal point on the rotation center axis of the reflecting mirror, and forms a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation arithmetic processing unit that generates a development table including information on a pixel correspondence relationship between the wide angle image and the development image when the development image generation unit develops the wide angle image; A development table storage unit that holds the development table generated by the processing device and is connected to the development image generation unit, andExhibitionA development table in which the pixel correspondence between the wide-angle image and the developed image is changed by changing the shape of the virtual projection plane when the open table generation arithmetic processing unit considers the pixel correspondence between the wide-angle image and the developed image. It is configured to generate.
  According to a fourteenth aspect of the present invention, there is provided a wide-angle image generating apparatus having a reflecting mirror having a convex reflecting surface and a principal point on the rotation center axis of the reflecting mirror, and forming a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation arithmetic processing unit that generates a development table including information on a pixel correspondence relationship between the wide angle image and the development image when the development image generation unit develops the wide angle image; A development table storage unit that holds the development table generated by the processing device and is connected to the development image generation unit, andExhibitionAn unfolding table for processing an open table generating arithmetic processing unit to arbitrarily change a pixel correspondence relationship between a wide-angle image and a unfolded image in an arbitrary mask area provided on the unfolded image and mask the unfolded image in the mask area It is characterized by having produced | generated.
  The wide-angle image generating apparatus according to claim 15 of the present invention has a reflecting mirror having a convex reflecting surface, a principal point on the rotation center axis of the reflecting mirror, and forms a reflected image of the reflecting mirror. An imaging device that images an object in a range of 360 degrees around the rotation center axis of the reflecting mirror, and a developed image generation unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device; A development table generation arithmetic processing unit that generates a development table including information on a pixel correspondence relationship between the wide angle image and the development image when the development image generation unit develops the wide angle image; A development table storage unit that holds the development table generated by the processing device and is connected to the development image generation unit, and a post-process of the development image generation unit in the subsequent stage of the development image generation unit, Characterized in that adding the image quality correction processing unit for performing high-frequency component enhancement processing for changing the degree of emphasis in accordance with the position of the opening image.
[0058]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to FIGS.
First, based on each embodiment of (Embodiment 1) to (Embodiment 4) shown in FIGS. 1 to 9, the wide-angle imaging device has a rotation center axis in a plane perpendicular to the reference optical axis direction. Even when tilted with respect to the vertical direction, the output as the wide-angle image generation device is an image in which the horizontal direction in the image coincides with the horizontal direction of the TV monitor, and the wide-angle image generation device that does not give a strange feeling to the viewer will be described .
[0059]
(Embodiment 1)
1 to 5 show (Embodiment 1) of the present invention.
FIG. 1 shows the wide-angle image generation apparatus of (Embodiment 1), in which the coordinate axes are the X axis in the vertical direction of the paper, the Y axis from the paper to the near side, and the Z axis in the horizontal direction of the paper. The X axis is the vertical direction.
[0060]
The wide-angle imaging device 8 includes a hyperboloid reflecting mirror 1, an imaging lens 2, and an imaging element 3. The imaging device includes an imaging lens 2 and an imaging element 3. The hyperboloid reflecting mirror 1 has a reflecting surface for ± 90 degrees around the rotation center axis of the hyperboloid. The direction of 0 degree, that is, the reference optical axis direction is the Y-axis direction. The reference optical axis passes through the internal focal point 4 of the hyperboloidal reflector 1. The rotation center axis 6 of the hyperboloidal reflector 1 coincides with the Z axis (horizontal direction) by tilting 90 degrees from the vertical direction with the Y axis as the rotation axis.
[0061]
Light rays a, b, and c incident on the internal focal point 4 are reflected by the reflecting surface of the hyperboloidal reflecting mirror 1 and are collected to an external focal point (not shown) on the rotation center axis 6. The position of the external focus and the position of the principal point 7 of the imaging lens 2 are arranged so as to coincide with each other or substantially coincide with each other, and the light rays a, b and c pass through the imaging lens 2 and the imaging surface of the imaging device 3. 5 is imaged.
[0062]
In FIG. 1, the imaging lens 2 is drawn as a single lens, but it may be composed of a plurality of lenses. Although the reflecting mirror 1 is limited to a hyperboloid shape here, it may be a rotationally symmetric body such as a hemisphere or a cone. However, in this case, there is no external focal point, so the principal point 7 of the imaging lens 2 is an arbitrary position on the rotation center axis 6.
[0063]
A gyroscopic Y-axis angle sensor 12 as a rotation center axis in-plane tilt direction detection unit provided in the wide-angle imaging device 8 has a rotation center axis 6 in a plane perpendicular to the reference optical axis direction (in the XZ plane). Is output with respect to the reference axis (here, the vertical axis).
[0064]
The arithmetic processing unit 11 includes a developed image generation unit 9 and an output image generation unit 10. In the developed image generation unit 9, the wide-angle image with a 180 ° field angle obtained from the image sensor 3 of the wide-angle image pickup device 8 depends on the shape of the hyperboloid reflector 1 and the optical characteristics of the imaging lens 2 and the image sensor 3. The wide-angle image is converted into a developed image according to the algorithm determined by the above.
[0065]
The output image generation unit 10 performs an inclination correction process, which will be described later, on the developed image obtained from the developed image generation unit 9 based on the output of the Y-axis angle sensor 12, and converts it into the NTSC system. Reference numeral 13 denotes a TV monitor that displays an output image from the arithmetic processing unit 11.
[0066]
The configuration of FIG. 1 will be described in more detail based on the operation.
In FIG. 1, light rays a, b, and c indicate arbitrary light rays from the object. Therefore, the object of each light beam is called an object a, b, c. The objects a, b, and c are characters “a”, “b”, and “c” drawn on the screen 14 as shown in FIG. 2A, and this screen 14 is shown in FIG. 2B. Consider a case where the surface on which the objects a, b, and c are drawn is visible from the hyperboloid reflector 1 so that the distance from the Z axis to the screen surface is constant around the Z axis.
[0067]
As already described, all rays incident on the reflecting surface of the hyperboloidal reflector 1 toward the internal focal point 4 are reflected to the external focal point (not shown), and the positions of the external focal point and the principal point coincide with each other. The image is formed on the image pickup surface 5 of the image pickup device 3 through the image forming lens 2 installed so as to coincide with each other.
[0068]
Accordingly, a range of ± 90 degrees with respect to the reference optical axis direction (Y-axis direction) forms an image on the imaging surface 5. At this time, an image obtained by the image sensor 3 is shown in FIG. It can be seen that the objects a, b, and c are all included.
[0069]
In the arithmetic processing unit 11, a wide-angle image as shown in FIG. 3 output from the wide-angle imaging device 8 is converted into a developed image by the developed image generation unit 9.
As described above, the development process determines the corresponding plane in the wide-angle image of each pixel of the developed image by determining the projection plane (virtual cylindrical surface 21) of the developed image and considering the mapping from the wide-angle image to the projection plane. Then, the developed image is generated by determining the luminance values of all the pixels of the developed image accordingly. For example, when the expansion process is performed on the wide-angle image as shown in FIG. 3, a developed image as shown in FIG. 4 is obtained.
[0070]
For the developed image created by the processing in the developed image generation unit 9 as described above, the output image generation unit 10 performs conversion processing to an output image.
The output image generation unit 10 performs two processes.
[0071]
First, the developed image inclination correction processing will be described.
When the rotation center axis 6 is tilted 90 degrees with respect to the vertical direction in a plane perpendicular to the reference optical axis direction, the developed image is 90 degrees with respect to the direction in which the vertical direction in the image should be as shown in FIG. The image is tilted.
[0072]
Therefore, the inclination is corrected by subjecting the horizontally developed image to image rotation processing. Specifically, when the gyroscopic Y-axis angle sensor 12 incorporated in the wide-angle imaging device 8 is used and the tilt angle with respect to the vertical direction of the rotation center axis 6 of the wide-angle imaging device 8 is viewed in the + direction of the Y-axis. The clockwise direction of is measured as positive.
[0073]
The developed image is rotated by the image processing calculation with the clockwise direction being positive by the inclination angle obtained in this way.
Next, the output imaging process will be described.
[0074]
The wide-angle image development process and the inclination correction process are performed in a digitalized state. Therefore, a process of converting to a format suitable for displaying a digitized image, for example, the NTSC system is performed. Specifically, first, a blank area is added so that the image size matches the NTSC size at the digital data stage. This blank area may be a zero brightness area, or some information (for example, a background image or a caption indicating the shooting time) may be inserted. Next, the NTSC-sized digital data is converted into NTSC by configuring a circuit using an existing NTSC encoding chip or the like. Even in the developed image as shown in FIG. 4, the output image is displayed on the TV monitor 13 in a state where the horizontal direction in the image matches or substantially matches the actual horizontal direction, as shown in FIG.
[0075]
The output format is not limited to NTSC, and other formats such as PAL may be used.
Here, the rotation center axis 6 is tilted by 90 degrees, but the tilt angle is not limited to 90 degrees. However, if the tilt angle is 90 degrees, a field of view of 180 degrees in the vertical direction, which is difficult to achieve with a normal wide-angle optical system, can be obtained. Therefore, there is an advantage that it can be applied to monitoring applications such as an ATM (automatic cash dispenser) and a vending machine.
[0076]
Further, the present invention is not limited to the case where the rotation center axis 6 is inclined at an arbitrary angle and the wide-angle imaging device 8 is fixed. For example, as schematically shown in FIG. 6, the wide-angle imaging device 8 is rotated at a constant speed with the reference optical axis as the rotation axis, and the output image is output to the TV monitor 13. At this time, the output image also travels through the TV monitor at a constant speed, but the content of the image is that the horizontal direction coincides with the horizontal direction of the TV monitor, and can be used as a special display used for events or the like.
[0077]
Further, in (Embodiment 1), the tilt correction process is controlled so that the horizontal direction in the developed image and the horizontal direction of the output image coincide with each other. However, the present invention is not limited to this. For example, control that makes the horizontal deviation between the horizontal direction in the developed image and the output image constant, or control that changes the horizontal deviation between the horizontal direction in the developed image and the horizontal direction of the output image according to a certain rule. Also good.
[0078]
In (Embodiment 1), the hyperboloidal reflector 1 has a reflection surface of ± 90 degrees, but is not limited to this. It may be in a range of at least ± 90 degrees or less.
[0079]
In (Embodiment 1), the gyroscopic Y-axis angle sensor 12 is attached to the wide-angle imaging device 8 to obtain tilt information. As another method, the inclination information may be obtained using image processing in the arithmetic processing unit 11.
[0080]
Specifically, photographing is performed, and photographing conditions are determined so that a target object for angle measurement is included in the obtained developed image. Then, the inclination angle of the object in the developed image is obtained by image processing, and the inclination angle at the time of output imaging processing is determined so that the angle is constant. This method has the advantage that no angle sensor is required.
[0081]
In (Embodiment 1), the developed image generation unit 9 performs a process of generating a developed image from a wide-angle image, and the output image generation unit 10 performs a developed image inclination correction process and an output imaging process to NTSC or the like. However, the developed image generation unit 9 may correct the developed image at the same time as the developed image is generated, and the output image generating unit 10 may perform only output imaging processing.
[0082]
Specifically, the corresponding point in the wide-angle image of each pixel of the developed image is determined by the method described as the prior art, and the luminance value of each pixel of the developed image is determined accordingly. The brightness value of each pixel obtained in this way is sequentially stored in a position in consideration of the tilt of the developed image on a memory (not shown) in the developed image generation unit 9, thereby generating a tilt corrected developed image. The tilt amount of the developed image at this time is acquired from the Y-axis angle sensor 12. The output image generation unit 10 receives the uncorrected developed image and performs output imaging processing.
[0083]
Such a configuration has an advantage that the processing speed can be expected to be improved since the generation of the developed image and the inclination correction can be performed at the same time.
As described above (Embodiment 1), even when the wide-angle imaging device 8 is in a state in which the rotation center axis 6 is inclined with respect to the vertical direction in a plane perpendicular to the reference optical axis direction, the wide-angle image generation device. Output is an image in which the horizontal direction in the image coincides with the horizontal direction of the TV monitor 13, so that the viewer does not feel uncomfortable.
[0084]
(Embodiment 2)
FIG. 7 shows a wide-angle image generation apparatus according to (Embodiment 2) of the present invention.
The difference from (Embodiment 1) shown in FIG. 1 is that the Y-axis rotation angle sensor 12 as the rotation center axis in-plane tilt direction detecting unit is eliminated, and the rotation center axis in-plane tilt direction is manually input and set. The rotation center axis in-plane inclination direction input unit 15 is provided in the arithmetic processing unit 11.
[0085]
The wide-angle image generating apparatus configured as described above will be described with a focus on differences from the first embodiment.
The rotation center axis in-plane inclination direction input unit 15 is an input setting device such as a variable resistor. Through this input setter, the value of the tilt angle with respect to the vertical direction in the plane perpendicular to the reference optical axis direction of the rotation center axis 6 of the wide-angle imaging device 8 secured as constant data in the output image generation unit 10 is obtained. Set. Other operations are the same as those in the first embodiment.
[0086]
As described above, according to (Embodiment 2), it is possible to configure by omitting the angle sensor 12 around the Y axis, so that the entire apparatus can be configured at a lower cost. Although the rotation center axis in-plane tilt direction input unit 15 is an input setting device such as a variable resistor here, the present invention is not limited to this configuration. For example, the rewriting of the constant data secured in the output image generation unit 10 is performed using a personal computer from the outside of the output image generation unit 10, and the rotation center axis in-plane inclination direction input unit 15 It is good also as a connection part of the production | generation part 10 and the said personal computer.
[0087]
In the above embodiment, the rotation center axis in-plane inclination direction input unit 15 is provided in the arithmetic processing unit 11. However, the variable resistor for inputting the rotation center axis in-plane inclination direction to the output image generation unit 10. The connection unit between the input setting device such as the output image generation unit 10 and the personal computer may be provided on the wide-angle imaging device 8 side.
[0088]
In (Embodiment 2), the hyperboloidal reflector 1 has a reflection surface of ± 90 degrees, but is not limited to this. It may be in a range of at least ± 90 degrees or less.
[0089]
(Embodiment 3)
FIG. 8 shows a wide-angle image generation apparatus according to (Embodiment 3).
In FIG. 8, 1 'indicates the position of the hyperbolic reflector in the reference state. 6 ′ indicates the position of the rotation center axis in the reference state. The rotation center axis 6 'in the reference state is assumed to be coincident with or substantially coincide with the horizontal direction, which is the state before tilting. Hyperboloid reflector 1, imaging lens 2, imaging device 3, internal focal point 4 of hyperboloid reflector 1, imaging surface 5 of imaging device 3, rotation center axis 6 of hyperboloid reflector 1, and main components of imaging lens 2 Point 7 is the same as in (Embodiment 1).
[0090]
The difference is that the rotation center axis 6 is installed in a plane including the reference optical axis and the rotation center axis 6 with an inclination angle θ around the internal focal point 4 with respect to the rotation center axis 6 ′ in the reference state. And the measurement direction of the X-axis angle sensor 16 is the tilt angle θ.
[0091]
Reference numerals 18 'and 19' denote the field-of-view limit of the wide-angle imaging device 8 in the reference state, 20 'denotes the field-of-view range of the wide-angle imaging device 8 in the reference state, and 18 and 19 denote the rotation center axis 6 by the tilt angle θ from the reference state. The field limit of the wide-angle imaging device 8 at the time of tilting (when tilted) indicates a field-of-view range of the wide-angle imaging device 8 when the rotation center axis 6 is tilted by the tilt angle θ from the reference state (when tilted). Reference numeral 21 denotes a virtual cylindrical surface serving as a virtual projection surface when a mapping from a wide-angle image to a developed image is considered.
[0092]
Next, the developed image generation unit 9 converts the wide-angle image obtained by the wide-angle imaging device 8 into a developed image according to an algorithm described later using the output of the X-axis rotation angle sensor 16 with respect to the wide-angle image having a 180-degree field angle. Process. The output image generation unit 10 converts the developed image obtained from the developed image generation unit 9 into the NTSC system.
[0093]
Reference numeral 13 denotes a TV monitor that displays an output image from the arithmetic processing unit 11 as in the first embodiment.
Here, the rotation center axis 6 ′ and the rotation center axis 6 in the reference state coincide with the horizontal direction, but the present invention is not limited to this.
[0094]
Next, with reference to FIG. 8, the operation will be described focusing on the differences from the first embodiment.
The operation of obtaining a wide-angle image with a 180-degree field of view of the wide-angle imaging device 8 is the same as in (Embodiment 1). The difference is that the visual field changes from the visual field range 20 ′ in the reference state shown in FIG. 8 by the tilt angle θ to the visual field range 20 when tilted.
[0095]
In the arithmetic processing unit 11, the developed image generation unit 9 performs conversion processing into a developed image on the wide angle image output from the wide angle imaging device 8. In this process, the concept of the correspondence between an arbitrary point P in the space and a point on the imaging surface 5 (that is, a point on the wide-angle image) p is the same as in (Embodiment 1). The difference is the projection plane when considering the developed image, that is, how to take the point P in the space. Specifically, a virtual cylindrical surface 21 as shown in FIG. 8 is considered. Usually, the virtual cylindrical surface 21 is assumed to have a center axis coinciding with the rotation center axis 6, but in (Embodiment 3), the rotation of the reference state obtained from the angle sensor 16 around the X axis is assumed. When the inclination angle of the rotation center axis 6 after tilting with respect to the center axis 6 ′ is θ, the center axis of the virtual cylindrical surface 21 with respect to the rotation center axis 6 after tilt includes the reference optical axis and the rotation center axis 6. In the plane, it is tilted by −θ around the internal focal point 4. That is, the center axis of the virtual cylindrical surface 21 is made to coincide with the rotation center axis 6 'in the reference state. The brightness of the point on the developed image corresponding to the arbitrary spatial point P on the virtual cylindrical surface 21 determined in this way is obtained using the brightness value of the point p on the wide-angle image corresponding to the spatial point P, and a developed image is created. To do.
[0096]
Here, the center of rotation when the wide-angle imaging device 8 is tilted from the reference state is the internal focus 4, but the present invention is not limited to this. However, if the rotation center coincides with or substantially coincides with the internal focal point 4, there is an advantage that the tilt angle can be adjusted while the viewpoint of the wide-angle imaging device 8 remains substantially constant.
[0097]
For the developed image created by the processing in the developed image generation unit 9 as described above, the output image generation unit 10 performs conversion processing to an output image. In (Embodiment 3), there is no “tilt correction processing” included in (Embodiment 1), but only output imaging processing. In the output imaging process, the same process as that in the first embodiment is performed.
[0098]
As described above, according to (Embodiment 3), the wide-angle imaging device 8 is tilted in a plane including the reference optical axis and the rotation center axis 6 so that the visual field range in the direction of the rotation center axis 6 as a wide-angle image generation device. Can be changed.
[0099]
Further, the inclination angle of the wide-angle imaging device 8 with respect to the reference state is measured, and the value is used to always match the central axis of the virtual cylindrical surface 21 that is the projection surface of the developed image with the rotation center axis 6 ′ of the reference state. Thus, the position of the center of the line of sight in the developed image (the position where the elevation angle in the developed image is 0. Light rays traveling toward the internal focal point 4 equivalent to the principal point 7 of the imaging lens 2 are applied to the virtual cylindrical surface 21 that is the projection surface. It is possible to change the visual field range without changing the position that intersects perpendicularly. In particular, the absolute value of the angle from the reference optical axis to the visual field limit 18 ′ in the reference state is φ, the absolute value of the angle from the reference optical axis to the visual field limit 19 ′ in the reference state is ψ, and θ is clockwise in FIG. Is positive, even if the hyperboloidal reflector 1 is made so that φ <ψ, if θ is (φ−ψ) / 2, the center of the line of sight will be A developed image as in the center in the short direction can be obtained.
[0100]
For this reason, the distance from the line-of-sight center in the developed image can be minimized, and the part far from the line-of-sight center in the developed image is seen obliquely than the other parts, so compared to other parts The problem of appearing distorted can be reduced.
[0101]
Note that the present invention is not limited to the wide-angle image generation device in which the wide-angle imaging device 8 is tilted and fixed at an arbitrary angle.
Specifically, wide-angle imaging is achieved by dynamically controlling the inclination of the central axis of the virtual cylindrical surface 21 according to the output of the X-axis rotation angle sensor 16 and maintaining the coincidence with the rotation center axis 6 'in the reference state. It is also possible to deal with a dynamic inclination in a plane including the reference optical axis (Y axis) of the apparatus 8 and the rotation center axis 6.
[0102]
In (Embodiment 3), the hyperboloidal reflector 1 has a reflection surface of ± 90 degrees, but is not limited to this. It may be in a range of at least ± 90 degrees or less.
[0103]
In (Embodiment 3), the case where the virtual cylindrical surface 21 is tilted by −θ so as to cancel the tilt angle θ with respect to the reference state of the wide-angle imaging device 8 has been described, but the present invention is not limited to this.
[0104]
For example, the control may be such that the inclination of the central axis of the virtual cylindrical surface 21 with respect to the rotation center axis 6 ′ in the reference state is set to a constant value other than 0, or is changed according to a fixed rule.
[0105]
(Embodiment 4)
FIG. 9 shows a wide-angle image generation apparatus according to (Embodiment 4) of the present invention.
The difference from (Embodiment 3) is that the X-axis rotation angle sensor 16 is eliminated and a rotation center axis forward tilt direction input unit 17 is provided in the arithmetic processing unit 11.
[0106]
This wide-angle image generation device will be described based on its operation, focusing on the differences from (Embodiment 3).
The rotation center axis forward tilt direction input unit 17 is an input setting device such as a variable resistor. The rotation center axis 6 ′ in the reference state in the plane including the reference optical axis (Y axis) and the rotation center axis 6 of the wide-angle imaging device 8 secured as constant data in the developed image generation unit 9 by this input setting device. The value of the inclination angle of the rotation center axis 6 with respect to is set / changed. Other operations are the same as those in the third embodiment.
[0107]
According to this (Embodiment 4), it is possible to configure by omitting the angle sensor 16 around the X axis, so that the entire apparatus can be configured at a lower cost. Here, the rotation center axis forward tilt direction input unit 17 is provided in the arithmetic processing unit 11, but the wide angle imaging device 8 may be configured by providing the rotation center axis forward tilt direction input unit 17.
[0108]
Here, although the rotation center axis forward tilt direction input unit 17 is an input setting device such as a variable resistor, it is not limited to this form. For example, the rewriting of the constant data secured in the developed image generation unit 9 is performed using a personal computer from the outside of the developed image generation unit 9, and the rotation center axis forward tilt direction input unit 17 generates the developed image. It is good also as a connection part of the part 9 and a personal computer. This connecting portion is provided in the wide-angle imaging device 8 or the arithmetic processing device 11.
[0109]
In (Embodiment 4), the hyperboloidal reflector 1 has a reflection surface of ± 90 degrees, but is not limited to this. It may be in a range of at least ± 90 degrees or less.
[0110]
In the fourth embodiment, the case where the virtual cylindrical surface 21 is tilted by −θ so as to cancel the tilt angle θ with respect to the reference state of the wide-angle imaging device 8 has been described, but the present invention is not limited to this. For example, when the tilt angle θ with respect to the reference state of the wide-angle imaging device 8 is 0, only the virtual cylindrical surface 21 may be tilted. Then, it becomes possible to change the line-of-sight center while the wide-angle imaging device 8 is fixed.
[0111]
Next, based on the respective embodiments of (Embodiment 5) to (Embodiment 7) shown in FIG. 10 to FIG. 32, among the reflected images formed by the reflecting surface, it is perpendicular to the rotation center axis of the reflecting surface. When considering a wide-angle image generation device that captures only about ± 90 degrees with any arbitrary direction as a reference direction, the same processing as in the case of an omnidirectional camera is applied to a wide-angle image to generate a developed image. When trying to obtain, there is a problem that the developed image obtained due to the deformation of the image due to the inclination of the optical axis of the imaging device or the optical aberration is not correct, but a wide-angle image generation device that can solve this problem will be described. .
[0112]
(Embodiment 5)
10 to 23 and 30 show (Embodiment 5) of the present invention.
FIG. 10 shows a wide-angle image generation apparatus according to (Embodiment 5) of the present invention.
[0113]
The coordinate axes are a Z axis in the vertical direction of the paper, a Y axis in the forward direction from the paper, and an X axis in the horizontal direction of the paper. The Z axis is the vertical direction.
The wide-angle image generation device includes a wide-angle imaging device 8 that captures a reflection image created by a reflecting mirror and an image processing device 22.
[0114]
First, the wide-angle imaging device 8 will be described.
The wide-angle imaging device 8 includes a hyperboloidal reflecting mirror 1, an imaging lens 2, and an imaging element 3 such as a CCD. The image pickup apparatus includes an imaging lens 2 and an image pickup element 3.
[0115]
The hyperboloidal reflecting mirror 1 has a reflecting surface for approximately ± 90 degrees around the rotation center axis of the hyperboloid. The 0 degree direction is referred to as a reference optical axis direction, and here it is assumed to coincide with the Y axis direction.
[0116]
Reference numeral 4 denotes an internal focal point of the hyperboloidal reflecting mirror 1. Reference numeral 5 denotes an imaging surface of the image sensor 3. Reference numeral 6 denotes a rotation center axis of the hyperboloid reflecting mirror 1. Here, it is assumed that the rotation center axis 6 coincides with the Z axis (vertical direction). Light rays a, b, and c incident on the internal focal point 4 are reflected by the reflecting surface of the hyperboloidal reflecting mirror 1 and are collected to an external focal point (not shown) on the rotation center axis 6. The position of the external focal point and the position of the principal point 7 of the imaging lens 2 are arranged so as to substantially coincide with each other, and the light rays a, b, c are passed through the imaging lens 2 on the imaging surface 5 of the imaging device 3. To form an image. Here, the optical axis (not shown) shared by the imaging lens 2 and the image pickup device 3 is installed to be inclined in the direction of the reference optical axis with respect to the rotation center axis 6 of the hyperboloidal reflecting mirror 1. This inclined configuration will be described later with reference to FIG.
[0117]
In FIG. 10, the imaging lens 2 is drawn as a single lens, but it may be composed of a plurality of lenses.
Next, the image processing device 22 will be described.
[0118]
The image processing apparatus 22 includes a development table generation unit 23, a development image generation unit 24, an image quality correction processing unit 25, and an image mask processing unit 26.
The development table generation unit 23 calculates a wide-angle image obtained from the wide-angle imaging device 8 into a developed image, which is calculated by an algorithm determined according to the shape of the hyperboloidal reflecting mirror 1 and the optical characteristics and arrangement of the imaging lens 2 and the image sensor 3. And processing for calculating a development table used for conversion. This algorithm will be described later.
[0119]
The developed image generation unit 24 refers to the development table obtained from the development table generation unit 23 and performs processing for converting a wide-angle image into a developed image.
The image quality correction processing unit 25 performs image quality correction processing described later on the developed image generated by the developed image generation unit 24.
[0120]
The image mask processing unit 26 performs image mask processing described later on the developed image after the image quality correction generated by the image quality correction processing unit 25. A monitor 27 displays an output image from the image processing device 22.
[0121]
Next, the configuration of the wide-angle imaging device 8 will be described in detail with reference to FIG.
FIG. 11A is a cross-sectional view in the YZ plane of the wide-angle imaging device 8 and shows a basic configuration. The difference from the omnidirectional camera 100 is that the hyperboloidal reflecting mirror 1 has a reflecting surface for ± 90 degrees around the rotation center axis of the hyperboloid, the principal point 7 of the imaging lens 2 and the hyperboloid. The optical axis 28 shared by the imaging lens 2 and the imaging device 3 is in the reference optical axis direction (Y-axis direction) by an angle γ with respect to the rotation center axis 6 while making the external focus of the reflecting mirror 1 substantially coincide. The point is that it is installed at an angle. 2 indicated by a broken line₀, 3₀These show an imaging lens and an image sensor when arranged in the same manner as the omnidirectional camera 100 shown in FIG.
[0122]
Rays 29a and 29b in the YZ plane that are incident toward the internal focal point 4 are reflected by the incident points 30a and 30b of the hyperboloidal reflecting mirror 1 and condensed to an external focal point (not shown) on the rotation center axis 6. Is done. The position of the external focal point and the position of the principal point 7 of the imaging lens 2 are arranged so as to substantially coincide with each other, and the light rays 29a and 29b are connected to the imaging surface 5 of the imaging device 3 via the imaging lens 2. An image is formed on the image points 31a and 31b.
[0123]
The imaging lens 2₀And image sensor 3₀The image forming point of the rays 29a and 29b is 31a.₀, 31b₀It becomes. Therefore, the obtained image is the imaging lens 2.₀And image sensor 3₀32 is indicated by a broken line in FIG.₀It becomes a semicircular image like this. This is exactly a half of the circular image 111 of FIG. 34 (b) obtained by the omnidirectional camera 100. FIG. In this way, the imaging lens 2 as used in the omnidirectional camera 100 is used.₀And image sensor 3₀It is possible to obtain a wide-angle image having a field angle of 180 degrees by using. However, in this state, the image sensor 3₀The efficiency of use is very bad.
[0124]
On the other hand, when the imaging lens 2 and the image sensor 3 indicated by the solid line in FIG. 11A are used, the obtained image is a semicircular image 32 indicated by the oblique line in FIG. . That is, by tilting the optical axis 28, the position in the vertical direction (corresponding to the Y-axis direction) of the semicircular image 32 on the imaging surface 5 can be shifted. As a result, the left and right direction of the semicircular image 32 on the imaging surface 5 is adjusted by adjusting the magnification and adjusting the tilt angle γ of the optical axis 28 as in the semicircular image 32. By maximizing the length, it is possible to perform imaging using the pixels on the imaging surface 5 as effectively as possible.
[0125]
In this (Embodiment 5), the semicircular image 32 is formed on the imaging surface 5 at a position where the arc portion is close to the edge of the imaging surface 5. However, the present invention is not limited to this. is not. For example, the chord portion of the semicircular image 32 may be formed at a position close to the edge of the imaging surface 5.
[0126]
Further, in this (Embodiment 5), the hyperboloid reflector 1 has a reflection surface of ± 90 degrees, but is not limited to this. It may be within a range of ± 180 degrees or less. For example, when the hyperboloidal reflector 1 having a reflection surface of ± (90 + θ) degrees is used, an image of ± (90 + θ) degrees can be obtained as shown in FIG.
[0127]
The operation of the wide-angle image generating apparatus configured as described above will be described in more detail.
First, an image forming mechanism in the wide-angle imaging device 8 will be described.
In FIG. 10, light rays a, b, and c indicate arbitrary light rays from the object. Therefore, the object of each light beam is called an object a, b, c. The objects a, b, and c are characters “a”, “b”, and “c” drawn on the screen 33 as shown in FIG. 13A, and the screen 33 is shown in FIG. 13B. Consider a case where the surface on which the objects a, b, and c are drawn is visible from the hyperboloid reflector 1 so that the distance from the Z axis to the screen surface is constant around the Z axis.
[0128]
As already described, all the light rays incident on the reflecting surface of the hyperboloid reflector 1 toward the internal focal point 4 are reflected to the external focal point (not shown), and the positions of the external focal point and the principal point substantially coincide. The image is formed on the imaging surface 5 of the imaging device 3 through the imaging lens 2 installed in this manner. Accordingly, a range of about ± 90 degrees with respect to the reference optical axis direction (Y-axis direction) is imaged on the imaging surface 5. At this time, a wide-angle image obtained by the image sensor 3 is shown in FIG. It can be seen that the objects a, b, and c are all included.
[0129]
Next, the image processing device 22 performs a developing process on the wide-angle image output from the wide-angle imaging device 8 as shown in FIG. First, in the initial stage before starting the expansion process, the expansion table generation unit 23 calculates the expansion table used for the expansion process. Here, the development table is a table of coordinate values on the wide-angle image corresponding to each pixel of the development image. FIG. 15 shows a flow of processing performed by the development table generation unit 23.
[0130]
In step S1 of FIG. 15, information such as the shape parameter of the reflecting surface of the hyperboloidal reflecting mirror 1 and the focal length of the imaging device necessary for calculating the development table is read. For example, such information may be written in a ROM in the development table generation unit 23 and read. Of course, it may be configured to input from the outside by some method.
[0131]
In step S2, each element of the development table is calculated. The expansion table is generated by determining the projection plane of the expansion image and determining the corresponding points in the wide-angle image of each pixel of the expansion image by considering the mapping from the wide-angle image to the projection plane. Details of the corresponding point determination method will be described later.
[0132]
In step S3, the influence of the optical distortion of the image pickup apparatus including the imaging lens 2 and the image pickup device 3 is corrected with respect to the development table calculated in step S2. Details of this correction method will also be described later.
[0133]
The expansion table calculated by the flow as described above is stored in the RAM in the expansion table generation unit 23.
Here, the corresponding point determination method between the wide-angle image and the projection plane in step S2 will be described in detail with reference to FIGS.
[0134]
FIG. 16 schematically shows a cross section of a plane (YZ plane) including the rotation center axis 6 and the reference optical axis direction of the wide-angle imaging device 8. The hyperboloid reflecting mirror 1, the imaging lens 2, the imaging device 3, the internal focal point 4, the imaging surface 5, the rotation center axis 6, and the principal point 7 are the same as those described in FIG.
[0135]
Reference numeral 34 denotes a virtual projection plane when a mapping from a wide-angle image is considered. Here, the virtual projection plane 34 is assumed to be a cylindrical surface whose rotation center axis coincides with the rotation center axis 6 of the hyperboloidal reflector 1. Note that the virtual projection surface 34 is not limited to a cylindrical surface. For example, it may be a spherical surface.
[0136]
35 is an arbitrary point P (X_P, Y_P, Z_P) Toward the internal focal point 4.
In the spatial coordinate system, the Z axis coincides with the rotation center axis 6, and the origin O is a point that bisects a straight line connecting the internal focus 4 and the principal point 7. Also, the front direction from the page is the X axis, and the left and right direction of the page is the Y axis. The coordinate system on the wide-angle image (on the imaging surface 5) has an intersection point between the rotation center axis 6 and the imaging surface 5 as the origin o, the front direction from the page is the x-axis, and the horizontal direction of the page as shown in FIG. Y axis.
[0137]
An arbitrary point P (X on the virtual projection plane 34_P, Y_P, Z_P) On the imaging surface 5 (ie, a point on the wide-angle image) p (x_p, Y_p)think of. The hyperboloid shape of the hyperboloidal reflector 1 is expressed by (Equation 7) as in the case of the omnidirectional camera 100 described in the section of (Prior Art).
[0138]
(X²+ Y²/ A²-Z²/ B²  = −1 (Formula 7)
Here, a and b are constants that determine the shape of the hyperboloid. Also
c = (a²+ B²)^0.5                                (Formula 8)
It is. Further, (Equation 9) and (Equation 10) are also established as in the case of the omnidirectional camera 100.
[0139]
Z = (X²+ Y²)^0.5・ Tanα + c (Formula 9)

Here, the angle α is an angle formed by a straight line connecting the internal focal point 4 and the arbitrary point P and the XY plane. The angle β is an angle formed by a straight line connecting the principal point 7 (substantially coincident with the external focal point of the hyperboloid reflector 1) and the point p on the imaging surface 5 and the XY plane.
[0140]
X from (Equation 9)_P, Y_P, Z_PIs determined, α is determined. If α, b, and c are determined from (Equation 10), β is determined.
In the omnidirectional camera 100, if the value of F is determined by using (Equation 11), (x²+ Y²)^0.5That is, the distance from the origin on the wide-angle image to the point p was found. However, in (Embodiment 5), since the imaging surface 5 is inclined by γ,²+ Y²)^0.5The value of cannot be obtained. Therefore, the expansion process cannot be performed correctly as it is.
[0141]
Therefore, (Equation 13) is used in (Embodiment 5).
(X²+ Y²)^0.5   = F · cosβ / cosγ / sin (β + γ) (Formula 13)
The derivation method of (Formula 13) will be described with reference to FIG.
[0142]
FIG. 17 is an enlarged view of the vicinity of the imaging surface 5 of the wide-angle imaging device 8 of FIG. In FIG. 17, let q be the intersection of a light beam from an arbitrary point P with a straight line that is in the YZ plane, is parallel to the Y axis, and passes through the origin o on the imaging surface 5. At this time, ∠pqo = β and ∠poq = γ. Further, the length | oq | of the line segment oq is expressed by (Expression 14).
[0143]
| oq | = F · tan (90 ° −β) / cosγ = F / tanβ / cosγ (Formula 14)
Therefore, according to the sine theorem in Δopq, | op |²+ Y²)^0.5Is obtained by (Equation 13).

Here, F is a focal length of an image pickup apparatus including the imaging lens 2 and the image pickup device 3.
[0144]
(Formula 9) (Formula 10) (Formula 13) X_P, Y_P, Z_PAnd b, c, and F are determined (x_p ²+ Y_p ²)^0.5That is, the distance from the origin o on the imaging surface 5 to the point p is determined. The above discussion is valid for planes other than the YZ plane including the Z axis, but it should be noted that the inclination γ of the imaging surface 5 changes depending on which plane other than the YZ plane including the Z axis is taken. Don't be. This point will be described with reference to FIG.
[0145]
FIG. 18 shows the hyperboloid reflector 1 and the imaging surface 5 when viewed from the Z-axis direction.
Reference numeral 35 denotes a light ray emitted from an arbitrary point P (not shown in FIG. 18) on the virtual projection plane 34 and reflected by the hyperboloid reflector 1 to reach the point p on the imaging plane 5. As shown in FIG. 18, the angle θ is defined as + in the left direction, with the − direction of the X axis being 0 degrees. For example, when the light beam 35 is in the YZ plane, θ = 90 degrees. The relationship between θ and the inclination γ (θ) of the imaging surface 5 at this time is as shown in (Equation 15).
[0146]
tanγ (θ) = tanγ (90 °) · sinθ (Formula 15)
Here, γ (90 °) means γ when θ = 90 °. Further, (Equation 13) becomes (Equation 13a).
[0147]

With the above discussion, the distance from the origin o to the point p in each plane including the Z axis could be obtained. Next, consider the relationship between the direction θ from the origin O to the point P on the XY plane and the direction φ from the origin o to the point p on the xy plane, that is, the imaging surface 5. Here, φ is an angle parameter for defining the direction of a point on the imaging surface 5 where the + direction of the X axis is 0 degree as shown in FIG.
[0148]
In the case of the omnidirectional camera 100, θ = φ, and the directions of the points P and p on the XY plane and the xy plane are the same. However, in the case of the wide-angle imaging device 8, since the imaging surface 5 is inclined by γ (90 °) with respect to the XY plane, the relationship between θ and φ is as shown in (Expression 16).
[0149]
cosφ = cosθ / (1 + tan²γ (90 °) ・ sin²θ)^0.5      (Formula 16)
The derivation method of (Formula 16) is demonstrated using FIG.
The Xa and Ya planes in FIG. 19 are obtained by rotating the XY plane by 180 degrees about the Z axis and translating the Xa axis so as to coincide with the x axis. Here, θ is a parameter equivalent to θ on the XY plane, with the − direction of the Xa axis being 0 degrees and the clockwise direction on the Xa and Ya planes being the + direction. For this reason, the parameter name is not changed here and is used as it is. Further, φ is 0 degree in the − direction of the x axis, and the clockwise direction on the xy plane is the + direction. With this setting, the relationship between θ and φ is derived. The arc CAD is half of the unit circle. The CBD is a line of intersection between the Xa, Ya plane and a semicircle CAD, which is perpendicular to the Xa, Ya plane, and the xy plane. At this time, the vectors oA and oB are as shown in (Expression 17) and (Expression 18).
[0150]
oA = (− cos θ, sin θ, 0) (Formula 17)
oB = (− cos θ, sin θ, tan γ (90 °) · sin θ) (Formula 18)
Therefore, (Expression 16) is obtained by using the inner product with the unit vector -X in the -x direction.
[0151]

From the above, if β, γ (= γ (90 °)), and F are known, P (X_P, Y_P, Z_P) Corresponding to a point on the imaging surface 5 (that is, a point on the wide-angle image) p (x_p, Y_p) Can be determined uniquely. X_P, Y_P, Z_PThis value can be expressed as follows, where M is the radius of the virtual projection plane 34, ISX is the number of pixels in the horizontal direction (longitudinal direction) of the developed image, and ISY is the number of pixels in the vertical direction (short direction).
[0152]
X_P = M ・ cosθ
Y_P = M · sinθ
Z_P = 2π · M · (j−j0) / ISX (Formula 19)
However, θ = 180 ° · i / ISX, i is an X coordinate value on the developed image and an integer 0 ≦ i <ISX, j is a Y coordinate value on the developed image, 0 ≦ j <ISY integer, and j₀Is Z_PThis is the Y coordinate value on the developed image when = c (when the viewing direction is the horizontal direction). By using (Equation 19), the point on the developed image and the point P (X_P, Y_P, Z_P).
[0153]
Accordingly, by determining the projection plane (virtual projection plane 34) of the developed image and considering the mapping from the projection plane to the wide-angle image, ideal corresponding points on the wide-angle image for each pixel of the developed image can be determined. .
[0154]
Next, a method for correcting the influence of the optical distortion of the imaging apparatus including the imaging lens 2 and the imaging element 3 in step S3 in FIG. 15 will be described in detail with reference to FIG.
[0155]
FIG. 20 schematically shows the influence of optical distortion. If an ideal image without optical distortion is obtained, a semicircular image as shown in FIG. 20A is obtained when the angle of view is 180 degrees. However, optical distortion is actually included in the optical system, and a semicircular image is distorted and obtained, for example, as shown in FIG. For this reason, the developed image obtained by performing the developing process without taking this optical distortion into consideration is not correct.
[0156]
Therefore, in (Embodiment 5), the distortion amount of the optical system used in advance is grasped, and the development table created in step S2 is corrected so as to cancel the influence.
First, a method for grasping the amount of distortion will be briefly described.
[0157]
A high-order polynomial is often used as a general method for describing optical distortion with mathematical expressions. Here, a case where a polynomial as shown in (Equation 20) is used is considered.
Δx = kx (x²+ Y²)
Δy = ky (x²+ Y²(Formula 20)
Here, Δx and Δy are distortion amounts in the x and y directions, respectively. k is a parameter for adjusting the amount of distortion. (Expression 20) is a very simple expression showing the amount of distortion, and the only parameter to be set is k. When applied to an actual optical system, (Equation 20) may not be sufficient to describe the amount of optical distortion. In that case, what considers a high order term should just be used suitably.
[0158]
A test pattern 36 serving as a reference (that is, a pattern having a known distance is included in order to measure the amount of distortion) is imaged by an imaging device including the imaging lens 2 and the imaging device 3. As the test pattern 36, for example, a pattern in which a two-dimensional lattice as shown in FIG. Here, for example, an accurate value of the grating pitch may be known. The test pattern image 37 obtained as a result is a pattern different from the original test pattern 36 due to the influence of optical distortion, as shown in FIG. By comparing the coordinate value of the ideal grid point that can be calculated from the grid pitch and the imaging conditions with the coordinate value of the corresponding grid point obtained from the test pattern image 37 (n), the distortion amount can be calculated. Take a sample. Of the n samples, the ideal coordinate values of the i-th data are x (i) and y (i), the measured strain amount in the x direction at that time is Δx (i), and the measured strain amount in the y direction is Δy. When (i) is set, k is calculated by the method of least squares so that the two error functions of (Expression 21) are minimized.
[0159]
Ex = Σ (Δx (i) −kx (i) (x (i)²+ Y (i)²))
Ey = Σ (Δy (i) −ky (i) (x (i)²+ Y (i)²)) (Formula 21)
With the above processing, (Equation 20) indicating the distortion amount of the imaging apparatus is determined. Next, using this, the ideal development table obtained in step S2 is corrected. Each pixel (each element) in the x direction and y direction of the development table includes pixel values of a wide-angle image corresponding to each pixel when optical distortion is not considered. This (x₀(X, Y), y₀(X, Y)). Here, (X, Y) is the coordinate value of the pixel (element) of the development table which is the current target. Substituting this value into (Equation 20), the corrected value (x₁(X, Y), y₁(X, Y)) is calculated. Specifically, (Equation 22) is performed for each pixel (element).
[0160]

By performing this process on all the pixels (elements) of the expansion table, it is possible to obtain a expansion table in which the influence of the optical distortion of the imaging device is corrected. Therefore, if a development process is performed using such a development table, a developed image in which the influence of optical distortion of the imaging device is eliminated can be obtained.
[0161]
In addition, such an optical distortion correction process can be performed on a developed image after the developed image is once created, but in that case, every time the developed image is created, a complicated process as shown in (Equation 22) is performed. Arithmetic processing must be performed. However, in (Embodiment 5), by incorporating the optical distortion correction process into the expansion table, a complicated calculation process like (Equation 22) is performed only when the expansion table generation unit 23 creates the initial expansion table. It is only necessary to perform a normal development process when creating a developed image, which is very advantageous from the viewpoint of calculation cost.
[0162]
Next, the developed image generation unit 24 will be described.
The developed image generation unit 24 refers to the development table generated by the development table generation unit 23 and converts the wide-angle image obtained from the wide-angle imaging device 8 into a developed image. At this time, the pixel value of the wide-angle image corresponding to each pixel of the developed image whose coordinate value is an integer value generally does not have an integer value. However, wide-angle images have only integer coordinate position data. As a countermeasure, when priority is given to processing speed over image quality, the coordinate value data of the expansion table is converted to an integer by means of rounding off, etc., so that the luminance value of the pixel in the vicinity of the originally necessary pixel is substituted. Also good. When priority is given to image quality over processing speed, luminance data of originally necessary pixels may be generated by interpolation calculation from luminance data of four pixels near the originally necessary pixels.
[0163]
Next, the image quality correction processing unit 25 will be described.
The image quality correction processing unit 25 performs image quality correction processing on the developed image created by the developed image generation unit 24. The image quality correction process will be described below.
[0164]
The developed image generated by the developed image generation unit 24 has a problem that the resolution is different between the upper part and the lower part in the image. That is, as shown in FIG. 22, in the correspondence between the wide-angle image 38 and the developed image 39 obtained by performing the development process on the wide-angle image 38, the relationship between the upper and lower ends of the developed image 39 is the relationship between the outer circumference and the inner circumference of the wide-angle image 38. Therefore, the difference in resolution is caused by the difference in information amount. Therefore, in the developed image 39, the portion corresponding to the inner peripheral portion looks more out of focus than the portion corresponding to the outer peripheral portion. As another factor, the imaging device including the imaging lens 2 and the imaging device 3 has a depth of field, and although it is small, the semicircular wide-angle image 38 has a non-uniform focus (for example, wide-angle) in the radial direction. The image 38 is well focused on the inner peripheral portion, but there may be unevenness in which the outer peripheral portion is slightly focused. The image finally obtained as the developed image 39 includes a blur as a result of superimposing the above two types of factors.
[0165]
For such a developed image 39, the image quality correction processing unit 25 performs relative sharpening for each location along the vertical direction of the developed image 39 (that is, corresponding to the radial direction of the wide-angle image 38 which is the original image). A coefficient is determined, and a sharpening process is performed by emphasizing high-frequency components of the image using the coefficient.
[0166]
As the sharpening process, for example, a technique called unsharp masking is used. This is expressed as (Equation 23).
B ′ (i, j) = B (i, j) + C (B (i, j) −B￣ (i, j)) (Formula 23)
Here, B (i, j) is the luminance value of the pixel at coordinates (i, j). B￣ (i, j) is a local average luminance value in the vicinity of the pixel at the coordinate (i, j) (for example, an average luminance value in the range of the neighborhood 3 × 3), and is a luminance value composed of a low frequency component. Therefore, (B (i, j) −B￣ (i, j)) is an operation for extracting the high frequency component by subtracting the low frequency component from the original image. C is a sharpening coefficient. By adding the high frequency component multiplied by C to the original image, a sharpened image in which the high frequency component is emphasized can be obtained.
[0167]
A sharpening process is performed by changing the sharpening coefficient C according to the vertical position of the developed image 39. First, a case where an image on the hyperboloidal reflecting mirror 1 can be captured ideally without blur by the imaging apparatus will be described.
[0168]
In the developed image 39 obtained in this case, as shown in FIG. 23A, the lower end portion is a blurred region 40 resulting from a difference in information amount between the inner peripheral portion and the outer peripheral portion of the original wide-angle image. Here, the blur region 40 is not uniformly blurred, but blurs from the center of the developed image 39 toward the lower end. At this time, as shown in the graph of FIG. 23A, the sharpening coefficient C is changed from 0 in accordance with the degree of blur, thereby performing a correction process (sharpening process) corresponding to the degree of blur.
[0169]
Although a graph in the case where the value of the sharpening coefficient C is increased from approximately the center in the vertical direction of the developed image 39 is shown here, the present invention is not limited to this. The gist of the present invention is to increase the value of the sharpening coefficient C according to the degree of blurring that gets worse as it approaches the lower end. For example, a place where the sharpening coefficient C is larger than 0 is the upper end of the developed image 39. It does not matter. The method of increasing the sharpening coefficient C can also be adjusted according to the degree of blur, the object to be photographed, and the like. For example, the sharpening coefficient C may be increased in proportion to the cube of the distance from a place where the sharpening coefficient C is greater than zero. .
[0170]
Next, a case where there is a difference in focus depending on the location when an image on the hyperboloid reflector 1 is picked up by the image pickup apparatus will be described with reference to FIG.
[0171]
In this case, the blurred area of the developed image 39 obtained is adjusted by adjusting the focus of the imaging apparatus. Since the sharpening process is a process that emphasizes the high-frequency component of the image, if the target image is blurred more than a certain amount, a significant effect cannot be exhibited even if the process is performed. Therefore, adjust the focus so as to improve as much as possible the part with the worst blur condition (the worst blur). Specifically, when the focus is adjusted so that the optically blurred portion is located at the inner peripheral portion of the wide-angle image 38, the condition for deteriorating the blurred state overlaps with the inner peripheral portion, and correction cannot be performed. In order to avoid this, adjustment is made so that the focus peak comes slightly inward from the center in the radial direction of the wide-angle image 38, and finally the blurred regions 40 at the upper and lower ends in the developed image 39 are as shown in FIG. Adjust the focus so that it is almost balanced. At this time, the blur area 40 is not uniformly blurred, but blurs from the center of the developed image 39 toward the upper and lower ends. At this time, as shown in the graph of FIG. 23 (b), the sharpening coefficient C is adjusted according to the degree of blur, and is changed from 0 toward the upper and lower ends from the vicinity of the center of the developed image 39. Correction processing (sharpening processing) is performed.
[0172]
Next, the image mask processing unit 26 will be described.
The image mask processing unit 26 that performs image mask processing on the developed image after the image quality correction generated by the image quality correction processing unit 25 is configured as follows.
[0173]
FIG. 30A shows a developed image when mask processing is not performed, and FIG. 30B shows a developed image when mask processing is performed. In the developed image, 41 is an automatic teller machine (ATM) terminal, 42 is an operator, and 47 is an operation panel of the ATM terminal 41. In FIG. 30A before the mask processing, the entire field of view is visible, and the operation of the operator 42, the input result, and the like are reflected, so there is a risk that private information such as a password will be leaked to the outside. In view of this, the image mask processing unit 26 performs a process of filling the mask area 48 stored in the ROM with, for example, a luminance value of 0, so that the operation of the operator 42, the input result, etc., as shown in FIG. Make it invisible.
[0174]
Note that although the luminance value is painted out to 0 here, the present invention is not limited to this. It is only necessary that the image in the mask area 48 can be filled with an image different from the original image, and a luminance value other than 0 or a random pattern image may be drawn.
[0175]
The developed image generated by performing the above processing is converted into a video signal such as NTSC by an image format conversion unit (not shown) and output from the image processing device 22. The monitor 27 receives the signal and displays a developed image.
[0176]
As described above (Embodiment 5), even if the optical system of the wide-angle imaging device 8 is configured such that the optical axis 28 of the imaging device including the imaging lens 2 and the imaging element 3 is inclined with respect to the rotation center axis 6, It is possible to determine the ideal corresponding point on the wide-angle image for each pixel of the developed image and correct the development table taking into account the influence of the optical distortion of the imaging device, so the correct development process Is possible. Further, by performing image quality correction processing, it is possible to reduce the difference in resolution depending on the location of the developed image.
[0177]
In addition, the image mask processing unit 26 performs mask processing on a part of the developed image and outputs the masked image so that it cannot be seen, thereby preventing the operator's operation content and input content from being shown and preventing leakage of privacy information. Is possible.
[0178]
In this (Embodiment 5), the reference optical axis is taken in an arbitrary direction perpendicular to the rotation center axis of the rotationally symmetric body, and the direction of the reference optical axis is set to 0 degree to approximately ± 90 degrees or ± 90 degrees. As a reflecting mirror having a convex reflecting surface in a range of at least degrees, it is a hyperboloid shape having an internal focal point on the rotation center axis, and the position of the external focal point of the hyperboloid shape and the position of the principal point of the imaging device are Although it was substantially coincident, the shape of the reflecting mirror is established even if it is not a hyperboloid.
[0179]
Further, in (Embodiment 5), the case where the wide-angle imaging device 8 captures a range of approximately ± 90 degrees around the rotation center axis is shown, but the wide-angle imaging device 8 is shown in FIG. Even when the omnidirectional camera 100 that captures a range of approximately 360 degrees around the rotation center axis is used and the image processing device 22 performs a development process corresponding to the omnidirectional camera 100, the optical described above Distortion influence correction, image quality correction processing, and image mask processing have the same effects as in the fifth embodiment.
[0180]
(Embodiment 6)
24 to 31 show (Embodiment 6) of the present invention.
FIG. 24 shows a wide-angle image generation apparatus according to (Embodiment 6) of the present invention.
[0181]
The coordinate axes are assumed to be the X axis in the vertical direction of the paper, the Y axis from the front to the paper, and the Z axis in the horizontal direction of the paper. The Z axis is the horizontal direction. The difference in configuration between (Embodiment 6) and (Embodiment 5) is that the wide-angle imaging device 8 is installed so that the Z axis coincides with the horizontal direction. The difference in processing is that the development table generation unit 23 of the image processing apparatus 22 has a feature in the shape of the virtual projection plane 34 used when generating the development table, and the specific processing of the image mask processing is different. The developed image generation unit 24 generates a developed image that is tilted by 90 degrees with respect to the case of (Embodiment 5) and corrects the top and bottom in the image. These differences will be described below.
[0182]
First, the mechanism of image formation will be described.
In FIG. 24, light rays a, b, and c indicate arbitrary light rays from the object. Therefore, the object of each light beam is called an object a, b, c. The objects a, b, and c are characters “a”, “b”, and “c” drawn on the screen 33 as shown in FIG. 25A, and this screen 33 is shown in FIG. 25B. Consider a case where the surface on which the objects a, b, and c are drawn is visible from the hyperboloid reflector 1 so that the distance from the Z axis to the screen surface is constant around the Z axis. A wide-angle image obtained by the image sensor 3 at this time is shown in FIG. It can be seen that the objects a, b, and c are all included.
[0183]
In this way, as an application example when the wide-angle imaging device 8 is installed so that the rotation center axis 6 coincides with the horizontal direction, there is monitoring of an ATM terminal or a vending machine. Since a field of view of about 180 degrees in the vertical direction, which is difficult to realize with a normal wide-angle optical system, can be obtained as a single image from the face of a person standing by the device to the hand, It can be applied to monitoring applications such as vending machines.
[0184]
Next, the difference between the processing in the image processing apparatus 22 and (Embodiment 5) will be mainly described.
First, as in the case of (Embodiment 5), in the initial stage before starting the expansion process, the expansion table generation unit 23 calculates the expansion table used for the expansion process. A flow of processing performed by the expansion table generation unit 23 will be described with reference to FIG. The step of determining an ideal corresponding point between the wide-angle image and the projection plane is one of the two points different from the case of the fifth embodiment in the process performed in the flow shown in FIG. In S2, the assumed shape of the virtual projection plane 34 is changed. This point will be described in detail with reference to FIG.
[0185]
FIG. 28 shows the relationship between the operator 42 who operates the ATM terminal 41, the wide-angle imaging device 8 and the virtual projection plane 34. 34 is a cylindrical virtual projection surface whose rotation center axis passes through the internal focal point 4 of the hyperboloidal reflector 1 of the wide-angle imaging device 8 and is perpendicular to the paper surface. Conventionally, projection is performed on such a projection surface. I was thinking of playing back the image. When using the ATM terminal 41, the operator 42 of the ATM terminal 41 usually takes an inclination angle 43 forward of about 15 to 30 degrees. The distance from the internal focal point 4 which is the viewpoint of the wide-angle imaging device 8 is farther in the face than in the chest, so that the vicinity of the chest is larger in the obtained developed image, and the face appears smaller in the face. Will cause a sense of incongruity.
[0186]
Therefore, in (Embodiment 6), as shown in FIG. 28, a new virtual projection plane 44 in which the central axis 45 is set in a direction opposite to the reference optical axis direction 46 which is the central direction of the obtained visual field is set. Considering the projection onto the new virtual projection plane 44 corresponds to moving the internal focal point 4 as the viewpoint of the wide-angle imaging device 8 virtually to the position of the central axis 45 in the direction opposite to the reference optical axis direction 46. To do. Therefore, the difference in the distance between the face position and the chest position with respect to the viewpoint of the wide-angle imaging device 8 when the operator 42 is using the ATM terminal 41 is alleviated.
[0187]
A specific method of changing the virtual projection plane shape will be described with reference to FIG.
FIG. 29 shows a conventional virtual projection plane 34 viewed from the Z-axis direction and a virtual projection plane 44 whose shape has been changed as in FIG. θ is the azimuth angle on the XY plane of the light ray from the point P on the virtual projection plane 44 to the internal focus 4 (not shown) of the wide-angle imaging device 8 is set with the X-axis direction set to 0 degree. α is an XY plane of a ray from a point P on the virtual projection plane 44 to a virtual internal focal point (not shown) on the central axis 45 of the virtual projection plane 44 (that is, the intersection of the reference optical axis passing through the internal focal point 4 and the central axis 45). The azimuth angle above is set with the X-axis direction set to 0 degree. In FIG. 29, the position of the internal focal point 4 is set to O.₁The position of the central axis 45 is O₂And
[0188]
Similar to (Embodiment 5), a point P (X on the virtual projection plane is obtained using (Expression 9), (Expression 10), (Expression 13a), (Expression 15), and (Expression 16)._P, Y_P, Z_P) Of the wide-angle image corresponding to)_p, Y_pHowever, the difference from (Embodiment 5) at this time is how to give θ in (Equation 16). In (Embodiment 5), θ = 180 ° · i / ISX (where i is an integer and 0 ≦ 0), particularly when the horizontal (longitudinal) number of pixels of the developed image is ISX and the horizontal viewing angle is 180 degrees. What is necessary is just to take (theta) at equal intervals like i <ISX). In (Embodiment 6), θ is determined so as to be equivalent to the case where α is taken at equal intervals. △ PO₁O₂(Equation 24) holds from the sine theorem.
[0189]
R / sin (90 ° + θ) = r / sin (90 ° −α) (Formula 24)
By arranging (Equation 24), (Equation 25) is obtained.
θ = cos^-1(R · cosα / r) (Formula 25)
Where α₁≦ α <α₂
Therefore, α = (α₂-Α₁) · I / ISX (where n is an integer, 0 ≦ i <ISX) is substituted for θ, and if a development table is created using the θ, the virtual projection plane shape can be changed.
[0190]
Here, a cylinder is considered as the virtual projection plane 34, but other shapes such as a spherical surface may be used.
Next, in the process performed in the flow shown in FIG. 27, the mask to be performed after step S3 for correcting the influence of optical distortion, which is another one of the two points different from the case of the fifth embodiment. The process (step S4) will be described with reference to FIG.
[0191]
FIG. 30A shows a developed image when mask processing is not performed, and FIG. 30B shows a developed image when mask processing is performed. In the developed image, 41 is an ATM terminal, 42 is an operator, and 47 is an operation panel of the ATM terminal 41. In FIG. 30A before the mask processing, the entire field of view is visible, and the operation of the operator 42, the input result, and the like are shown, so there is a risk that private information such as a password will be leaked to the outside. Here, the coordinate value data of the development table corresponding to the mask area 48 is different from the original coordinate value data, for example, all the development table elements corresponding to the mask area 48 are converted into the hyperboloid reflector 1 in the wide-angle image. The mask processing is realized by replacing the upper left corner or the like where the image from is not formed.
[0192]
By embedding the mask processing information in the expansion table as described above, the mask processing can be performed without increasing the calculation load at the same time when the expansion image generation unit 24 in FIG. 24 performs the expansion processing.
[0193]
Next, the developed image generating unit 24 generates a developed image using the developed table generated by the developed table generating unit 23 by performing the above processing. The difference from (Embodiment 5) in this case is that the expansion processing is performed so that the vertical direction has an angle of view of 180 degrees so that the top and bottom of the development image is correct as shown in FIG. is there.
[0194]
Next, the image quality correction processing unit 25 performs the same processing as in the case of (Embodiment 5), and the image format conversion unit (not shown) converts it into a video signal such as NTSC and displays the developed image on the monitor 27.
[0195]
Thus, according to (Embodiment 6), since the viewpoint of the wide-angle imaging device 8 can be virtually changed by changing the shape of the virtual projection plane 44, the internal focus 4 of the wide-angle imaging device 8 can be changed. It is possible to reduce the uncomfortable feeling of the developed image due to the difference in the distances. Further, by performing mask processing on a part of the developed image and outputting it after making it invisible, the operation contents and input contents of the operator are not shown, and it is possible to prevent leakage of privacy information.
[0196]
Although a method of embedding mask processing information in the development table is shown here, the image mask processing may be performed after the development image is generated. For example, as shown in FIG. 31, an image mask processing unit 49 may be provided immediately after the image quality correction processing unit 25, and processing for filling the inside of the mask area 48 with a luminance value of 0 may be performed.
[0197]
In (Embodiment 6), the hyperboloid reflector 1 has a reflection surface of ± 90 degrees, but is not limited to this. It may be within a range of ± 180 degrees or less.
[0198]
Further, in (Embodiment 6), the case where the wide-angle imaging device 8 captures a range of about ± 90 degrees around the rotation center axis is shown, but the wide-angle imaging device 8 is shown in FIG. Even when the omnidirectional camera 100 that captures a range of about 360 degrees around the rotation center axis is used and the image processing device 22 performs the expansion process corresponding to the omnidirectional camera 100, the virtual described above The change of the projection plane shape and the mask process have the same effects as in the case of (Embodiment 6). In addition, when changing the shape of the virtual projection plane with an omnidirectional camera, it is possible to express the movement of the viewpoint using a single wide-angle image, so use multiple wide-angle images with slightly different imaging locations. And walk-through software that can move the viewpoint smoothly.
[0199]
(Embodiment 7)
FIG. 32 shows (Embodiment 7) of the present invention.
The difference from the sixth embodiment is that the development table generation unit 23 which is one of the processing units in the image processing apparatus 22 is separated and independent, and a new expansion table generation arithmetic processing unit 51 is provided. In the image processing apparatus 22, a development table storage unit 50 is provided instead of the development table generation unit 23.
[0200]
The operation of FIG. 32 will be described focusing on the difference from (Embodiment 6).
The wide-angle imaging device 8 is the same as that used in (Embodiment 6).
The expansion table generation arithmetic processing unit 51 is specifically an arithmetic unit such as a microcomputer or a personal computer (PC). Here, the expansion table generation unit 23 of (Embodiment 6) performs the expansion table generation. Generate. The generated expansion table is sent to the image processing apparatus 22 and stored in the expansion table storage unit 50.
[0201]
The expansion table transfer method uses, for example, a memory card 52 such as an SD card (Secure Digital card registered trademark). An SD card slot (not shown) is provided in both the development table generation processing unit 51 and the image processing unit 22. The generated expansion table is recorded on the SD card by the expansion table generating arithmetic processing unit 51. The SD card is inserted into the SD card slot of the image processing apparatus 22, and the expansion table recorded in the SD card is read and stored in the expansion table storage unit 50.
[0202]
Next, the developed image generation unit 24 reads the development table stored in the development table storage unit 50 and performs a development process. The subsequent processing is the same as in (Embodiment 6).
As described above, according to (Embodiment 7), since it is not necessary to perform the expansion table generation process in the image processing apparatus 22, it is possible to eliminate the startup time for the expansion table generation process when the apparatus is activated.
[0203]
In (Embodiment 7), the expansion table generation unit 23, which is one of the processing units in the image processing apparatus 22 of (Embodiment 6), is separated and independent, and a new expansion table generation arithmetic processing device is provided. In this example, 51 is provided, and a development table storage unit 50 is provided in the image processing apparatus 22 instead of the development table generation unit 23. Even when the same change is made in the case of (Embodiment 5), the same effect as in the case of (Embodiment 7) can be obtained. In either case, the wide-angle imaging device 8 uses the omnidirectional camera 100 that captures a range of about 360 degrees around the rotation center axis as shown in FIG. Even if the generation arithmetic processing unit 51 is changed so as to perform processing corresponding to the omnidirectional camera 100, the same effect as in the seventh embodiment can be obtained.
[0204]
【The invention's effect】
As described above, according to the present invention described by taking (Embodiment 1) to (Embodiment 4) as examples, the wide-angle imaging device has a rotation center axis in a plane perpendicular to the reference optical axis direction. Even when tilted with respect to the vertical direction, the output of the wide-angle image generation device is an image in which the horizontal direction in the image matches the horizontal direction of the TV monitor, so that it does not give the viewer a sense of incongruity. Become.
[0205]
In addition, even when the wide-angle imaging device is tilted with respect to a certain reference state within a plane including the rotation center axis and the reference optical axis, the tilt angle is measured and the developed image is projected using the measured value. By making the central axis of the virtual cylindrical surface that is a surface always coincide with the rotation center axis in the reference state, it is possible to change the visual field range without changing the position of the line-of-sight center in the developed image. Furthermore, using the entire visual field range, it is possible to obtain a developed image in which the line-of-sight center is in the center in the short direction of the developed image. For this reason, the maximum distance from the line-of-sight center of an arbitrary point in the developed image can be minimized, and the part far from the line-of-sight center in the developed image is seen more obliquely than the other parts. The problem of distorted appearance compared to other parts can be reduced.
[0206]
In addition, according to the present invention described by taking (Embodiment 5) to (Embodiment 7) as examples, the optical axis of the imaging apparatus including the imaging lens and the imaging element is rotated as the optical system of the wide-angle imaging apparatus. Even with a configuration tilted with respect to the central axis, it is possible to determine the ideal corresponding point on the wide-angle image for each pixel of the developed image, and further, a development table that takes into account the effects of optical distortion of the imaging device Since this correction is performed, correct development processing can be performed. Further, by performing image quality correction processing, it is possible to reduce the difference in resolution depending on the location of the developed image.
[0207]
In addition, since the viewpoint of the wide-angle imaging device can be virtually changed by changing the shape of the virtual projection plane, the uncomfortable feeling of the developed image due to the difference in distance from the internal focal point of the wide-angle imaging device can be reduced. Is possible. Further, by performing mask processing on a part of the developed image and outputting it after making it invisible, the operation contents and input contents of the operator are not shown, and it is possible to prevent leakage of privacy information.
[0208]
Further, if the configuration using the development table generation arithmetic processing device is used separately from the image processing device, there is no need for the development table generation processing in the image processing device. Time can be lost.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of a wide-angle image generation apparatus according to (Embodiment 1) of the present invention.
FIG. 2 is an explanatory diagram of objects a, b, and c according to the embodiment.
FIG. 3 is a schematic diagram of an image formed on the imaging surface in the wide-angle image generation device according to the embodiment;
FIG. 4 is a schematic diagram of a result obtained by performing development processing on an image captured by the wide-angle image generation device according to the embodiment;
FIG. 5 is a schematic diagram of an output image output to the TV monitor according to the embodiment;
FIG. 6 is a schematic diagram of rotation of the wide-angle imaging device and an output image at that time.
FIG. 7 is a configuration diagram of a wide-angle image generation device according to (Embodiment 2) of the present invention.
FIG. 8 is a configuration diagram of the wide-angle image generation device according to (Embodiment 3) of the present invention as viewed from the X-axis direction (vertically upward).
FIG. 9 is a configuration diagram of the wide-angle image generation device according to (Embodiment 4) of the present invention as viewed from the X-axis direction (vertically upward).
FIG. 10 is a configuration diagram of a wide-angle image generation device according to (Embodiment 5) of the present invention.
FIG. 11 is an explanatory diagram of the wide-angle imaging device according to the embodiment.
FIG. 12 is a schematic diagram of an image formed on the imaging surface of the wide-angle imaging device according to the embodiment.
FIG. 13 is an explanatory diagram of objects a, b, and c in the same embodiment
FIG. 14 is an explanatory diagram of a wide-angle image obtained in the embodiment.
FIG. 15 is a development table creation flowchart according to the embodiment;
FIG. 16 is a configuration diagram of the wide-angle imaging device according to the embodiment;
FIG. 17 is an enlarged view of the vicinity of the imaging surface in the embodiment.
FIG. 18 is a layout diagram of a hyperboloid reflector and an imaging surface when viewed from the Z-axis direction in the same embodiment;
FIG. 19 is an explanatory diagram of the relationship between the angle θ on the Xa and Ya planes and the angle φ on the xy plane.
FIG. 20 is an explanatory diagram of the influence of optical distortion of the imaging apparatus.
FIG. 21 is an explanatory diagram of a test pattern image
FIG. 22 is an explanatory diagram of a difference in resolution between right and left of a developed image.
FIG. 23 is an explanatory diagram of a relationship between a position on a developed image and a sharpening coefficient.
FIG. 24 is a configuration diagram of a wide-angle image generation device according to (Embodiment 6) of the present invention.
FIG. 25 is an explanatory diagram of objects a, b, and c in the same embodiment
FIG. 26 is a schematic diagram of an image formed on the imaging surface in the embodiment.
FIG. 27 is a development table creation flowchart according to the embodiment;
FIG. 28 is an explanatory diagram showing a relationship between a virtual projection plane and an operator.
FIG. 29 is an explanatory diagram of a method for changing the virtual projection plane shape.
FIG. 30 is a schematic diagram when an ATM operation panel is masked.
FIG. 31 is a configuration diagram of a wide-angle image generation device according to the embodiment.
FIG. 32 is a configuration diagram of a wide-angle image generation device according to (Embodiment 7) of the present invention.
FIG. 33 is an explanatory diagram of development processing.
FIG. 34 is an explanatory diagram of an omnidirectional camera.
FIG. 35 is an explanatory diagram of development processing in an omnidirectional camera.
[Explanation of symbols]
1 Hyperbolic reflector
2 Imaging lens
3 Image sensor
4 Internal focal point of hyperboloid reflector 1
5 Imaging surface of image sensor 3
6 Rotation center axis of hyperboloid reflector 1
7 Main points of imaging lens 2
8 Wide-angle imaging device
9 Expanded image generator
10 Output image generator
11 arithmetic processing unit
12 Y-axis rotation angle sensor (inclination direction detector in the rotation center axis)
13 TV monitor
14 screens
15 Rotation center axis in-plane tilt direction input section
16 X-axis angle sensor
17 Rotation center axis forward tilt direction input section
18 Limit of field of view of wide-angle imaging device 8 when the rotation center axis is tilted
19 Field-of-view limitation of wide-angle imaging device 8 when the rotation center axis is tilted
20 Field-of-view range of the wide-angle imaging device 8 when the rotation center axis is tilted
Field of view of the wide-angle imaging device 8 in the 18 ′ reference state
19 'Limit of field of view of wide-angle imaging device 8 in reference state
Field of view of the wide-angle imaging device 8 in the 20 ′ reference state
21 Virtual cylindrical surface
22 Image processing device
23 Development table generator
24 Development image generation unit
25 Image quality correction processing section
26 Image mask processing section
27 Monitor
28 Optical axis
29a, 29b rays
30a, 30b incident point
31a, 31b, 31a₀, 31b₀    Imaging point
32, 32₀    Semicircular statue
33 screens
34 Virtual projection plane
35 rays
36 test patterns
37 Test pattern image
38 Wide angle image
39 Expanded image
40 blur area
41 ATM terminal
42 Operator
43 Tilt angle of the upper body of the operator
44 Virtual projection plane of (Embodiment 6)
45. Center axis of virtual projection plane of (Embodiment 6)
46 Reference optical axis direction
47 ATM terminal operation panel
48 Mask area
49 Image mask processing section
50 Development table storage unit
51 Arithmetic processing device for generating expansion table
52 Memory card

Claims (15)

The reference optical axis is taken in an arbitrary direction perpendicular to the rotation center axis of the rotationally symmetric body, the direction of the reference optical axis is 0 degree, and at least an angle θ range (θ1 ≦ θ ≦ θ2, −180 ° ≦ θ1 ≦ A reflecting mirror having a convex reflecting surface of 180 °, −180 ° ≦ θ2 ≦ 180 °, θ1 <θ2);
The principal axis is on the rotation center axis of the reflecting mirror, and when the angle of view is 2α, the optical axis is inclined in the reference optical axis direction by an angle of α or less with respect to the rotation center axis, and the reflection An imaging device that forms a reflection image of the mirror and images an object in a range of at least an angle θ around the rotation center axis of the reflection mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A wide-angle image generation apparatus provided with a development table generation unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image. The reference optical axis is taken in an arbitrary direction perpendicular to the rotation center axis of the rotationally symmetric body, the direction of the reference optical axis is 0 degree, and at least an angle θ range (θ1 ≦ θ ≦ θ2, −180 ° ≦ θ1 ≦ A reflecting mirror having a convex reflecting surface of 180 °, −180 ° ≦ θ2 ≦ 180 °, θ1 <θ2);
The principal axis is on the rotation center axis of the reflecting mirror, and when the angle of view is 2α, the optical axis is inclined in the reference optical axis direction by an angle of α or less with respect to the rotation center axis, and the reflection An imaging device that forms a reflection image of the mirror and images an object in a range of at least an angle θ around the rotation center axis of the reflection mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation processing unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
A wide-angle image generation apparatus that includes the expansion table storage unit that holds the expansion table generated by the expansion table generation processing unit and is connected to the expansion image generation unit. The hyperboloid shape of the reflecting mirror having an internal focal point on a rotation center axis, and a position of an external focal point of the hyperboloid shape and a position of a principal point of the imaging device are substantially coincident with each other. The wide-angle image generation device according to claim 1. The expansion table generation unit or the expansion table generation processing unit,
A development table including information on correspondence between a wide-angle image including distortion due to optical aberration of the imaging device and a wide-angle image after distortion correction, and pixel correspondence information obtained by combining the correspondence between the wide-angle image after distortion correction and the development image. The wide-angle image generation device according to claim 1 or 2, configured to generate the wide-angle image. The expansion table generation unit or the expansion table generation processing unit,
2. The development table configured to generate a development table in which the pixel correspondence between the wide-angle image and the developed image is changed by changing the shape of the virtual projection plane when considering the pixel correspondence between the wide-angle image and the developed image. The wide-angle image generation device according to claim 2. The expansion table generation unit or the expansion table generation processing unit,
2. A configuration is provided in which a pixel correspondence relationship between a wide-angle image and a development image in an arbitrary mask area provided on the development image is arbitrarily changed, and a development table that masks the development image in the mask area is generated. Alternatively, the wide-angle image generation device according to claim 2. 3. An image quality correction processing unit for performing high-frequency component enhancement processing in which the degree of enhancement is changed according to the position on the developed image is added as a post-process of the developed image generating unit after the developed image generating unit. The wide-angle image generation device described in 1. A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation unit that generates a development table including information on the pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
Expand table generation unit
A development table including information on correspondence between a wide-angle image including distortion due to optical aberration of the imaging device and a wide-angle image after distortion correction, and pixel correspondence information obtained by combining the correspondence between the wide-angle image after distortion correction and the development image. A wide-angle image generating device configured to generate. A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation unit that generates a development table including information on the pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
Expand table generation unit
A wide-angle image generation device configured to generate a development table in which the pixel correspondence between the wide-angle image and the developed image is changed by changing the shape of the virtual projection plane when considering the pixel correspondence between the wide-angle image and the developed image . A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation unit that generates a development table including information on the pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
Expand table generation unit
Wide-angle image generation configured to arbitrarily change the pixel correspondence between a wide-angle image and a developed image in an arbitrary mask area provided on the developed image, and generate a development table that masks the developed image in the mask area apparatus. A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation unit that generates a development table including information on the pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
A wide-angle image generation apparatus in which an image quality correction processing unit that performs high-frequency component enhancement processing in which the degree of enhancement is changed according to the position on the development image is added as a post-processing of the development image generation unit after the development image generation unit. A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation processing unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
While holding the expansion table generated by the expansion table generation arithmetic processing unit and providing a expansion table storage unit connected to the expansion image generation unit,
Expand the table for generating the processing unit,
A development table including information on correspondence between a wide-angle image including distortion due to optical aberration of the imaging device and a wide-angle image after distortion correction, and pixel correspondence information obtained by combining the correspondence between the wide-angle image after distortion correction and the development image. A wide-angle image generating device configured to generate. A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation processing unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
While holding the expansion table generated by the expansion table generation arithmetic processing unit and providing a expansion table storage unit connected to the expansion image generation unit,
Expand the table for generating the processing unit,
A wide-angle image generation device configured to generate a development table in which the pixel correspondence between the wide-angle image and the developed image is changed by changing the shape of the virtual projection plane when considering the pixel correspondence between the wide-angle image and the developed image . A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation processing unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
While holding the expansion table generated by the expansion table generation arithmetic processing unit and providing a expansion table storage unit connected to the expansion image generation unit,
Expand the table for generating the processing unit,
Wide-angle image generation configured to arbitrarily change the pixel correspondence between a wide-angle image and a developed image in an arbitrary mask area provided on the developed image, and generate a development table that masks the developed image in the mask area apparatus. A reflecting mirror having a convex reflecting surface;
An imaging device having a principal point on the rotation center axis of the reflecting mirror, imaging a reflection image of the reflecting mirror, and imaging an object in a range of 360 degrees around the rotation center axis of the reflecting mirror;
A developed image generating unit that generates a developed image viewed from the rotation center axis from a wide-angle image obtained from the imaging device;
A development table generation processing unit that generates a development table including information on a pixel correspondence relationship between the wide-angle image and the development image when the development image generation unit develops the wide-angle image;
While holding the expansion table generated by the expansion table generation arithmetic processing unit and providing a expansion table storage unit connected to the expansion image generation unit,
A wide-angle image generation apparatus in which an image quality correction processing unit that performs high-frequency component enhancement processing in which the degree of enhancement is changed according to the position on the development image is added as a post-processing of the development image generation unit after the development image generation unit.

Images