JP2005045513A - Image processor and distortion correction method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor and a distortion correction method for calculating a spatial position of a small region, realizing distortion correction processing in units of small regions, and applying accurate distortion correction even to image data captured through thinning and also data obtained by segmenting an optional region. <P>SOLUTION: The image processor with a distortion correction means for applying the distortion correction processing to image data generates a first position in a coordinate system corresponding to each pixel of a corrected image, converts the first position into a second position in the coordinate system in the image data before distortion correction according to the distortion correction coordinate conversion [formula 1], converts the second position in the coordinate system in the image data before the distortion correction into a coordinate in a setting region whose reference coordinate system corresponds to the coordinate system of an imaging face and thereafter carries out interpolation processing on the basis of the coordinate in the setting region to generate data of each pixel of the corrected image. The image processor carries out the distortion correction processing in units of the small regions included in the corrected image. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

［式１］におけるＺは、歪中心（Ｘｄ，Ｙｄ）から今注目している点（Ｘ，Ｙ）までの距離である。［式１］によって、補正画像の点（Ｘ，Ｙ）に対して歪んでいる元画像の方の座標（Ｘ’，Ｙ’）が算出される。［式１］について補足説明すると、Ｍは光学系のデータを用いて理論的に補正した際、補正後に画像がはみ出したり、不足したりする現象を補正するための補正倍率である。Ｓｘ，Ｓｙは間引き取込みなど、縦横で空間的なサンプリング間隔が異なる現象の補正をするためのサンプリング比である。Ｘｏｆｆ，Ｙｏｆｆは歪補正処理によって、歪補正処理後に、被写体位置が撮影時の位置とはずれてしまう現象の補正をする中心ずれの値である。
【００５３】
本発明に係る［式１］では、高次項（具体的にはＺ^４、Ｚ^６、…）を考慮することで、より複雑な陣笠型の歪（図４参照）にも対応できるようにしている。また、中心が一致していても撮像素子の一部しかデータを取り込まない場合や、光軸中心と撮像素子中心がずれている場合など、歪中心（Ｘｄ，Ｙｄ）を考慮する必要がある。さらに、歪補正処理により被写体位置が撮影時の位置からずれる現象を補正するため、中心ずれ補正値（Ｘｏｆｆ，Ｙｏｆｆ）を考慮する必要がある。間引き取込みの場合や輝度データ（Ｙ）と色データ（Ｃｂ，Ｃｒ）からなるＹＣ画像を処理する場合は、Ｘ，Ｙで空間的なサンプリングが異なる。そこで係数としてサンプリング比（ＳＸ，Ｓｙ）を導入して、座標変換を補正するようにしている。
【００５４】
次に、図５，図６を参照して画像データの書き込み及び読み出しの処理及び順序について説明する。
【００５５】
図５は、本実施の形態におけるフレームメモリからの画像データの読み出し順序を説明する図である。
画像データは、通常は、ライン方向、つまり行方向にスイープさせて書き込まれていて、読み出し時は行方向に読み出されるようになっており、１ラインの画像データを全て読み出して、次に隣接するラインの画像データを全て読み出すといった動作を繰り返して行うのが普通である。
【００５６】
これに対して、本発明に係る画像処理装置は、行方向にスイープさせて書き込まれた画像データを、列方向にある一定の長さを単位に、行方向の画像データを順番にイメージプロセス部７に入力し、以降、順次隣の列をイメージプロセス部７に入力して、画像の右端まで繰り返して得られる小領域（矩形状の画像データ）をブロックライン（ＢＬ）と呼ぶ。
【００５７】
こうした画像データの読み出しを可能にするための第１データ順序変換部６と、第２データ順序変換部９との構成を、図６を参照して説明する。図６は、第１，第２データ順序変換部の構成を示すブロック図である。
【００５８】
第１データ順序変換部６は、図６（Ａ）に示すように、ブロック単位の画像データを記憶可能なメモリを複数、ここでは２つ備えており、この２つのメモリ６ａとメモリ６ｂが書き込み側，読み出し側の各スイッチにて書き込み，読み出しが交互に切り換えられるようになっている。すなわち、フレームメモリ５は、書き込み側のスイッチにて、これらメモリ６ａとメモリ６ｂとに切換可能に接続されているとともに、イメージプロセス部７も読み出し側のスイッチにてこれらのモリ６ａとメモリ６ｂとに切換可能に接続されていて、フレームメモリ５がメモリ６ａとメモリ６ｂとの一方に接続されているときには、該メモリ６ａとメモリ６ｂとの他方がイメージプロセス部７に接続されるように切り換えられる。すなわち、メモリ６ａ，６ｂは、フレームメモリ５とイメージプロセス部７との両方に同時に接続されることがないようにスイッチングされ、交互に書き込み，読み出しが行われるようになっている。
【００５９】
フレームメモリ５に記憶されているフレーム画像の一部は、ブロック単位でライン方向に読み出され、一方のメモリ、ここでは例えばメモリ６ａに記憶される。
【００６０】
これと並行して、メモリ６ｂからは、既にフレームメモリ５から読み出されて記憶されているブロック単位の画像データが、列方向（縦方向）に順に読み出されて、イメージプロセス部７へ出力される。
【００６１】
フレームメモリ５からメモリ６ａへの書き込みと、メモリ６ｂからイメージプロセス部７への読み出しが終了すると、書き込み側のスイッチおよび読み出し側のスイッチが切り換えられて、次に、フレームメモリ５からメモリ６ｂへ次のブロック単位の画像データの書き込みが開始されるとともに、メモリ６ａからイメージプロセス部７へのブロック単位の画像データの読み出しが開始される。
【００６２】
第２データ順序変換部９も、図６（Ｂ）に示すように、上記第１データ順序変換部６とほぼ同様に構成されていて、ほぼ同様に動作するようになっている。
【００６３】
すなわち、第２データ順序変換部９は、メモリ９ａ，メモリ９ｂと、書き込み側スイッチ，読み出し側スイッチとを有して構成されている。
【００６４】
そして、該第２データ順序変換部９の動作時には、歪補正処理部８からの書き込みがメモリ９ａとメモリ９ｂとの一方に対して列方向（縦方向）に行われ、メモリ９ａとメモリ９ｂとの他方からは行方向（横方向）に読み出しが行われて、ＪＰＥＧ処理部１０へ出力されるようになっている。
歪補正座標変換の［式１］は、あくまでも全画面の中の画像であって、補正画像についてもその座標変換結果の（Ｘ’，Ｙ’）の画像にしても、全画面中で何処に位置しているかを計算している。
【００６５】
ところが、歪補正処理部８に入力される画像データはブロックライン単位である。歪補正処理部８としては、入力される画像データの空間的な位置関係（２次元空間における位置）を知る必要がある。制御レジスタ８５は、歪補正処理部８で処理するブロックラインの位置を指定している。
【００６６】
図７は、補正前の撮像画像データと補正後の補正画像の関係を示している。図７（ａ）は撮影画像データ、同図（ｂ）は補正画像を示している。図７（ａ），（ｂ）は補正画像（出力側画像）の或るブロックラインＢＬの或る画素位置Ａに注目している場合の概念図である。補正画像（出力画像）内の位置Ａの座標（Ｘ，Ｙ）は、歪補正座標変換の［式１］を用いて（入力画像）内の位置Ａ’の座標（Ｘ’，Ｙ’）に変換される。
【００６７】
座標位置（Ｘ，Ｙ）及び（Ｘｂｌｓｔ，Ｙｂｌｓｔ）、座標位置（Ｘ’，Ｙ’）及び（Ｘ’ｂｌｓｔ，Ｙ’ｂｌｓｔ）は、全画面の原点からの位置を示している。ここでは、原点は、全画面の左上端部としている。
【００６８】
歪補正ブロックへ入力される画像データは、符号ＢＬにて示すブロックライン単位である。そのため、まず補正画像の例えば画素Ａを歪座標変換処理するには、最終的な補間処理ができるようにブロックラインＢＬ内での座標系における座標位置（ｘ’，ｙ’）に変換する必要がある。
【００６９】
まず、図７（ｂ）の補正画像（出力画像）の各画素に対する座標位置（Ｘ，Ｙ）は、下記の式２で生成する。
【００７０】
［式２］
Ｘ＝Ｘｂｌｓｔ＋ｍ×ΔＸ
Ｙ＝Ｙｂｌｓｔ＋ｎ×ΔＹ
ΔＸ，ΔＹは補間ピッチで、１．０以上なら縮小画像が生成され、１．０以下なら拡大画像が生成される。座標位置（Ｘｂｌｓｔ，Ｙｂｌｓｔ）は歪補正ブロックの出力側ブロックラインの左肩の位置、すなわち、ブロックラインのスタート位置に相当している。（ｍ，ｎ）＝（０，０）のとき、Ｘ＝Ｘｂｌｓｔ，Ｙ＝Ｙｂｌｓｔとなるので、［式２］から座標位置（Ｘｂｌｓｔ，Ｙｂｌｓｔ）を算出できることになる。ｍ＝０，ｎ＝０から始めて、ｍ，ｎを例えば交互に１ずつ増やしていけば、ブロックラインＢＬ内の全ての画素の座標位置（Ｘ，Ｙ）を生成することができる。
【００７１】
このＢＬに対し、ＢＬ内全ての画像の座標位置を式１に従って変換すると、図７（ａ）の変形したハッチング領域ＢＬ″にマッピングされる。図７（ａ）のＢＬ’は、図７（ｂ）のＢＬを処理するのに必要な入力データの範囲となる。（Ｘ’ｂｌｓｔ，Ｙ’ｂｌｓｔ）は、ＢＬ’の左上位置であり、補間に必要な画素等を考慮すると、変形したハッチング領域の左上位置とは一致しない。ＢＬ’は、本願と同日出願の特願２００３‐２０２４９３号に詳細を説明したサポート機能（歪補正範囲算出部）を利用して設定する。
【００７２】
［式２］で生成された補正後の出力画像の座標位置（Ｘ，Ｙ）を、［式１］で座標変換して図７（ａ）の補正前の入力画像の座標位置（Ｘ’，Ｙ’）を求める。そして、座標変換された座標位置（Ｘ’，Ｙ’）と、座標位置（Ｘ’ｂｌｓｔ，Ｙ’ｂｌｓｔ）との差分をとることにより、ブロックライン（太い点線の矩形）の左肩を原点とする座標位置（ｘ’，ｙ’）を求めることができる。
【００７３】
その結果、このブロックライン内での座標位置（ｘ’，ｙ’）に基づき、ブロックライン単位での補間処理を行えることになる。
【００７４】
図７（ａ）に示す入力画像側の太い点線で示すブロックラインＢＬ’はレジスタ８５によって設定するが、その値を生成するためには、歪の変形を考慮して画像データの入力範囲（歪補正範囲）を算出可能とするサポート機能が必要となる。このサポート機能は、歪補正処理部８の歪補正処理機能に付加するなどして設ければよい。
【００７５】
次に、図８を参照して歪補正処理フローを説明する。図８は図７の歪補正処理のフローチャートである。
【００７６】
まず、図７（ｂ）のハッチングを施したところのブロックライン処理を開始する（ステップＳ１）。前述の［式２］で、（ｍ，ｎ）＝（０，０）とおくことにより（ステップＳ２）、図７（ｂ）のハッチング領域の左肩、すなわちブロックラインのスタート座標位置（Ｘｂｌｓｔ，Ｙｂｌｓｔ）を求める。そして、［式２］から補正画像の座標位置（Ｘ，Ｙ）を求めることができる。この場合、ｍ，ｎをそれぞれ１ずつ増やしていけば、図７（ｂ）のハッチングで示すブロックラインＢＬ内の各画素の座標位置（Ｘ，Ｙ）を生成することができる（ステップＳ３）。
【００７７】
次に、［式１］を用いて補正画像における座標系の位置（Ｘ，Ｙ）を、補正前の撮影画像における座標系の位置（Ｘ’，Ｙ’）に変換する（ステップＳ４）。
【００７８】
そして、撮影画像における座標系の位置（Ｘ’，Ｙ’）を、前述した方法で入力ブロックラインＢＬ内の座標系の位置（ｘ’，ｙ’）に変換する（ステップＳ５）。
【００７９】
次に、このブロックラインＢＬ内での座標位置（ｘ’，ｙ’）に基づいて補間処理を行い、座標位置（Ｘ’，Ｙ’）における１６点補間などで周辺の複数の画素データから補間データを算出することで、１画素分の補正画像データが得られる（ステップＳ６）。
【００８０】
そして、以上のステップが、補正画像の座標系のＢＬ内全画素について行われたか否かについて判断し（ステップＳ７）、終了していなければ次のｍ，ｎを設定して（ステップＳ８）、ステップＳ３〜Ｓ７を繰り返し行い、ステップＳ７の判定で全画素について処理が終了していれば、ステップＳ９に移行する。
【００８１】
ステップＳ９では、全画面（フレーム）内の全てのブロックラインＢＬについて歪補正処理が終了したか否かを判断しており、終了していなければ次の入力ブロックラインＢＬを選択し、ステップＳ１〜ステップＳ９を繰り返し、ステップＳ９の判定で全画面ついて処理が終了していれば、終了する。
【００８２】
図９及び図１０は、図８の変形例とも言うべきものである。
デジタルズームを行う場合や注目する小領域のみを切り出す場合、撮像データの一部しか使わなくても構わない。選択的にＣＣＤから撮像データの一部を取り込むことでメモリを節約することができる。この場合に歪補正をブロックライン単位で行うためには、取り込まれたデータが、撮像素子上でどこに位置するかの情報（Ｘ’ｉｍｇｓｔ，Ｙ’ｉｍｇｓｔ）が必要である。
【００８３】
図９は、デジタルズームの場合を示し、撮像した画像データから中央部分のみを取り込むものである。図８の場合と異なる点は、上述したように、全画面の中で何処を取り込むかをプリプロセス回路２で設定している点である。この設定に基づいて必要な部分のみを取り込んでおり、その範囲を図９（ａ）のＥ’で示す。
【００８４】
まず、図９（ｂ）で全画面の中のデジタルズームする範囲Ｅの左肩に当たるスタート位置（Ｘｉｍｇｓｔ，Ｙｉｍｇｓｔ）を設定する。次に、ブロックラインＢＬを設定するが、ここでは梨地にて示す部分である。このとき、取り込み範囲Ｅの左肩を原点とする、位置座標（Ｘｂｌｓｔ，Ｙｂｌｓｔ）が決定し、［式２］と同様にｍ，ｎを変化させることで、ブロックラインＢＬ内における各画素の座標を生成することができる。
【００８５】
この結果、座標位置（Ｘｉｍｇｓｔ，Ｙｉｍｇｓｔ）、（Ｘｂｌｓｔ，Ｙｂｌｓｔ）、（ｍ×ΔＸ，ｎ×ΔＹ）から全画面の座標における位置（Ｘ，Ｙ）が決定される。
【００８６】
撮影画像における座標位置（Ｘ’，Ｙ’）は、上記のように決定された座標位置（Ｘ，Ｙ）から［式１］を用いて求めることができる。
【００８７】
そして次に、撮影画像データにおけるブロックラインＢＬ’の中での座標位置（ｘ’，ｙ’）を求める。
【００８８】
まず、取り込まれた撮像データの撮像面における設定範囲Ｅ’から左肩の座標（Ｘ’ｉｍｇｓｔ，Ｙ’ｉｍｇｓｔ）が分かる。取り込み範囲Ｅ’の左肩を原点とする座標で、ブロックラインＢＬ’の左肩の座標位置（Ｘ’ｂｌｓｔ，Ｙ’ｂｌｓｔ）はサポート機能を利用して予め求められており、ブロックラインＢＬ’内での座標における座標位置（ｘ’，ｙ’）が決定される。その結果、ブロックラインごとの補間処理が行われる。
【００８９】
図１０は撮像データから或る領域、例えば画面の端の方を切り出す場合である。これは、監視カメラなどにおいて、撮像データの端部分を取り込むような場合に相当する。図１０の場合も、図９と同様な符号を付してある。図９と切り出す位置が異なっているだけで、図９の場合と同様な動作となる。
【００９０】
なお、通常のデジタルカメラでは、ＣＰＵが制御レジスタに切り出す領域を設定するため、切り出し位置が分かっており、上記の変換が可能だが、記録後のデータに歪補正処理を行う場合ではタグ等に切り出し領域の位置とサイズを同時に記録する必要がある。このとき、同時に補正係数も記録しておく。
【００９１】
ところで、撮像素子としてのＣＣＤにはモニターモード（撮像した全画面を読み出さないモード）が用意されることが多い。モニターモードでは、ラインが間引かれることが多く、撮像素子における２次元空間的には同一でも、データ上ではサイズが異なることがある。また、プリプロセスにおいて、横方向の画素も間引かれることがある。また、ＣＣＤからの撮像データにローパスフィルタ（ＬＰＦ）などのフィルタリング処理をかけたり、補間処理をしてフレームメモリに取り込むことがある。この場合、縦横のサンプリング比が整数にならない場合が生じる。さらに、ＪＰＥＧにおけるＹＣｂＣｒの記録では、ＣｂやＣｒ成分の横方向は２倍の間隔でサンプリングされることが多く、データとしては横方向が縮んだ形になる。これらは、［式１］では、Ｓｘ，Ｓｙによって補正する。
【００９２】
図１１（ａ）は、撮像時のデータ取り込みにおいて、実際にＣＣＤから送られてくるデータが黒丸印のところしかなく、縦方向に３ライン間引いている状態を示している。モニターモードでは、このようにＣＣＤからは間引かれたデータしか出力されず、これをメモリに取り込むと、図１１（ｂ）のように間引きによって上下方向につぶれた画像データとなる。図１１（ｃ）のように、空間的に中心から同じ距離にあるデータを取り出しても、同図（ｂ）のように横方向に対して画像が短く出力されるという現象が現れる。この結果、［式１］による歪補正では、中心からの距離が遠いほど歪補正の効果が大きくなるようにしてあるので、縦と横で、歪補正の効果が変わってしまう。
【００９３】
このように縦と横で画像が非対称となるのを補正する目的で、補正係数Ｓｘ，Ｓｙが設けられている。図１１（ｂ）の場合には、縦方向に伸ばす必要があるため、Ｓｘ＝１，Ｓｙ＝４とする。
【００９４】
このＳｘ，Ｓｙの効果により、縦，横のサンプリング比が異なっても正確な歪補正が可能となる。
【００９５】
また、前述のように、サンプリング比が整数でない場合は、Ｓｘ，Ｓｙも非整数で設定すればよい。さらに、撮像素子の各画素の間隔が縦横で異なる場合でも可能（例えば長方画素）である。
【００９６】
ところで、通常、ＪＰＥＧだとＹＣｂＣｒというデータで記録されるのが一般的であるが、そのときの設定として、図１２に示すような場合が考えられる。例えば、ＹＣ４２２という形式のデータの場合は、輝度データに比べて色データについての解像度は横方向が１／２となっている。この場合は、色データについてはＳｘ＝２，Ｓｙ＝１と設定して、補正処理を行えばよい。
【００９７】
ところで、歪の中心と画像の中心が一致しない場合、補正前に中心にあった被写体が、補正後は中心からずれてしまう。［式１］ では、中心ずれ補正値（Ｘｏｆｆ，Ｙｏｆｆ）で補正することにより、補正前後で画像の中心に来る被写体が変化しないようにすることが可能である。
【００９８】
図１３は、画像の中心ずれを説明する図である。
図１３は補正画像を示す。前述の図３（ｂ）では、歪の中心と画像の中心が一致している、すなわち、ＣＣＤの中心（画像の中心）がレンズ光軸（歪の中心）に一致している場合であった。
【００９９】
図１３は、ＣＣＤの中心に対して歪の中心がずれている場合である。図１３では、歪中心（Ｘｄ，Ｙｄ）が画像の中心と異なるので、中心ずれを考慮せずに歪補正を行った場合、出力範囲Ｈに対する画像の中心ＰがＱ点（大きな黒丸にて示す）にずれる。ユーザにとっては、実際に撮影した画像と歪補正後に表示される画像の中心がずれると、非常に違和感を感じるものである。そこで、元々歪補正前に出力範囲Ｈの中心にあったデータを、出力範囲に対して中心の位置にくるようにするために、歪補正された画像データの切り出した方を変える。それを行うためには、画像の中心ずれを（Ｘｏｆｆ，Ｙｏｆｆ）で定義して、画像データの切り出した方を変えればよい。画像の中心ずれ（Ｘｏｆｆ，Ｙｏｆｆ）を設定することで出力範囲をＨからＨ’に移動させ、元々画像の中心Ｐにあった被写体を、補正後の出力範囲Ｈ’においても中心にくるように補正することができる。
【０１００】
図１３の実施の形態によれば、中央の被写体がずれることによる違和感を無くすことが出来る。
【０１０１】
【発明の効果】
以上述べたように本発明によれば、小領域（例えばブロックライン）の空間的な位置を算出可能とし、小領域単位での歪補正処理を実現することができる。また、間引き取り込みされた画像データに対しても、正確な歪補正を行える一方、デジタルズームなど、任意領域を切り出したデータに対しても正確な歪補正を行うことができる。
【図面の簡単な説明】
【図１】本発明の一実施の形態の画像処理装置の全体構成を示すブロック図。
【図２】図１における歪補正処理部の構成を示すブロック図。
【図３】歪補正処理部における座標変換の概念図。
【図４】陣笠型の歪を生じた撮影画像を示す図。
【図５】本実施の形態におけるフレームメモリからの画像データの読み出し順序を説明する図。
【図６】第１，第２データ順序変換部の構成を示すブロック図。
【図７】補正前の撮像画像データと補正後の補正画像の関係を示す図。
【図８】図７の歪補正処理のフローチャート。
【図９】デジタルズームの場合で、補正前の撮像画像データと補正後の補正画像の関係を示す図。
【図１０】撮像データから或る領域を切り出す場合で、補正前の撮像画像データと補正後の補正画像の関係を示す図。
【図１１】ＣＣＤのモニターモードにおけるライン間引きを説明する図。
【図１２】ＹＣ４２２形式のデータにおける横方向の間引きを説明する図。
【図１３】画像の中心ずれを説明する図。
【図１４】格子状の被写体、樽型歪を生じた撮影画像、糸巻き型歪を生じた撮影画像を示す図。
【図１５】一般的なデジタルカメラの画像処理手順の概念、及び従来のデジタルカメラの画像処理装置におけるブロック構成を示す図。
【図１６】ＣＣＤの画素と像の関係を示す図。
【符号の説明】
１…ＣＣＤ
３…バス
４…ＣＰＵ
５…フレームメモリ
６…第１データ順序変換部
７…イメージプロセス回路
６…第１データ順序変換部
８…歪補正処理部（歪補正手段）
９…第２データ順序変換部
１０…ＪＰＥＧ処理部
１１…メモリカード（記録媒体）
８２…内部メモリ部（バッファ）
８３…メモリ制御部
８５…制御レジスタ[0001]
BACKGROUND OF THE INVENTION
The present invention mainly relates to an image processing apparatus and a distortion correction method used in an electronic imaging apparatus such as a digital camera, and in particular, when performing distortion correction processing in units of small areas (for example, block lines). The present invention relates to an image processing apparatus and a distortion correction method capable of calculating a spatial position.
[0002]
[Prior art]
In general, distortion aberration appears in the optical system of a camera regardless of whether it is a digital camera or a silver salt camera. In addition, most cameras that are currently available are capable of optical zooming, and in this case, the state of distortion changes from the wide end to the tele end. There are many barrel distortions at the wide end, and there are many pincushion distortions at the tele end.
[0003]
The distortion is observed as barrel distortion or pincushion distortion when, for example, a grid-like subject is photographed.
[0004]
FIG. 14A shows a grid-like subject, FIG. 14B shows a captured image with barrel distortion, and FIG. 14C shows a captured image with pincushion distortion.
[0005]
By the way, in a digital camera, after various image processing is performed on data of an image sensor such as a CCD, data compressed by a compression method such as JPEG is recorded on a recording medium such as a memory card.
[0006]
FIG. 15A shows the concept of an image processing procedure performed by a general digital camera. The imaging signal captured from the CCD is subjected to processing such as pixel defect processing and A / D conversion in a preprocess, and is temporarily stored in a frame memory such as an SDRAM. Next, the image data stored in the frame memory is read out, and various image processes are performed by the image process 1, the image process 2,. The image data after the image processing is further compressed by a compression method such as JPEG processing and recorded on a memory card or the like as a recording medium.
[0007]
FIG. 15B is a block diagram of an image processing apparatus of a conventional digital camera (for example, a conventional example in Japanese Patent Laid-Open No. 2000-31327).
[0008]
In FIG. 15B, a conventional image processing apparatus includes a pre-processing circuit 102, a plurality of image processing circuits 106-1 to 106-n, a JPEG processing unit 107, a frame memory 105, and a recording medium as well as a CPU 104 on a bus 103. A memory card 108 is connected. Under the control of the CPU 104, the image pickup signal from the CCD 101 is subjected to pixel defect processing, A / D conversion, and the like in the preprocess circuit 102, and then temporarily stored in the frame memory 105 through the bus 103. Next, image data is read from the frame memory 105 and input to the image process circuit 106-1 through the bus 103 to perform predetermined image processing, and the image data is rewritten to the frame memory 105 through the bus 103 again. In the same manner, data is sequentially exchanged between the frame memory 105 and the image process circuits 106-2 to 106-n via the bus 103. Finally, the JPEG processing unit 107 performs JPEG compression processing, and the image The processed data is temporarily stored in the frame memory 105, and the processed data read from the frame memory 105 is recorded in the memory card 108 or the like. In the above image process and JPEG processing in each stage, image processing is performed in units of small areas (block lines).
[0009]
On the other hand, reading of an image pickup device such as a CCD can normally be performed in two types of reading: (1) all-pixel reading for main shooting and (2) thinning-out reading for through images. Image data obtained by extracting a part of the imaged data by thinning-out reading or the like has an aspect ratio that is not 1: 1, so that accurate distortion correction cannot be performed as it is.
[0010]
Conventionally, prior art for correcting distortion as part of this image processing has been disclosed. For example, there exists Unexamined-Japanese-Patent No. 6-181530.
[0011]
Japanese Patent Laid-Open No. 6-181530 discloses an image generated by an imaging zoom lens when the detection unit detects that the imaging position of the imaging zoom lens at the time of shooting is within a position where distortion is large. Is corrected by reading out imaging data of a solid-state imaging device based on geometric deformation.
[0012]
[Patent Document 1]
JP 2000-31327 A (page 3, FIG. 19)
[0013]
[Patent Document 2]
Japanese Patent Laid-Open No. 6-181530 (first and second pages, FIG. 2)
[0014]
[Problems to be solved by the invention]
However, the conventional example of Patent Document 1 has a problem in that data exchange between the frame memory and each image process circuit is large, and the data transfer amount of the bus increases. Furthermore, there is no description of distortion correction processing in units of small areas (block lines).
[0015]
In Patent Document 2, it is described that distortion correction processing is performed. However, as shown in FIG. 16, the pixel is essentially a straight line with respect to pixels (vertical and horizontal lattice points indicated by black circles) 31 on the CCD imaging surface. When a desired subject image is curved and formed by the distortion of the optical lens (indicated by reference numeral 32), a large-capacity buffer memory is required for the line memory in the distortion correction circuit. Furthermore, the image size that can be processed is limited by the capacity of the buffer memory. Furthermore, there is no description about a method for determining the spatial position of data held in the buffer memory. Furthermore, Patent Document 2 does not describe any distortion correction processing in units of small areas (block lines).
[0016]
Therefore, in view of the above problems, the present invention provides an image processing apparatus and a distortion correction method capable of calculating a spatial position of a small area (for example, a block line) and realizing distortion correction processing in units of small areas. It is intended to do.
[0017]
The present invention also provides an image processing apparatus and distortion correction capable of performing accurate distortion correction on thinned-in image data, and capable of performing accurate distortion correction on data obtained by cutting out an arbitrary area, such as digital zoom. It is intended to provide a method.
[0018]
[Means for Solving the Problems]
The invention according to claim 1 is an image processing apparatus including a distortion correction unit that performs distortion correction processing on image data. The distortion correction unit is a position corresponding to each pixel of the corrected image in the image data before correction. Is calculated in a coordinate system in which a spatial position on the imaging surface can be described.
[0019]
In this configuration, when calculating the position corresponding to each pixel of the corrected image in the image data before correction, the calculation is performed based on the coordinate system corresponding to the imaging surface (that is, the position in the two-dimensional space).
[0020]
According to a second aspect of the present invention, in the image processing apparatus according to the first aspect, the distortion correction unit performs processing in units of first small areas included in the corrected image, and each pixel of the first small area. After the position in the coordinate system corresponding to the imaging surface is converted into the coordinates in the second small region for the second small region on the imaging surface including the small region converted according to the correction formula The data of each pixel of the corrected image is generated.
[0021]
In this configuration, interpolation processing can be performed based on the coordinates in the small area to be converted.
[0022]
According to a third aspect of the present invention, in the image processing apparatus according to the first or second aspect, the image data is partial image data in which only a part of the imaging data is captured.
[0023]
In this configuration, as the image data, not all of the captured data, but also a part thereof, for example, data cut out from the central portion of the imaged data at the time of digital zoom, or reduced data generated by thinning out of the imaged data, Can be applied.
[0024]
According to a fourth aspect of the present invention, in the image processing apparatus according to the third aspect, the partial image data is data obtained by cutting out a part of the imaging data.
[0025]
Data in this configuration includes, for example, data cut out from the central portion of the imaging data at the time of digital zooming, and data of an area corresponding to the end portion of the screen captured from the full screen data like a surveillance camera.
[0026]
According to a fifth aspect of the present invention, in the image processing apparatus according to the third aspect, the partial image data is data thinned out in at least one horizontal or vertical direction of the imaging data.
[0027]
As data in this configuration, reduced data generated by thinning out of imaged data, reduced data, data obtained by thinning out lines in the CCD monitoring mode, and color data in YC422 data are horizontally thinned. There is data drawn. The data may be thinned out in both the vertical and horizontal directions.
[0028]
According to a sixth aspect of the present invention, in the image processing apparatus according to the first aspect, the image data is data generated by performing at least one of filtering, spatial sampling, and interpolation processing on the imaging data. It is characterized by that.
[0029]
In this configuration, as the image data, filtering that is limited in terms of frequency, sampling appropriately from a two-dimensional space such as the imaging surface of the imaging device, or pixel that has been discretized by interpolation processing (sampling processing, etc.) Between the pixel and the pixel, there is data obtained by performing a process of generating a pixel by estimation from surrounding pixels).
[0030]
According to a seventh aspect of the present invention, in the image processing apparatus according to the first aspect, in the image data, an interval in which each pixel is spatially positioned on the imaging surface is different in the vertical direction and the horizontal direction.
[0031]
The data in this configuration is data when the spatial sampling interval is different in the vertical direction and the horizontal direction. For example, there are data in which the lines in the CCD monitoring mode are thinned out, color data in the YC422 data thinned out in the horizontal direction, and the like.
[0032]
In the image processing apparatus according to any one of claims 1 to 7, when the spatial sampling interval of each pixel in the image data is different vertically and horizontally, the coordinates in the image before distortion correction are provided. The correction equation for obtaining the above is characterized in that it includes a coefficient for correcting the difference in the sampling interval in the vertical and horizontal directions.
[0033]
This configuration means that the correction coefficients Sx and Sy for correcting the phenomenon in which the vertical and horizontal spatial sampling ratios are different in the distortion correction formula are included.
[0034]
According to a ninth aspect of the present invention, in the image processing apparatus according to any one of the first to eighth aspects, the correction formula for obtaining coordinates in the image before distortion correction can perform a predetermined amount of offset correction. Features.
[0035]
In this configuration, it means that the phenomenon (offset phenomenon) in which the subject position is shifted from the time of shooting is corrected by the distortion correction processing in the distortion correction formula.
[0036]
According to a tenth aspect of the present invention, in the image processing apparatus according to the ninth aspect, the offset correction is performed when a distortion center and an image center do not coincide with each other.
[0037]
In this configuration, for example, when the distortion center (optical axis) corresponding to the center of geometric deformation of the optical system lens does not coincide with the center of the captured image (CCD center), the image is corrected before and after the correction. The center shift of the image relative to the output image range can be eliminated by the center shift correction coefficients Xoff and Yoff.
[0038]
The invention of claim 11 is a distortion correction method of an image processing apparatus having a distortion correction means for performing distortion correction processing on image data, wherein the distortion correction method is performed in a coordinate system corresponding to each pixel of the corrected image. Generating a first position, converting the first position into a second position in a coordinate system in image data before distortion correction according to a distortion correction formula, and the image before distortion correction A step of converting the second position in the coordinate system in the data into a coordinate in the setting area with reference to the coordinate system corresponding to the imaging surface, and performing an interpolation process based on the coordinates in the setting area And a step of generating data of each pixel of the corrected image.
[0039]
In this distortion correction method, for example, when data of each pixel of the corrected image is obtained by a correction formula that corrects geometric distortion caused by the imaging lens, the first position in the coordinate system corresponding to each pixel of the corrected image is obtained. In contrast, the second position in the coordinate system in the image data before distortion correction is converted according to the distortion correction formula, and the converted second position is set with reference to the coordinate system corresponding to the imaging surface. Conversion into coordinates within the region is performed, and interpolation processing is performed based on the coordinates to generate pixel data of a corrected image.
[0040]
According to a twelfth aspect of the present invention, in the distortion correction method according to the eleventh aspect, the processing is performed in units of small areas included in the corrected image.
[0041]
In this method, processing is performed in units of small areas (for example, areas called block lines) included in the corrected image.
[0042]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing the overall configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing the configuration of the distortion correction processing unit in FIG.
[0043]
In the image processing apparatus of FIG. 1, an image pickup signal from the CCD 1 is subjected to pixel defect processing, A / D conversion, and the like by the preprocess circuit 2 under the control of the CPU 4 that controls each unit connected to the bus 3. The obtained image data is temporarily stored in the frame memory 5 via the bus 3. The frame memory 5 is configured by an SDRAM or the like and stores data before image processing and data after image processing. Next, the image data read from the frame memory 5 is input to the first data order conversion unit 6 via the bus 3. As will be described later with reference to FIG. 6A, the first data order conversion unit 6 includes a plurality of, in this case, two memories capable of storing data in block units. The first data order conversion unit 6 reads and stores data from the frame memory 5 in the row direction, and then sequentially reads the data in the column direction and outputs the data to the image process circuit 7.
[0044]
The image process circuit 7 performs predetermined image processing on the input data and transfers it to the distortion correction processing unit 8 as the distortion correction means at the next stage. The distortion correction processing unit 8 performs distortion correction processing on the input data and transfers it to the second data order conversion unit 9 at the next stage. As will be described later with reference to FIG. 6B, the second data order conversion unit 9 includes a plurality of, in this case, two memories that can store data in block units. The second data order conversion unit 9 reads and stores data in the column direction from the distortion correction processing unit 8, then sequentially reads the data in the row direction, and transfers the data to the JPEG processing unit 10. The JPEG processing unit 10 performs JPEG compression processing, temporarily stores the processing data in the frame memory 5, and records the processing data read from the frame memory 5 in the memory card 11 or the like.
[0045]
As shown in FIG. 2, the distortion correction processing unit 8 positions the corrected image after distortion correction (interpolation positions, X and Y) and the corresponding original image position before distortion correction (X ′, Y). ') And an internal memory unit as a buffer memory (hereinafter simply referred to as a buffer) for temporarily storing part of the image data from the circuit of the preceding block of the distortion correction processing unit 8 82, a memory control unit 83 for controlling writing and reading with respect to the internal memory unit 82, and image processing is performed according to the converted coordinates (X ′, Y ′) of the image position before distortion correction to correct distortion of the data. And an interpolation calculation unit 84.
[0046]
As shown in FIG. 2, the interpolation coordinate generation unit 81 includes a coordinate generation unit 811 that generates an interpolation coordinate (X, Y), and a predetermined distortion correction formula [formula for the generated interpolation coordinate (X, Y). 1] A distortion correction coordinate conversion unit 812 that outputs coordinates (X ′, Y ′) before correction converted by applying (described later), and interpolation coordinates (X, Y) and distortion correction from the coordinate generation unit 811 The selector 813 can selectively output the converted coordinates (X ′, Y ′) from the coordinate conversion unit 812. The coordinate generation unit 811, the distortion correction coordinate conversion unit 812, the selector 813, and the memory control unit 83 in the distortion correction processing unit 8 operate according to the set value for each block set in the control register 85 that stores control data. Further, the status of the processing result can be referred from the CPU.
[0047]
In the image processing apparatus according to the present invention configured as described above, the first data order conversion unit 6 to the JPEG processing unit 10 are piped through an information transmission path different from the bus 3 without going through the bus 3. It is connected so that line processing is possible, and image data is transferred and processed in units of predetermined blocks in a two-dimensional pixel array. As described above, the data transfer via the bus 3 includes the transfer from the frame memory 5 to the first data order conversion unit 6, the transfer from the JPEG processing unit 10 to the frame memory 5, and the transfer from the frame memory 5 to the memory card 11. Therefore, compared with the conventional example (FIG. 15B) in which data is exchanged between the frame memory and each image process circuit, the data transfer amount by the bus 3 can be greatly reduced. The load on the bus 3 can be greatly reduced. Further, in the example shown in FIG. 1, only one image process circuit 7 for performing image processing is provided, but a plurality of image process circuits 7 may be provided. Furthermore, in the figure, the distortion correction processing unit 8 is provided at the subsequent stage of the image process circuit 7, but the configuration may be reversed.
[0048]
In the image process circuit unit configured by the first-stage image process circuit 7 and the distortion correction processing unit 8 which is the second-stage image process circuit, a pipeline register is not shown as a pipeline register before or inside each of the image process circuits 7 and 8. A small-capacity memory is arranged, and the image process circuits 7 and 8 are configured to perform pipeline processing operations via the small memory. These small-capacity memories are used to store peripheral data necessary for image processing when the image processing circuits 7 and 8 perform spatial image processing, and read out the image data in units of blocks and rearrange the data. It is provided because it is necessary to carry out processing and the like.
[0049]
FIG. 3 shows a conceptual diagram of coordinate conversion in the distortion correction processing unit 8. 3A shows the captured image data as the original data, FIG. 3B shows the corrected image, FIG. 3C shows the coordinate position (X, Y) of the corrected image in FIG. Data coordinates (X ′, Y ′) at the coordinate position P converted to the coordinates of the original data (the coordinates are not necessarily coincident with the positions of a plurality of pixels actually constituting the original data) ). The coordinate position (X ′, Y ′) at the P point is calculated using the coordinates of the 16 pixels around the P point, and the image data at the P point is the pixel data of the surrounding 16 points. Is used for interpolation calculation. The interpolation calculation unit 84 performs processing to interpolate the data at the position of the point P from the surrounding 16 pixel values (luminance data). Interpolation coordinates are generated by the interpolation position generation unit 811 in FIG. 2 as to which pixel position (X, Y) is indicated on the corrected image side in FIG.
[0050]
In [Expression 1], the pixel position (X ′, Y ′) before distortion correction with respect to the pixel position (X, Y) after distortion correction can be calculated. However, as described above, the pixel position (X ′, Y ′) before distortion correction is not always an integer value corresponding to the pixel position on the original image data.
[0051]
According to [Expression 1], the coordinates (X, Y) in FIG. 3B are converted into coordinates (X ′, Y ′) as shown in FIG. As a result, it is possible to know where the coordinate position data in the original data should be created, using an interpolation formula that performs 16-point interpolation processing on the data at that position (X ′, Y ′). It can be obtained by calculating from 16 known pixel values (input image data indicated by black circles in FIG. 3C).
[0052]
[Formula 1]

Z in [Formula 1] is the distance from the strain center (Xd, Yd) to the point (X, Y) of interest. [Expression 1] calculates the coordinates (X ′, Y ′) of the original image that is distorted with respect to the point (X, Y) of the corrected image. A supplementary explanation of [Formula 1] is that M is a correction magnification for correcting a phenomenon in which an image protrudes or becomes insufficient after correction when theoretically correcting using data of the optical system. Sx and Sy are sampling ratios for correcting a phenomenon in which spatial sampling intervals differ vertically and horizontally, such as thinning-in. Xoff and Yoff are center deviation values for correcting a phenomenon in which the subject position deviates from the shooting position after the distortion correction process by the distortion correction process.
[0053]
In [Formula 1] according to the present invention, a high-order term (specifically, Z ⁴ , Z ⁶ ,...), It is possible to cope with more complicated Jinkasa type distortion (see FIG. 4). In addition, it is necessary to consider the distortion center (Xd, Yd), for example, when only a part of the image sensor is captured even if the centers coincide, or when the center of the optical axis and the center of the image sensor are shifted. Furthermore, in order to correct a phenomenon in which the subject position deviates from the position at the time of shooting by the distortion correction process, it is necessary to consider center deviation correction values (Xoff, Yoff). In the case of thinning-in or when processing a YC image composed of luminance data (Y) and color data (Cb, Cr), spatial sampling differs between X and Y. Therefore, the sampling ratio (SX, Sy) is introduced as a coefficient to correct the coordinate conversion.
[0054]
Next, processing and order of image data writing and reading will be described with reference to FIGS.
[0055]
FIG. 5 is a diagram for explaining the reading order of image data from the frame memory in the present embodiment.
The image data is normally written by sweeping in the line direction, that is, in the row direction, and is read out in the row direction at the time of reading. All the image data of one line is read and then adjacent. It is common to repeat the operation of reading all the image data of the line.
[0056]
On the other hand, the image processing apparatus according to the present invention is configured so that image data written by sweeping in the row direction is sequentially processed in the row direction image data in units of a certain length in the column direction. A small area (rectangular image data) obtained by sequentially inputting the next column to the image processing unit 7 and repeatedly to the right end of the image is referred to as a block line (BL).
[0057]
The configuration of the first data order conversion unit 6 and the second data order conversion unit 9 for enabling reading of such image data will be described with reference to FIG. FIG. 6 is a block diagram showing the configuration of the first and second data order conversion units.
[0058]
As shown in FIG. 6 (A), the first data order conversion unit 6 includes a plurality of memories that can store image data in block units, two in this case, and the two memories 6a and 6b write data. Writing and reading can be switched alternately by the switches on the side and reading side. That is, the frame memory 5 is connected to the memory 6a and the memory 6b by a switch on the writing side so as to be switchable, and the image processing unit 7 also has the memory 6a and the memory 6b in the reading side switch. When the frame memory 5 is connected to one of the memory 6a and the memory 6b, the other of the memory 6a and the memory 6b is switched to be connected to the image processing unit 7. . That is, the memories 6a and 6b are switched so as not to be simultaneously connected to both the frame memory 5 and the image processing unit 7, and are alternately written and read.
[0059]
A part of the frame image stored in the frame memory 5 is read in the line direction in units of blocks and stored in one memory, for example, the memory 6a in this case.
[0060]
In parallel with this, the block-unit image data that has already been read from the frame memory 5 and stored in the memory 6b is sequentially read in the column direction (vertical direction) and output to the image processing unit 7. Is done.
[0061]
When writing from the frame memory 5 to the memory 6a and reading from the memory 6b to the image processing unit 7 are finished, the switch on the writing side and the switch on the reading side are switched, and then the frame memory 5 transfers to the memory 6b. The writing of the block unit image data is started, and the reading of the block unit image data from the memory 6a to the image processing unit 7 is started.
[0062]
As shown in FIG. 6B, the second data order conversion unit 9 is configured in substantially the same manner as the first data order conversion unit 6 and operates in substantially the same manner.
[0063]
That is, the second data order conversion unit 9 includes a memory 9a and a memory 9b, a write side switch, and a read side switch.
[0064]
During the operation of the second data order conversion unit 9, writing from the distortion correction processing unit 8 is performed in the column direction (vertical direction) with respect to one of the memory 9a and the memory 9b, and the memory 9a and the memory 9b Reading from the other is performed in the row direction (horizontal direction) and output to the JPEG processing unit 10.
[Equation 1] of distortion correction coordinate conversion is an image on the entire screen to the last. Even if the corrected image is an image of (X ′, Y ′) as a result of the coordinate conversion, the image is displayed anywhere on the entire screen. It is calculating whether it is located.
[0065]
However, the image data input to the distortion correction processing unit 8 is in block line units. The distortion correction processing unit 8 needs to know the spatial positional relationship (position in the two-dimensional space) of the input image data. The control register 85 designates the position of the block line processed by the distortion correction processing unit 8.
[0066]
FIG. 7 shows the relationship between the captured image data before correction and the corrected image after correction. FIG. 7A shows photographed image data, and FIG. 7B shows a corrected image. FIGS. 7A and 7B are conceptual diagrams when attention is paid to a certain pixel position A of a certain block line BL of the corrected image (output side image). The coordinates (X, Y) of the position A in the corrected image (output image) are changed to the coordinates (X ′, Y ′) of the position A ′ in the (input image) using [Expression 1] of distortion correction coordinate conversion. Converted.
[0067]
The coordinate positions (X, Y) and (Xblst, Yblst), the coordinate positions (X ′, Y ′), and (X′blst, Y′blst) indicate positions from the origin of the entire screen. Here, the origin is the upper left corner of the entire screen.
[0068]
The image data input to the distortion correction block is in units of block lines indicated by reference numeral BL. For this reason, first, in order to perform the distortion coordinate conversion processing on, for example, the pixel A of the corrected image, it is necessary to convert it to the coordinate position (x ′, y ′) in the coordinate system in the block line BL so that final interpolation processing can be performed. is there.
[0069]
First, the coordinate position (X, Y) for each pixel of the corrected image (output image) in FIG.
[0070]
[Formula 2]
X = Xblst + m × ΔX
Y = Yblst + n × ΔY
ΔX and ΔY are interpolation pitches, and if it is 1.0 or more, a reduced image is generated, and if it is 1.0 or less, an enlarged image is generated. The coordinate position (Xblst, Yblst) corresponds to the position of the left shoulder of the output side block line of the distortion correction block, that is, the start position of the block line. When (m, n) = (0, 0), X = Xblst, Y = Yblst, and the coordinate position (Xblst, Yblst) can be calculated from [Equation 2]. Starting from m = 0, n = 0, for example, if m and n are alternately increased by one, the coordinate positions (X, Y) of all the pixels in the block line BL can be generated.
[0071]
When the coordinate positions of all the images in the BL are converted according to the expression 1, the BL is mapped to the deformed hatching area BL ″ in FIG. 7A. BL ′ in FIG. b) is a range of input data necessary to process BL, where (X′blst, Y′blst) is the upper left position of BL ′, and considering the pixels necessary for interpolation, etc. The BL ′ is set using the support function (distortion correction range calculation unit) described in detail in Japanese Patent Application No. 2003-202493 filed on the same day as the present application.
[0072]
The coordinate position (X, Y) of the output image after correction generated in [Expression 2] is coordinate-converted in [Expression 1] and the coordinate position (X ′, Y) of the input image before correction in FIG. Y ′). Then, by taking the difference between the coordinate position (X ′, Y ′) after the coordinate conversion and the coordinate position (X′blst, Y′blst), the left shoulder of the block line (thick dotted rectangle) is used as the origin. The coordinate position (x ′, y ′) can be obtained.
[0073]
As a result, based on the coordinate position (x ′, y ′) in the block line, interpolation processing in block line units can be performed.
[0074]
The block line BL ′ indicated by the thick dotted line on the input image side shown in FIG. 7A is set by the register 85. In order to generate the value, the input range (distortion) of the image data is taken into consideration for distortion deformation. A support function that enables calculation of the correction range is required. This support function may be provided by being added to the distortion correction processing function of the distortion correction processing unit 8.
[0075]
Next, the distortion correction processing flow will be described with reference to FIG. FIG. 8 is a flowchart of the distortion correction process of FIG.
[0076]
First, the block line processing where the hatching in FIG. 7B is performed is started (step S1). By setting (m, n) = (0,0) in [Expression 2] described above (step S2), the left shoulder of the hatched area in FIG. 7B, that is, the start coordinate position of the block line (Xblst, Yblst) ) Then, the coordinate position (X, Y) of the corrected image can be obtained from [Equation 2]. In this case, if m and n are increased by 1, respectively, the coordinate position (X, Y) of each pixel in the block line BL indicated by hatching in FIG. 7B can be generated (step S3).
[0077]
Next, the position (X, Y) of the coordinate system in the corrected image is converted into the position (X ′, Y ′) of the coordinate system in the photographed image before correction using [Equation 1] (step S4).
[0078]
Then, the coordinate system position (X ′, Y ′) in the captured image is converted to the coordinate system position (x ′, y ′) in the input block line BL by the method described above (step S5).
[0079]
Next, interpolation processing is performed based on the coordinate position (x ′, y ′) in the block line BL, and interpolation is performed from a plurality of surrounding pixel data by 16-point interpolation at the coordinate position (X ′, Y ′). By calculating the data, corrected image data for one pixel is obtained (step S6).
[0080]
Then, it is determined whether or not the above steps have been performed for all pixels in the coordinate system of the corrected image (step S7). If not completed, the next m and n are set (step S8). Steps S3 to S7 are repeatedly performed, and if the processing has been completed for all pixels in the determination in step S7, the process proceeds to step S9.
[0081]
In step S9, it is determined whether or not the distortion correction processing has been completed for all the block lines BL in the entire screen (frame). If not, the next input block line BL is selected, and steps S1 to S1 are selected. Step S9 is repeated, and if the process has been completed for all the screens in the determination in step S9, the process ends.
[0082]
9 and 10 should be referred to as a modification of FIG.
When performing digital zoom or cutting out only a small area of interest, only a part of the imaging data may be used. Memory can be saved by selectively capturing a part of the image data from the CCD. In this case, in order to perform distortion correction in units of block lines, information (X′imgst, Y′imgst) on where the captured data is located on the image sensor is necessary.
[0083]
FIG. 9 shows the case of digital zoom, in which only the central portion is captured from captured image data. The difference from the case of FIG. 8 is that, as described above, the preprocess circuit 2 sets where to take in the entire screen. Only necessary portions are taken in based on this setting, and the range is indicated by E ′ in FIG.
[0084]
First, in FIG. 9B, a start position (Ximgst, Yimgst) corresponding to the left shoulder of the digital zoom range E in the entire screen is set. Next, a block line BL is set. Here, it is a portion indicated by satin. At this time, the position coordinates (Xblst, Yblst) are determined with the left shoulder of the capture range E as the origin, and the coordinates of each pixel in the block line BL are changed by changing m and n as in [Formula 2]. Can be generated.
[0085]
As a result, the position (X, Y) in the coordinates of the entire screen is determined from the coordinate positions (Ximgst, Yimgst), (Xblst, Yblst), and (m × ΔX, n × ΔY).
[0086]
The coordinate position (X ′, Y ′) in the captured image can be obtained using [Equation 1] from the coordinate position (X, Y) determined as described above.
[0087]
Then, the coordinate position (x ′, y ′) in the block line BL ′ in the captured image data is obtained.
[0088]
First, the left shoulder coordinates (X′imgst, Y′imgst) are known from the set range E ′ on the imaging surface of the captured imaging data. The coordinate position (X'blst, Y'blst) of the left shoulder of the block line BL 'is determined in advance using the support function in the coordinates with the left shoulder of the capture range E' as the origin, and within the block line BL ' The coordinate position (x ′, y ′) in the coordinates is determined. As a result, interpolation processing for each block line is performed.
[0089]
FIG. 10 shows a case where a certain area, for example, the edge of the screen is cut out from the image data. This corresponds to a case where an end portion of the imaged data is captured by a monitoring camera or the like. In the case of FIG. 10 as well, the same reference numerals as those in FIG. 9 are given. The operation is the same as that in FIG. 9 except that the cutting position is different from that in FIG.
[0090]
In a normal digital camera, the CPU sets the area to be cut out in the control register, so the cutout position is known and the above conversion is possible. However, when distortion correction processing is performed on the recorded data, it is cut out to a tag or the like. It is necessary to record the position and size of the area at the same time. At this time, the correction coefficient is recorded at the same time.
[0091]
By the way, in many cases, a monitor mode (a mode in which the entire captured image is not read out) is prepared for the CCD as the image sensor. In the monitor mode, lines are often thinned out, and the two-dimensional space in the image sensor is the same, but the size may be different on the data. In the preprocess, pixels in the horizontal direction may be thinned out. Further, filtering data such as a low-pass filter (LPF) may be applied to the image data from the CCD, or interpolation processing may be performed to capture the data into the frame memory. In this case, the vertical / horizontal sampling ratio may not be an integer. Further, in the recording of YCbCr in JPEG, the horizontal direction of Cb and Cr components is often sampled at double intervals, and the horizontal direction is reduced as data. These are corrected by Sx and Sy in [Formula 1].
[0092]
FIG. 11A shows a state in which data actually sent from the CCD is only indicated by black circles, and three lines are thinned out in the vertical direction during data capture at the time of imaging. In the monitor mode, only the thinned data is output from the CCD as described above, and when this data is taken into the memory, the image data is crushed vertically by thinning as shown in FIG. 11B. As shown in FIG. 11C, even if data at the same distance from the center is extracted, a phenomenon that an image is output shorter in the horizontal direction as shown in FIG. 11B appears. As a result, in the distortion correction according to [Equation 1], the effect of the distortion correction increases as the distance from the center increases. Therefore, the effect of the distortion correction changes vertically and horizontally.
[0093]
Correction coefficients Sx and Sy are provided for the purpose of correcting the image asymmetry in the vertical and horizontal directions. In the case of FIG. 11B, since it is necessary to extend in the vertical direction, Sx = 1 and Sy = 4.
[0094]
Due to the effects of Sx and Sy, accurate distortion correction is possible even if the vertical and horizontal sampling ratios are different.
[0095]
As described above, when the sampling ratio is not an integer, Sx and Sy may be set as non-integers. Furthermore, it is possible even when the intervals between the pixels of the image sensor differ vertically and horizontally (for example, rectangular pixels).
[0096]
By the way, normally, in the case of JPEG, it is generally recorded as data of YCbCr. As a setting at that time, a case as shown in FIG. 12 is conceivable. For example, in the case of data in the format of YC422, the resolution for color data is ½ in the horizontal direction compared to luminance data. In this case, color data may be corrected by setting Sx = 2 and Sy = 1.
[0097]
By the way, when the center of distortion and the center of the image do not coincide with each other, the subject at the center before the correction is shifted from the center after the correction. In [Formula 1], it is possible to prevent the subject at the center of the image from changing before and after the correction by correcting with the center deviation correction value (Xoff, Yoff).
[0098]
FIG. 13 is a diagram for explaining the center misalignment of an image.
FIG. 13 shows a corrected image. In FIG. 3B described above, the center of distortion coincides with the center of the image, that is, the center of the CCD (center of the image) coincides with the lens optical axis (center of distortion). .
[0099]
FIG. 13 shows a case where the center of distortion is deviated from the center of the CCD. In FIG. 13, since the distortion center (Xd, Yd) is different from the center of the image, the center P of the image with respect to the output range H is indicated by a Q point (large black circle) when distortion correction is performed without taking the center deviation into consideration. ). For the user, when the center of the image actually captured and the image displayed after distortion correction is shifted, the user feels very uncomfortable. Therefore, in order to bring the data originally in the center of the output range H before distortion correction to the center position with respect to the output range, the method of cutting out the image data subjected to distortion correction is changed. In order to do this, it is only necessary to define the center deviation of the image by (Xoff, Yoff) and change the cut-out direction of the image data. By setting the center deviation (Xoff, Yoff) of the image, the output range is moved from H to H ′ so that the subject that was originally at the center P of the image is also centered in the corrected output range H ′. It can be corrected.
[0100]
According to the embodiment of FIG. 13, it is possible to eliminate the uncomfortable feeling caused by the shift of the center subject.
[0101]
【The invention's effect】
As described above, according to the present invention, it is possible to calculate the spatial position of a small area (for example, a block line), and to realize distortion correction processing in units of small areas. In addition, accurate distortion correction can be performed on the thinned and captured image data, and accurate distortion correction can be performed on data obtained by cutting out an arbitrary area such as digital zoom.
[Brief description of the drawings]
FIG. 1 is a block diagram showing the overall configuration of an image processing apparatus according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a distortion correction processing unit in FIG. 1;
FIG. 3 is a conceptual diagram of coordinate conversion in a distortion correction processing unit.
FIG. 4 is a view showing a captured image in which a Jinkasa type distortion is generated.
FIG. 5 is a view for explaining the reading order of image data from a frame memory according to the present embodiment.
FIG. 6 is a block diagram showing a configuration of first and second data order conversion units.
FIG. 7 is a diagram illustrating a relationship between captured image data before correction and a corrected image after correction.
8 is a flowchart of distortion correction processing in FIG.
FIG. 9 is a diagram illustrating a relationship between captured image data before correction and a corrected image after correction in the case of digital zoom.
FIG. 10 is a diagram illustrating a relationship between captured image data before correction and a corrected image after correction when a certain region is cut out from the captured data.
FIG. 11 is a diagram for explaining line thinning in a monitor mode of a CCD.
FIG. 12 is a diagram for explaining thinning in the horizontal direction in YC422 format data;
FIG. 13 is a diagram for explaining a center shift of an image.
FIG. 14 is a diagram showing a grid-like subject, a captured image with barrel distortion, and a captured image with pincushion distortion.
FIG. 15 is a diagram illustrating a concept of a general digital camera image processing procedure and a block configuration of an image processing apparatus of a conventional digital camera.
FIG. 16 is a diagram showing a relationship between a CCD pixel and an image.
[Explanation of symbols]
1 ... CCD
3 ... Bus
4 ... CPU
5 ... Frame memory
6 ... 1st data order conversion part
7. Image process circuit
6 ... 1st data order conversion part
8: Distortion correction processing unit (distortion correction means)
9: Second data order conversion unit
10 ... JPEG processing unit
11 ... Memory card (recording medium)
82 ... Internal memory section (buffer)
83. Memory control unit
85: Control register

Claims (12)

In an image processing apparatus having distortion correction means for performing distortion correction processing on image data,
The distortion correction means includes
Image processing characterized by calculating a spatial position on the imaging surface in a coordinate system that can be described when calculating a position corresponding to each pixel of the corrected image in the image data before correction according to a predetermined correction formula apparatus. The distortion correction means includes
Processing is performed in units of the first small area included in the corrected image,
For the second small region on the imaging surface including the small region obtained by converting the coordinate position of each pixel of the first small region according to the correction formula, the position in the coordinate system corresponding to the imaging surface is set to the second After converting to coordinates in a small area,
The image processing apparatus according to claim 1, wherein data of each pixel of the corrected image is generated. The image data is
The image processing apparatus according to claim 1, wherein the image processing apparatus is partial image data in which only a part of the imaging data is captured. The partial image data is
The image processing apparatus according to claim 3, wherein the image processing apparatus is data obtained by cutting out a part of the imaging data. The partial image data is
The image processing apparatus according to claim 3, wherein the image processing apparatus is data thinned out in at least one horizontal or vertical direction of the imaging data. The image data is
The image processing apparatus according to claim 1, wherein the image processing apparatus is data generated by performing at least one of filtering, spatial sampling, and interpolation processing on the imaging data. The image data is
The image processing apparatus according to claim 1, wherein an interval in which each pixel is spatially positioned on the imaging surface is different in the vertical direction and the horizontal direction. When the spatial sampling interval of each pixel in image data differs vertically and horizontally, a correction formula for obtaining coordinates in an image before distortion correction includes a coefficient for correcting a difference in sampling interval between the portrait and landscape. Item 8. The image processing apparatus according to any one of Items 1 to 7. The correction formula for obtaining coordinates in the image before distortion correction is as follows:
The image processing apparatus according to claim 1, wherein a predetermined amount of offset correction is possible. The offset correction is
The image processing apparatus according to claim 9, which is performed when the center of distortion and the center of the image do not coincide with each other. A distortion correction method for an image processing apparatus having distortion correction means for performing distortion correction processing on image data,
The distortion correction method includes:
Generating a first position in a coordinate system corresponding to each pixel of the corrected image;
Converting the first position to a second position in a coordinate system in image data before distortion correction according to a distortion correction equation;
Converting the second position in the coordinate system in the image data before the distortion correction into coordinates in a setting region with reference to the coordinate system corresponding to the imaging surface;
Performing an interpolation process based on the coordinates in the setting region, and generating data of each pixel of the corrected image;
A distortion correction method comprising: 12. The distortion correction method according to claim 11, wherein the process is performed in units of small areas included in the corrected image.

Images