JP4213327B2 - Method and apparatus for estimating light source direction and three-dimensional shape, and recording medium - Google Patents

Description

但し、Ｎ：プロット数
ｘｎ：余弦の値
ｙｎ：反射光の強度値
として、ａ，ｂの値を求めることができる。そしてこのようにして求められたａ，ｂの値から、下記の式（７）に示す評価関数ｃｆを算出する。
【００５３】
【数２】

そしてこの評価関数ｃｆの値が最も小さい値となった仮想光源の方向を撮影時における仮の光源方向とすればよい。
【００５４】
次に、顔の３次元形状の推定について説明する。図７は仮の光源方向を決定するために用いたプロットおよびこのプロットから求められた直線Ｃを示す図である。図７に示すように、位置合わせ後の３次元形状データＳ１′は、顔の近似的な３次元形状を表すものであることから、実際の顔の形状は３次元形状データＳ１′により表される形状とはずれているため、プロットは完全に直線Ｃ上に位置するものではなく、直線Ｃから隔たって存在するものである。ここで、本実施形態においては、３次元形状データＳ１′はｘ，ｙ方向については位置合わせがなされていることから、ｚ軸方向の値が実際の値からずれているためにこの隔たりが生じるものであると考えることができる。したがって、本実施形態は、各プロットの直線Ｃからの隔たりを修正することにより３次元形状データＳ１′のｚ軸方向のずれを修正して顔の３次元形状を推定するものである。
【００５５】
まず、上述したように求めた仮の光源方向を求めたプロットの直線Ｃからのずれは、そのプロットに対応する測定点における法線ベクトルの傾きのずれから生じるものであると考えることができる。例えば、図７における測定点Ｐ３に対応するプロットの場合、プロットが直線Ｃから左側にずれているため、測定点Ｐ３における法線ベクトルの傾きがずれて余弦の値が実際の余弦の値よりも小さくなっていると考えることができる。ここで、実際の余弦の値とは、仮の光源から実際の顔の３次元形状上に設定される測定点へ向かう方向ベクトルと、この測定点における法線ベクトルとがなす余弦のことである。そこで、余弦修正手段１５において、測定点Ｐ３における余弦の値が大きくなるように修正して法線ベクトルの傾きを修正することにより、測定点Ｐ３における３次元形状データＳ１′のｚ値を修正してプロットを直線Ｃに近づく方向に移動させる。すなわち、測定点Ｐ３に対応する２次元画像データＳ０のデータ値をｇ（Ｐ３）とすると、ｇ（Ｐ３）＝ａｘ＋ｂより、
ｘ＝ｃｏｓθｃ＝（ｇ（Ｐ３）−ｂ）／ａ （８）
但し、ｃｏｓθｃ：測定点Ｐ３が直線Ｃ上に位置する場合の余弦の値
そこで、測定点Ｐ３におけるθの値、すなわち方向ベクトルと法線ベクトルとがなす角度をθｃの値に近づけるように修正する。なお、余弦の値をＹとした場合、θの値は、
θ＝ｃｏｓ^−１Ｙ （９）
により求めることができる。なお、図７における測定点Ｐ４に対応するプロットのように、プロットが直線Ｃから右側にずれている場合は、測定点Ｐ４における余弦の値が小さくなるように測定点Ｐ４における法線ベクトルの傾きを修正することにより、３次元形状データのｚ値を修正してプロットを直線Ｃに近づく方向に移動させる。
【００５６】
図８は、θの値の修正を説明するための図である。図８において、θoldは修正前の測定点Ｐ３における方向ベクトルＰＬと法線ベクトルｎoldとがなす角度、θnewは修正後の測定点Ｐ３における方向ベクトルＰＬと法線ベクトルｎnewとがなす角度、θｃは測定点Ｐ３が直線Ｃ上に位置する場合の方向ベクトルＰＬと法線ベクトルとがなす角度である。まず、ベクトルｎnewを下記の式（１０）により算出する。
【００５７】
ｎnew＝ｎold＋α（θold−θｃ）・（ＰＬ−ｎold） （１０）
但し、α：係数（０＜α＜１）
ここで、αの値は１より小さい値を有するため、法線ベクトルｎnewはαの値に応じて法線ベクトルｎoldの終点から光源Ｌへ向かうベクトル（ＰＬ−ｎold）上を移動することとなる。ここで、αの値を１としないのは、直線Ｃはあくまでも仮の光源方向を表すため、実際の光源方向とは厳密には異なるものであり、このように実際の光源方向とは異なる仮の光源方向に基づいて測定点を一気に修正すると、顔の３次元形状を正確に求めることができなくなるからである。
【００５８】
このようにして法線ベクトルｎnewが算出されると、この法線ベクトルｎnewに基づいて、測定点修正手段１６において、測定点Ｐ３における３次元形状データＳ１′のｚ値が修正される。具体的には測定点Ｐ３が図９に示すような点Ａ，Ｂ，Ｃを頂点とするポリゴンに存在する場合に、点Ａ，Ｂ，Ｃの頂点のｚの値を修正して、３次元形状データＳ１′のｚ値を修正する。ここで、法線ベクトルｎnewを（ａ′，ｂ′，ｃ′）、測定点Ｐ３の座標を（ｘ１，ｙ１，ｚ１）とすると、測定点Ｐ３を通り法線ベクトルがｎnewである平面の方程式が定まる。
【００５９】
ａ′ｘ＋ｂ′ｙ＋ｃ′ｚ＋ｄ′＝０ （１１）
ｄ′＝−（ａ′ｘ１＋ｂ′ｙ１＋ｃ′ｚ１） （１２）
点Ａ，Ｂ，Ｃにおいてはｘ、ｙの値は変更されておらず、ｚの値のみが変更されるため、変更されたｚの値をｚnewとすると、
ｚnew＝−（ａ′ｘ＋ｂ′ｙ＋ｄ）／ｃ′ （１３）
により求めることができる。これにより、修正後の点Ａ，Ｂ，Ｃの座標は、（ｘ，ｙ，ｚnew）と表すことができる。
【００６０】
以上の処理を仮の光源方向を決定したプロットに対応する全ての測定点について行い、新たな測定点の座標値を得、これを修正３次元データＳｃとする。なお、修正３次元データＳｃ上における測定点を修正測定点とする。
【００６１】
そして、修正測定点を新たな測定点とし、新たな仮想光源を設定して、新たな仮想光源から修正測定点へ向かう方向ベクトルおよび修正測定点の法線ベクトルｎnewがなす余弦ｃｏｓθnewと、修正測定点における反射光の強度値との関係をプロットし、さらに上記式（７）および（８）により新たな直線を求めて新たな仮想光源の方向を決定するとともに、評価関数ｃｆを算出する。この際、新たな仮想光源は、仮の光源の近傍に設定すればよい。そして、評価関数ｃｆの値が予め定められた所定値よりも大きいか否かが判断され、所定値を越える場合には、上述したように測定点のｚ値を修正して新たな修正測定点を求め、この新たな修正測定点についての仮の光源方向および評価関数ｃｆの値の算出を、評価関数ｃｆの値が上記所定値よりも小さい値となるまで繰り返す。そして、評価関数ｃｆが所定値以下となったときの仮想光源の方向を光源の方向と推定し、この際の修正測定点により定められる３次元形状を顔の３次元形状と推定するものである。なお、ここで、推定される顔の３次元形状は顔の肌色部分の３次元形状となる。以上の処理は、制御手段１７により、測定点設定手段１１、余弦算出手段１２、プロット手段１３、仮の光源方向推定手段１４、余弦修正手段１５、および測定点修正手段１６を制御することにより行われる。
【００６２】
次いで、本実施形態の動作について説明する。図１０は、本実施形態の動作を示すフローチャートである。まず、位置合わせ手段１において２次元画像データＳ０と３次元形状データＳ１との位置合わせが行われる（ステップＳ１）。そして位置合わせされた合成データＳ２が推定手段３に入力される。推定手段３においては、位置合わせ後の３次元形状データＳ１′により表される３次元形状上に複数の測定点を設定するとともに（ステップＳ２）、仮想光源Ｌ１〜Ｌ５から任意の仮想光源を選択する（ステップＳ３）。本実施形態においては、仮想光源Ｌ１から順に選択するものとする。そして、各測定点において、法線ベクトルと測定点および仮想光源間の方向ベクトルとのなす余弦が算出され（ステップＳ４）、各測定点における反射光の強度値と上述した余弦との関係が図６に示すようにプロットされる（ステップＳ５）。そして、反射光の強度値と余弦との関係において、評価関数ｃｆが予め定められた値以下となったか否かを判断することにより、プロットが直線上に位置するか否かが判断され（ステップＳ６）、ステップＳ６が否定された場合にはステップＳ３に戻り、異なる仮想光源を選択してステップＳ４からステップＳ６の処理を繰り返す。ステップＳ６が肯定された場合には、この仮想光源の方向を撮影時における仮の光源方向として決定する（ステップＳ７）。
【００６３】
このように仮の光源方向が決定されると、この仮の光源方向の直線を決定した際の評価関数ｃｆが所定値以下であるか否かが判断される（ステップＳ８）。ステップＳ８が否定されると、上述したように測定点の位置が修正され（ステップＳ９）、さらに仮の光源方向の近傍に新たな仮想光源を設定して（ステップＳ１０）、ステップＳ３に戻り、以下ステップＳ３からステップＳ８までの処理を繰り返す。そして、ステップＳ８が肯定されると、仮の光源方向を撮影時の光源の方向として決定するとともに、修正された測定点の座標位置から顔の３次元形状を決定し（ステップＳ１１）、処理を終了する。
【００６４】
このように、本実施形態においては、仮想光源の位置が正しい、すなわち撮影時の光源位置と一致する場合には、式（５）に示すように、顔の表面における反射光の強度値の比と上記余弦の比とが一致するという特徴に基づいて、撮影時の光源の方向を推定するようにしたため、測定点の設定については遮断ｓの値が０でないという条件のみを満たせばよく、また比較的簡易な演算により光源の方向を推定することができる。また、物体の３次元形状が未知であっても、推定された光源の方向に基づいて、上記池内の方法と比較して簡易に物体の３次元形状をも推定することができる。そしてこれにより、実画像とＣＧのような仮想画像とを合成する場合に、実画像において推定した光源の方向に基づいて、仮想画像が適切な光源から照明がなされているように修正した後に合成をすることができ、違和感のない合成画像を得ることができる。また、ＭＰＥＧ−４のように、オブジェクト毎に画像を転送して動画を見る段階において合成する符号化方式において、オブジェクトに実画像と仮想画像が含まれている場合に、実画像の照明方向に応じて仮想画像の照明方向を修正して合成することができ、これにより違和感のない合成動画像を得ることができる。さらに、実画像に含まれる物体を適切な３次元画像として表すことができるため、実画像から違和感のない３次元画像を生成することができる。さらにまた、太陽光下において撮影を行うことにより得られた画像の場合、画像中の太陽光の方向を推定することにより、画像に対する逆光補正や濃度補正を容易に行うことができる。
【００６５】
なお、上記実施形態においては、説明のために２次元状に仮想光源Ｌ１〜Ｌ５を配置しているが、実際には３次元形状データＳ１に対して３次元状に複数の仮想光源を配置し、各測定点毎に反射光の強度と上記余弦との関係をプロットして撮影時の光源方向および物体の３次元形状を推定するものである。
【００６６】
また、上記実施形態においては、式（６）および（７）に示すように、最小二乗法による評価関数ｃｆを用いて仮の光源方向を推定しているが、これに限定されるものではなく、図６に示すプロットをモニタに表示し、オペレータがこれを観察してプロットが直線上に位置する仮想光源の方向を仮の光源の方向として求めるようにしてもよい。
【００６７】
さらに、２次元画像データＳ０を取得する際において使用するカメラは、入力する光と出力する画像データとの関係が非線形となるものである。このため、予め使用するカメラによりグレーチャートを撮影して、カメラに対する入力と出力との関係を求めておき、この関係が直線となるようにカメラの入出力特性を補正しておくことが好ましい。
【００６８】
なお、実環境における物体表面の輝度は、上述した拡散反射光および環境光に加えて鏡面反射光も含まれ、これらの光の組み合わせによりモデル化することができるものである。すなわち、物体表面の輝度は下記の式（１４）により表すことができる。なお、ここでは相互反射は拡散反射、鏡面反射および環境光と比べて小さく、無視できるものとする。
【００６９】
Ｉ＝Ｉｄ＋Ｉｓ＋Ｉｅ （１４）
但し、Ｉ：物体表面の輝度
Ｉｄ：拡散反射光
Ｉｓ：鏡面反射光
Ｉｅ：環境光
ここで、上記実施形態と同様に光源が１つである場合を想定し、３次元形状データＳ１′を用いて、顔画像から鏡面反射位置を検出する場合について説明する。
【００７０】
上記式（１４）より、顔表面上の任意の２点における強度値すなわちデータ値の差ΔＩは、拡散反射光Ｉｄおよび鏡面反射光Ｉｓに依存する。まず、拡散反射光Ｉｄのみが存在すると仮定して考えると、表面反射率が一定の箇所において上記図５に示すような２つの測定点Ｐ１，Ｐ２における強度差ΔＩは下記の式（１５）に示すように、定式化することができる。
【００７１】
ΔI=k・s1・cosθ1・Lc-k・s2・cosθ2・Lc=k(s1・cos1-s2・cos2)Lc（１５）
光源の位置および強度は変化しないと仮定できるため、強度差ΔＩは２点における余弦の差すなわち法線ベクトルの方向差および遮断のいずれかにより生じるものとなる。ここで、基準となる測定点（例えばＰ１とする）が拡散反射する位置に存在するものとすると、強度差ΔＩが遮断により生じる場合は、他方の測定点Ｐ２は影となっているため、２点の位置がわずかに違っていても強度値は大きく低下するように変化する。一方、強度差ΔＩが余弦の差により生じる場合は、他方の測定点Ｐ２も拡散反射する位置にあると強度差ΔＩは略余弦の値に比例するものとなるが、他方の測定点Ｐ２が鏡面反射する位置にあると２点の位置がわずかに違っていても強度値は大きく増加するように変化する。このように、位置合わせ後の３次元形状データＳ１′上の拡散反射する位置に１つの測定点Ｐ１を設定し、この測定点Ｐ１のデータ値とこれに近接する任意の測定点Ｐ２（拡散反射する位置に限定されない）におけるデータ値との差を求め、２点の位置の差がわずかであるにもかかわらず、データ値が急激に大きくなる点を鏡面反射する位置として求めることができる。
【００７２】
一方、鏡面反射の式は下記の式（１６）に示すように定式化することができる。
【００７３】
Ｉｓ＝ｋ・ｓ・ｃｏｓ^ｎα・Ｌｃ （１６）
すなわち、図１１に示すような位置に光源があるとすると、物体を撮影した撮影手段が矢印Ｒの位置、すなわち光源Ｌから出射した光の点Ｐにおける正反射方向にある場合に、点Ｐは鏡面反射することとなり、カメラの位置がＲから角度αずれた位置Ｖに移動するほど鏡面反射光の強度は小さくなる。
【００７４】
したがって、図１２に示すように、測定点設定手段１１において設定された複数の測定点に鏡面反射する測定点が存在するか否かを判断し、存在する場合に鏡面反射する測定点を選択する選択手段２０を設け、仮の光源の方向を推測する前（図１０に示すフローチャートのステップＳ３の前）に、複数の測定点に鏡面反射する測定点が存在するか否かを判断し、鏡面反射する測定点が存在する場合には、図１１に示すように、その位置（Ｐとする）における法線ベクトルが、方向ベクトルＰＬと、方向ベクトルＰＲとがなす角度を２等分するものとなることから、光源が存在する方向をある程度推測することができる。したがって、上記実施形態のようにあらゆる方向に仮想光源を設定しなくとも、光源が存在しそうな方向にのみ仮想光源を設定すれば、仮の光源方向を推定することができるため、演算量を低減して処理を高速に行うことができる。
【００７５】
なお、上記実施形態においては、人物の顔の形状を推定しているが、これに限定されるものではなく、種々の物体の３次元形状を推定することができるものである。
【図面の簡単な説明】
【図１】本発明の実施形態による推定装置の構成を示す概略ブロック図
【図２】２次元画像データと３次元形状データとの位置合わせを説明するための図
【図３】推定手段の構成を示す概略ブロック図
【図４】仮想光源の方向を説明するための図
【図５】測定点における余弦の算出を説明するための図
【図６】余弦と反射光の強度値との関係を示すグラフ
【図７】仮の光源方向を決定するために用いるプロットを示す図
【図８】θの修正を説明するための図
【図９】ｚ値の修正を説明するための図
【図１０】本実施形態の動作を示すフローチャート
【図１１】鏡面反射を説明するための図
【図１２】選択手段を設けた実施形態の構成を示す概略ブロック図
【符号の説明】
１ 位置合わせ手段
２ メモリ
３ 推定手段
１１ 測定点設定手段
１２ 余弦算出手段
１３ プロット手段
１４ 仮の光源方向推定手段
１５ 余弦修正手段
１６ 測定点修正手段
１７ 制御手段
２０ 選択手段[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an estimation method and apparatus for estimating the direction of a light source and the three-dimensional shape of an object at the time of photographing from an image obtained by photographing the object using a single light source. The present invention relates to a computer-readable recording medium on which a program to be executed is recorded.
[0002]
[Prior art]
Technology for synthesizing real images and virtual images such as computer graphics (CG) without any sense of incongruity is called mixed reality technology and is expected to play an important role in various fields such as communication, transportation, and entertainment. Has been. Thus, in order to synthesize a real image and a virtual image without a sense of incongruity, it is necessary to synthesize them with geometric, optical and temporal consistency. In particular, optical consistency is extremely important because the direction and intensity of the light source in the actual image and the direction and intensity of the light source in the virtual image do not match, so that the composite image becomes very uncomfortable. For this reason, a method for estimating the intensity distribution of the light source in the image has been proposed (Sato et al., Estimation of actual lighting environment based on soft shadows of real objects, computer vision and image media, 110-3, 1998.3. 19, PP17-24).
[0003]
This method estimates the ambient light intensity distribution based on the soft shadow of a real object, and uses the shadow of an object with a known three-dimensional shape in the image to estimate the ambient light intensity distribution. Is going. Specifically, the relationship between the brightness of the virtual light source and the data value at a certain measurement point on the object is represented by a determinant, and the inverse distribution of the matrix in this determinant is obtained to obtain the intensity distribution of the light source. To be estimated. Then, if the intensity distribution of the light source in the image is estimated by such a method, the intensity distribution of the light source in the virtual image can be matched with the intensity distribution of the light source in the actual image. Can be synthesized.
[0004]
In addition, a method of generating an object image illuminated from an arbitrary light source direction using an image paste method has been proposed (Katsufumi Ikeuchi, automatic generation of a virtual reality model: optical aspects, scientific techniques, PRMU98- 122 (1998-11), PP75-82). This method is a method of creating an image illuminated from an arbitrary light source direction as a linear sum of a plurality of sample images obtained by photographing an object by changing the light source in a plurality of directions.
[0005]
[Problems to be solved by the invention]
However, in the method of Sato et al., It is necessary to set the position of the measurement point to a mathematically independent position where the inverse matrix can be calculated, and it is necessary to obtain the position of such a measurement point by manual operation. The operation is very troublesome. In addition, the three-dimensional shape of the object needs to be known. Furthermore, the method in the pond is not a realistic method because it requires a very long time for the calculation because it is necessary to create an image that has been captured in advance from 80 light source directions for the target object.
[0006]
The present invention has been made in view of the above circumstances, and an estimation method and apparatus capable of easily estimating the direction of a light source and the three-dimensional shape of an object, and a computer-readable recording of a program for causing a computer to execute the estimation method The object is to provide a possible recording medium.
[0007]
[Means for Solving the Problems]
The method of estimating the light source direction and the three-dimensional shape of the object according to the present invention is obtained by photographing an object having a three-dimensional shape including a portion having a substantially constant surface reflectance under a substantially single light source. An estimation method for estimating a direction of the light source and a three-dimensional shape of the object based on an image,
The data value of object image data representing the image of the object on the image, and the three-dimensional after the alignment between the object image data and three-dimensional shape data representing a general three-dimensional shape of the object The direction of the light source and the three-dimensional shape of the object are estimated based on the positional relationship between the approximate object represented by the shape data and a virtual light source set in any of a plurality of directions. .
[0008]
Here, “shooting with a substantially single light source” means that the presence of other light sources can be ignored as in the case of shooting under sunlight in fine weather, and only a single light source can be ignored. Shooting in a situation that is almost equivalent to using
[0009]
For example, when the object is a human face, the shape of the face varies depending on the person, but there is a shape that roughly matches any face shape. Therefore, the three-dimensional shape data representing a three-dimensional shape that roughly matches the shape of such an object is referred to as “three-dimensional shape data representing a general three-dimensional shape”.
[0010]
The skin color of the person's face can be regarded as having a substantially constant surface reflectance.
[0011]
The alignment of the object image data and the three-dimensional shape data matches the direction of the object in the image, the direction of the three-dimensional shape of the object, the center position, the size in the two-dimensional direction, that is, the direction in which the object image data exists. As described above, this means matching three-dimensional shape data with object image data. By such matching, the position of the three-dimensional shape data is aligned with the object image data only in the two-dimensional direction. In the present invention, the object represented by the three-dimensional shape data thus aligned is used. The shape is an approximate object.
[0012]
In the estimation method according to the present invention, a plurality of measurement points are set in a portion corresponding to a portion where the surface reflectance of the object on the approximate object is substantially constant,
Calculating a cosine formed by a direction vector from the virtual light source set in the plurality of arbitrary directions to the plurality of measurement points and a normal vector at the plurality of measurement points;
The relationship between the cosine and the data value of the object image data corresponding to the plurality of measurement points is plotted for each direction of the plurality of virtual light sources,
Estimating the direction of the virtual light source in which the plot is substantially a straight line as the temporary light source direction,
Modify the cosine in the plot so that the plot for the temporary light source direction is located on the straight line,
Correcting the position of the measurement point so that a cosine formed by the direction vector and the normal vector at the measurement point approaches the corrected cosine, and obtaining a corrected measurement point;
The calculation of the cosine using the corrected measurement point as the new measurement point, the plot, the estimation of the temporary light source direction, the correction of the cosine and the acquisition of the corrected measurement point are repeated a predetermined number of times,
Estimating the temporary light source direction obtained as a result of the iterative process as the direction of the light source, and estimating the shape determined by the position of the corrected measurement point obtained as a result of the iterative process as the three-dimensional shape of the object preferable.
[0013]
Here, the “direction vector” can be calculated from the coordinate position of the virtual light source and the coordinate position of the measurement point, and the “normal vector” is the data of the coordinate position of the measurement point on the three-dimensional shape data after alignment. It can be determined based on the value. Further, the cosine can be obtained from the inner product of the direction vector and the normal vector.
[0014]
“Correct the cosine in the plot so that the plot is on a straight line” means to change the cosine value at the plotted measurement point and move the plot so that the plot is on the straight line. Say.
[0015]
In addition, “correct the position of the measurement point so that it approaches the corrected cosine” means that the cosine formed by the direction vector and the normal vector at the measurement point before correction is completely matched with the corrected cosine. Rather than exceeding the corrected cosine value, the cosine formed by the direction vector and the normal vector at the corrected measurement point is calculated by correcting the position of the measurement point so that it is close to this value. This means that the value is close to the corrected cosine. Here, “correcting the position of the measurement point” means that the three-dimensional shape data is aligned in the two-dimensional direction in the object image data, so the position is not corrected in the two-dimensional direction. This refers to correcting the position in the depth direction with respect to image data. Specifically, when the plane in which the object image data exists is an xy plane, this means that the position of the measurement point is corrected only in the z direction.
[0016]
In the present invention, the three-dimensional shape is estimated only in a portion where the surface reflectance of the object is substantially constant, but the entire surface reflectance of the object having the three-dimensional shape is substantially constant. In this case, the entire three-dimensional shape of the object can be estimated.
[0017]
Further, in the estimation method according to the present invention, based on the data value of the object image data at the plurality of measurement points, it is determined whether there are specular reflection measurement points that are specularly reflected at the plurality of measurement points,
When the specular reflection measurement point exists, based on the specular reflection measurement point, select the direction of the virtual light source that is highly likely to be the direction of the light source,
The cosine is preferably calculated using a virtual light source in the selected direction.
[0018]
Further, in the estimation method according to the present invention, it is preferable that the direction of the light source is estimated by determining whether or not the plot is substantially a straight line based on an evaluation function using a least square method.
[0019]
In this case, the predetermined number of times is preferably the number of times until the evaluation function becomes a predetermined value or less.
[0020]
In the estimation method according to the present invention, it is preferable that the intensity of the light source is estimated based on a slope of a straight line set by a plot used for estimating the direction of the light source.
[0021]
Furthermore, in the estimation method according to the present invention, it is preferable that the object is a human face.
[0022]
Furthermore, it is preferable that the three-dimensional shape data is polygon data representing the three-dimensional shape of the object.
[0023]
The estimation apparatus according to the present invention is based on an image obtained by photographing an object having a three-dimensional shape including a portion having a substantially constant surface reflectance under a substantially single light source, and the direction of the light source. An estimation device for estimating a three-dimensional shape of the object,
The data value of object image data representing the image of the object on the image, and the three-dimensional after the alignment between the object image data and three-dimensional shape data representing a general three-dimensional shape of the object Characterized by comprising means for estimating the direction of the light source and the three-dimensional shape of the object based on the positional relationship between the approximate object represented by the shape data and a virtual light source set in any of a plurality of directions. To do.
[0024]
In the estimation apparatus according to the present invention, the means for estimating the direction of the light source and the three-dimensional shape of the object corresponds to a portion on the approximate object where the surface reflectance of the object is substantially constant. A cosine formed by a measurement point setting means for setting a plurality of measurement points, a direction vector from the virtual light source set in any of a plurality of directions to the plurality of measurement points, and a normal vector at the plurality of measurement points Cosine calculating means for calculating;
Plotting means for plotting the relationship between the cosine and the data value of the object image data corresponding to the plurality of measurement points for each direction of the plurality of virtual light sources;
A temporary light source direction estimating means for estimating a virtual light source direction in which the plot is substantially a straight line as a temporary light source direction;
Cosine correction means for correcting the cosine in the plot so that the plot for obtaining the temporary light source direction is positioned on the straight line;
Measurement point correcting means for correcting a position of the measurement point so as to obtain a corrected measurement point so that a cosine formed by the direction vector and the normal vector at the measurement point approaches the corrected cosine;
The measurement point setting means for repeating the calculation of the cosine using the corrected measurement point as the new measurement point, the plot, the estimation of the temporary light source direction, the correction of the cosine and the acquisition of the corrected measurement point a predetermined number of times. Control means for controlling the cosine calculating means, the plotting means, the temporary light source direction estimating means, the cosine correcting means, and the measuring point correcting means;
Estimating means for estimating the temporary light source direction obtained as a result of the iterative process as the direction of the light source, and estimating the shape defined by the position of the corrected measurement point obtained as a result of the iterative process as the three-dimensional shape of the object It is preferable to comprise.
[0025]
Further, in the estimation apparatus according to the present invention, based on the data value of the object image data at the plurality of measurement points, it is determined whether there are specular reflection measurement points that are specularly reflected at the plurality of measurement points, In the case where the specular reflection measurement point exists, selection means for selecting the direction of the virtual light source having a high probability of being the direction of the light source based on the specular reflection measurement point,
The cosine calculation means is preferably means for calculating the cosine by a virtual light source in the selected direction.
[0026]
Furthermore, in the estimation device according to the present invention, the temporary light source direction estimation means estimates the direction of the light source based on an evaluation function using a least square method and determines whether or not the plot is substantially a straight line. It is preferable to include a determination unit that performs the determination.
[0027]
In the estimation apparatus according to the present invention, it is preferable that the estimation unit is a unit that estimates the intensity of the light source based on a slope of a straight line set by a plot used for estimating the direction of the light source. .
[0028]
In the estimation apparatus according to the present invention, it is preferable that the object is a human face.
[0029]
Furthermore, it is preferable that the three-dimensional shape data is polygon data representing the three-dimensional shape of the object.
[0030]
The estimation method according to the present invention may be provided by being recorded on a computer-readable recording medium as a program for causing a computer to execute the estimation method.
[0031]
【The invention's effect】
According to the present invention, if only the data value of the object image data and the positional relationship between the approximate object and the virtual light source set in any of a plurality of directions are known, the direction of the light source and the three-dimensional shape of the object can be estimated. it can. Therefore, unlike the method of Sato et al., It is not necessary to obtain measurement points that can calculate the inverse matrix, and it is not necessary to perform matrix calculation, so the direction of the light source at the time of shooting is estimated by relatively simple calculation. Further, even if the three-dimensional shape of the object is unknown, this three-dimensional shape can be estimated. In addition, unlike the method in the pond, it is not necessary to take images from many directions, so the direction of the light source and the three-dimensional shape of the object can be easily estimated. As a result, when a real image and a virtual image such as a CG are combined, the virtual image is corrected so that it is illuminated from an appropriate light source based on the direction of the light source estimated in the real image and then combined. It is possible to obtain a composite image without any sense of incongruity. Further, in MPEG-4, for example, in an encoding method in which an image is transferred for each object and synthesized at the stage of viewing a moving image, if the object includes a real image and a virtual image, the actual image is illuminated in the illumination direction. Accordingly, the illumination direction of the virtual image can be corrected and synthesized, thereby obtaining a synthesized moving image with no sense of incongruity. Furthermore, since an object included in the real image can be represented as an appropriate three-dimensional image, a three-dimensional image without a sense of incongruity can be generated from the real image. Furthermore, in the case of an image obtained by shooting under sunlight, it is possible to easily perform backlight correction and density correction on the image by estimating the direction of sunlight in the image.
[0032]
In particular, when the position of the virtual light source coincides with the light source position at the time of object shooting, the direction vector from the light source to a certain measurement point on the object whose surface reflectance is substantially constant and the normal vector at the measurement point are It is preferable to estimate the direction of the light source at the time of photographing based on the characteristic that the relationship between the cosine formed and the intensity value of reflected light at the measurement point, that is, the data value of the object image data, is located on a straight line. That is, after the alignment of the object image data and the three-dimensional shape data, a plurality of measurements are performed on a portion corresponding to a portion where the surface reflectance of the object on the approximate object specified by the three-dimensional shape data is substantially constant. A point is set, and a cosine of a direction vector from the measurement point toward the virtual light source and a normal vector at the measurement point is obtained. Then, the relationship between the cosine and the data value of the object image data at the measurement point is plotted for each of the plurality of virtual light sources, and the direction of the virtual light source where these relationships are located on a straight line is the direction of the temporary light source at the time of shooting And Then, the cosine value in each plot is corrected so that the temporary light source direction plot is located on a straight line, and the cosine formed by the direction vector and the normal vector at the measurement point at which the temporary light source direction plot is obtained However, the position of the measurement point is corrected so as to substantially coincide with the corrected cosine to obtain a corrected measurement point.
[0033]
After this, the calculation of the cosine with the corrected measurement point as the new measurement point, plotting, estimation of the temporary light source direction, correction of the cosine and acquisition of the corrected measurement point are repeated a predetermined number of times, and the resulting temporary light source direction is photographed. It is preferable that the direction determined by the position of the corrected measurement point is estimated as the three-dimensional shape of the object by estimating the light source direction at the time.
[0034]
Before estimating the temporary light source direction, it is determined whether or not there are specular reflection measurement points at a plurality of measurement points, and if there are specular reflection measurement points, based on the specular reflection measurement points, Since the number of virtual light sources used for estimating the light source direction can be reduced by selecting the direction of the virtual light source that has a high probability of being the light source direction at the time of shooting, the calculation time can be shortened. That is, the existence of a specularly reflecting measurement point means that the light emitted from the actual light source to the measurement point is reflected regularly and proceeds in the direction of the imaging means that images the object. Therefore, the normal vector at the specular reflection measurement point bisects the angle formed by the vector from the light source to the specular reflection measurement point and the vector from the specular reflection measurement point to the imaging means. Therefore, when there is a specular reflection measurement point, the direction of the virtual light source used for the calculation can be narrowed down based on the normal vector at the measurement point and the vector from the specular reflection measurement point to the imaging means. Is.
[0035]
Further, when the real image and the virtual image are combined as described above by estimating the intensity of the light source, the virtual image is illuminated with an appropriate intensity based on the intensity of the light source estimated in the real image. Thus, the image can be synthesized after being corrected, whereby a synthesized image with a more uncomfortable feeling can be obtained.
[0036]
Furthermore, if the plane in which the measurement point exists is obtained by using the three-dimensional shape data as polygon data, the normal vector at the measurement point can be immediately calculated from the equation defining the plane, so the direction of the light source can be determined. It can be estimated more easily.
[0037]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
[0038]
FIG. 1 is a schematic block diagram showing a configuration of an estimation apparatus for estimating a light source direction and a three-dimensional shape of an object according to an embodiment of the present invention. As shown in FIG. 1, the estimation apparatus according to the present embodiment performs alignment between two-dimensional image data S0 representing an image including a person's face and three-dimensional shape data S1 representing a three-dimensional shape of the person's face. Positioning means 1 for obtaining combined data S2, memory 2 for storing three-dimensional shape data S1, and estimating means 3 for estimating the direction of the light source and the three-dimensional shape of the face in an image including a person based on the combined data S2. With. In the present embodiment, it is assumed that the light source is single and the surface reflectance of the skin color portion of the person's face is substantially constant.
[0039]
The alignment means 1 includes two-dimensional image data S0 representing the person's face image shown in FIG. 2A and three-dimensional shape data S1 approximately representing the three-dimensional shape of the person's face shown in FIG. Are aligned with each other. The three-dimensional shape data S1 can be changed in orientation, size, and center position according to the face image represented by the two-dimensional image data S0, and becomes polygon data that defines a three-dimensional shape by a plurality of surfaces. ing. In this alignment, the two-dimensional image data S0 and the three-dimensional shape data S1 are displayed on a monitor (not shown), and the three-dimensional shape represented by the three-dimensional shape data S1 is manually operated while observing the displayed image. This is performed by matching the orientation, size, and center position with the face image represented by the two-dimensional image data S0. By performing such alignment, synthesized data S2 representing a synthesized image as shown in FIG. 2C can be obtained. Here, the composite data S2 includes the two-dimensional image data S0 and the three-dimensional shape data S1 ′ after alignment. In FIG. 2, the horizontal direction in the drawing is the x direction, the vertical direction is the y direction, and the direction perpendicular to the paper surface is the z direction. In addition, the three-dimensional shape of the person's face is approximately represented by the three-dimensional shape data S1 ′ after the alignment.
[0040]
Here, assuming that the focal position f of the camera that captured the person image and the distance z0 from the camera to the person's face are known, they are on the three-dimensional shape represented by the three-dimensional shape data S1 ′ after alignment. The coordinates of the position (xw, yw, zw) and the coordinates (xv, yv) at the position corresponding to this position on the person image are described by the following perspective transformation formulas (1) and (2). Can do.
[0041]
xv = α · f · xw / (f + zw + z0) (1)
yv = α · f · yw / (f + zw + z0) (2)
Where α is the scale
x, y direction: direction on the plane of the person image
z direction: depth direction in a human image
Here, since the three-dimensional shape data S1 also has a value in the depth direction (z direction), the position on the three-dimensional shape data S1 ′ after alignment is obtained by using the relationship of the expressions (1) and (2). And the position on the two-dimensional image data S0 can be associated more accurately.
[0042]
When the focal position f of the camera is unknown, the three-dimensional shape data S1 ′ after the alignment and the two-dimensional image data S0 can be associated by the above formulas (1) and (2). However, since the depth dimension of the person is very small compared to the distance from the camera to the person who is the subject, in this case, the z direction can be ignored. Therefore, in this case, if the coordinates of an arbitrary position in the three-dimensional shape data S1 ′ after alignment are (x1, y1, z1), the coordinates at the position on the two-dimensional image data S0 corresponding to this position. May be (x1, y1).
[0043]
FIG. 3 is a schematic block diagram showing the configuration of the estimation means 3. As shown in FIG. 3, the estimation means 3 includes measurement point setting means 11 for setting a plurality of measurement points on the three-dimensional shape represented by the three-dimensional shape data S1 ′ after alignment, and setting in arbitrary plural directions. The cosine calculation means 12 for calculating the cosine formed by the direction vector from the virtual light source toward the plurality of measurement points and the normal vector at the plurality of measurement points, the calculated cosine, and the two-dimensional corresponding to the plurality of measurement points Plotting means 13 for plotting the relationship with the data value of the image data S0 for each direction of the plurality of virtual light sources, and provisional light source direction estimating means 14 for estimating the direction of the virtual light source in which the plot is substantially a straight line as the provisional light source direction. The cosine correction means 15 for correcting the cosine in the plot so that the plot for obtaining the temporary light source direction is located on a straight line, and the remainder formed by the direction vector and the normal vector at the measurement point However, the measurement point correction means 16 corrects the position of the measurement point so as to approach the corrected cosine and obtains the corrected measurement point, and the calculation, plotting, and temporary light source of the cosine using the corrected measurement point as a new measurement point Measurement point setting means 11, cosine calculation means 12, plotting means 13, provisional light source direction estimation means 14, cosine correction means 15 and measurement point correction so as to repeat direction estimation, cosine correction and correction measurement point acquisition a predetermined number of times. And a control means 17 for controlling the means 16. The provisional light source direction obtained as a result of the iterative process is estimated as the direction of the light source, and the shape determined by the position of the corrected measurement point obtained as a result of the iterative process is defined as Estimated as a dimensional shape.
[0044]
The estimation means 3 estimates the light source direction and the three-dimensional shape of the face as follows. First, estimation of the light source direction will be described. 4 and 5 are diagrams for explaining the estimation of the light source direction. In FIG. 4, for the sake of simplicity, the description will be given two-dimensionally using a cross-sectional view in a plane perpendicular to the y-axis of the three-dimensional shape represented by the three-dimensional shape data S1 ′ after alignment. First, the measurement point setting unit 11 sets a plurality of measurement points on the three-dimensional shape represented by the three-dimensional shape data S1 ′. In FIGS. 4 and 5, the measurement points are set to two points P1 and P2 for simplicity, but actually, a plurality of measurement points are set. Further, this measurement point is set to a portion corresponding to the skin color portion of the person's face on the three-dimensional shape represented by the three-dimensional shape data S1 ′. As shown in FIG. 4, it is assumed that there are five virtual light sources (assumed to be virtual light sources) L1 to L5 at intervals of 45 degrees from −90 degrees to +90 degrees around the y-axis. 5, the direction vectors LiP1 and LiP2 (i = 1 to 5) from the light source to the measurement points P1 and P2 for each of the light sources L1 to L5 and the measurement points for the plurality of measurement points P1 and P2 as shown in FIG. Cosine cos θ1 and cos θ2 with normal vectors n1 and n2 at P1 and P2 are calculated. Here, since the coordinate positions of the virtual light sources L1 to L5 and the measurement points P1 and P2 are known in advance, the direction vectors LiP1 and LiP2 can be calculated from the coordinate values of the virtual light sources L1 to L5 and the measurement points P1 and P2. . Further, since the three-dimensional shape data S1 ′ is polygon data, the surface on which the measurement points P1 and P2 exist can be easily obtained, and the normal vectors n1, n2 from the expressions ax + bx + cx + d = 0 that define the surface. Can be obtained as (a, b, c). Therefore, the cosine cos θ1 and cos θ2 can be calculated from the inner product of the direction vectors LiP1 and LiP2 and the normal vectors n1 and n2.
[0045]
Here, in the present embodiment, since the measurement point is set at a position where it is diffusely reflected, the intensity value of reflected light at the measurement point, the intensity of illumination, that is, the light emitted from the light source, the surface reflectance, the cutoff, and the cosine There is a relationship represented by the following equations (3) and (4).
[0046]
I (P1) = k1 · s1 · cos θ1 · Lc + Ie (3)
I (P2) = k2 · s2 · cos θ2 · Lc + Ie (4)
Where I (P1), I (P2): intensity values of reflected light at the measurement points P1, P2.
k1, k2: surface reflectance
s2, s2: cutoff values (0 or 1)
Lc: intensity of illumination
Ie: Ambient light intensity
Blocking refers to, for example, a part of a person's face that is generated depending on the direction of the light source and becomes a shadow of the nose and does not reach the light. In this way, the value of the blocking s is 0 in the portion where the light does not reach, and the value of the blocking s is 1 in the other portions because the light reaches and is reflected. Therefore, since the intensity value of the reflected light is 0 at the measurement point where s = 0, it is not used for the calculation of the light source direction in this embodiment. In this case, since there may be a shortage of measurement points used for calculation, it is preferable to set a new measurement point where s = 1 when there is a measurement point where s = 0.
[0047]
The intensity values I (P1) and I (P2) of the reflected light are the data values of the two-dimensional image data S0 at the measurement points P1 and P2. In the present embodiment, since the alignment unit 1 performs alignment between the two-dimensional image data S0 and the three-dimensional shape data S1, for example, the three-dimensional after alignment by the above formulas (1) and (2). Coordinate values corresponding to the measurement points P1 and P2 on the two-dimensional image data S0 are obtained from the coordinate values of the measurement points P1 and P2 in the shape data S1 ′. Then, the data values at the coordinate positions corresponding to the measurement points P1 and P2 in the two-dimensional image data S0 are obtained as the intensity values I (P1) and I (P2) of the reflected light at the measurement points P1 and P2.
[0048]
Since the intensity value of the reflected light becomes negative when the cosine cos θ1 and cos θ2 are negative, only measurement points that satisfy cos θ1, cos θ2> 0 are used in the present embodiment. In this case, since there may be a shortage of measurement points used for calculation, it is preferable to set new measurement points that satisfy cos θ1, cos θ2> 0 when cos θ1, cos θ2 ≦ 0.
[0049]
In the present embodiment, the surface reflectance at the point of diffuse reflection on the human face is substantially constant, and therefore k1 = k2 in the above formulas (3) and (4). The values of the interruptions s1 and s2 are both 1. From the above formulas (3) and (4), when the position of the virtual light source is correct, that is, coincides with the light source position at the time of shooting, as shown in the following formula (5), the reflected light on the face surface There is a feature that the ratio of the intensity values of the two coincides with the ratio of the cosine.
[0050]
Lc · k · s = (I (P1) -Ie) / cosθ1 = (I (P2) -Ie) / cosθ2 (5)
However, k: k1, k2
s: s1, s2
Lc · k · s = constant
Therefore, the plotting unit 13 plots the relationship between the intensity values of the reflected light at the plurality of measurement points P1 and P2 and the cosine cos θ1 and cos θ2 for each of the virtual light sources L1 to L5 as shown in FIG. In FIG. 6, the intensity value of the reflected light is normalized so that the maximum value is 1.0.
[0051]
As shown in FIG. 6, when the positions of the virtual light sources L1 to L5 coincide with the positions of the light sources at the time of photographing, the plot of the intensity value of the reflected light and the cosine is approximately from the relationship shown in the above equation (5). It will be located on a straight line. Therefore, the position of the virtual light source whose plot is located on a substantially straight line can be regarded as the position of the light source at the time of photographing. In FIG. 6, since the relationship between the intensity value of the reflected light and the cosine when the light source from the direction of −45 degrees, that is, the virtual light source L2 is used as the light source, is located on a substantially straight line, the provisional light source direction estimating means 14 The position of the virtual light source L2 is obtained as the position of the temporary light source at the time of shooting. Therefore, it can be considered that light is irradiated from the direction of −45 degrees in FIG. Note that the value of the intersection of this straight line and the vertical axis, that is, the intensity axis, is the intensity of the ambient light Ie. The slope of this straight line is Lc · k · s in the above equation (5), and the magnitude of the slope can be seen from FIG. Therefore, if the inclination is a, Lc = a / k · s, and the illumination intensity Lc can be obtained.
[0052]
Here, whether or not the plot is positioned on a straight line is determined by an evaluation function using, for example, the least square method. That is, if the straight line equation is y = ax + b,
[Expression 1]

N: Number of plots
xn: cosine value
yn: intensity value of reflected light
As a result, the values of a and b can be obtained. Then, an evaluation function cf shown in the following formula (7) is calculated from the values of a and b thus obtained.
[0053]
[Expression 2]

The direction of the virtual light source where the value of the evaluation function cf becomes the smallest value may be set as the temporary light source direction at the time of shooting.
[0054]
Next, estimation of the three-dimensional shape of the face will be described. FIG. 7 is a diagram showing a plot used for determining a temporary light source direction and a straight line C obtained from the plot. As shown in FIG. 7, since the three-dimensional shape data S1 ′ after alignment represents an approximate three-dimensional shape of the face, the actual face shape is represented by the three-dimensional shape data S1 ′. Therefore, the plot does not completely lie on the straight line C but is separated from the straight line C. Here, in this embodiment, since the three-dimensional shape data S1 ′ is aligned in the x and y directions, this gap occurs because the value in the z-axis direction deviates from the actual value. You can think of it as something. Therefore, in the present embodiment, the three-dimensional shape of the face is estimated by correcting the shift in the z-axis direction of the three-dimensional shape data S1 ′ by correcting the distance of each plot from the straight line C.
[0055]
First, it can be considered that the deviation from the straight line C of the plot obtained for the temporary light source direction obtained as described above is caused by the deviation of the inclination of the normal vector at the measurement point corresponding to the plot. For example, in the case of the plot corresponding to the measurement point P3 in FIG. 7, since the plot is shifted to the left side from the straight line C, the slope of the normal vector at the measurement point P3 is shifted and the cosine value is larger than the actual cosine value. You can think that it is getting smaller. Here, the actual cosine value is a cosine formed by a direction vector from the temporary light source toward the measurement point set on the three-dimensional shape of the actual face and a normal vector at the measurement point. . Therefore, the cosine correction means 15 corrects the z-value of the three-dimensional shape data S1 ′ at the measurement point P3 by correcting the inclination of the normal vector so as to increase the cosine value at the measurement point P3. The plot is moved in a direction approaching the straight line C. That is, if the data value of the two-dimensional image data S0 corresponding to the measurement point P3 is g (P3), g (P3) = ax + b
x = cos θc = (g (P3) −b) / a (8)
Where cos θc: cosine value when the measurement point P3 is located on the straight line C
Therefore, the value of θ at the measurement point P3, that is, the angle formed by the direction vector and the normal vector is corrected so as to approach the value of θc. When the cosine value is Y, the value of θ is
θ = cos ^-1 Y (9)
It can ask for. When the plot is shifted to the right from the straight line C as in the plot corresponding to the measurement point P4 in FIG. 7, the slope of the normal vector at the measurement point P4 is reduced so that the cosine value at the measurement point P4 becomes smaller. Is corrected, the z value of the three-dimensional shape data is corrected, and the plot is moved in the direction approaching the straight line C.
[0056]
FIG. 8 is a diagram for explaining the correction of the value of θ. In FIG. 8, θold is an angle formed by the direction vector PL and the normal vector nold at the measurement point P3 before correction, θnew is an angle formed by the direction vector PL and the normal vector nnew at the measurement point P3 after correction, and θc is This is the angle formed by the direction vector PL and the normal vector when the measurement point P3 is located on the straight line C. First, the vector nnew is calculated by the following equation (10).
[0057]
nnew = nold + [alpha] ([theta] old- [theta] c). (PL-nold) (10)
Where α: coefficient (0 <α <1)
Here, since the value of α has a value smaller than 1, the normal vector nnew moves on a vector (PL-nold) from the end point of the normal vector nold toward the light source L according to the value of α. . Here, the value of α is not set to 1 because the straight line C represents the provisional light source direction to the last, and thus is strictly different from the actual light source direction, and thus the provisional light source direction is different from the actual light source direction. This is because if the measurement points are corrected at once based on the light source direction, the three-dimensional shape of the face cannot be obtained accurately.
[0058]
When the normal vector nnew is calculated in this way, the z value of the three-dimensional shape data S1 'at the measurement point P3 is corrected by the measurement point correction means 16 based on the normal vector nnew. Specifically, when the measurement point P3 exists in a polygon having points A, B, and C as vertices as shown in FIG. The z value of the shape data S1 ′ is corrected. Here, assuming that the normal vector nnew is (a ′, b ′, c ′) and the coordinates of the measurement point P3 are (x1, y1, z1), an equation of a plane passing through the measurement point P3 and having a normal vector nnew. Is determined.
[0059]
a′x + b′y + c′z + d ′ = 0 (11)
d '=-(a'x1 + b'y1 + c'z1) (12)
At points A, B, and C, the values of x and y are not changed, and only the value of z is changed. Therefore, when the changed value of z is znew,
znew =-(a'x + b'y + d) / c '(13)
It can ask for. Thereby, the coordinates of the corrected points A, B, and C can be expressed as (x, y, znew).
[0060]
The above processing is performed for all the measurement points corresponding to the plot for which the temporary light source direction is determined, and the coordinate values of new measurement points are obtained, and this is used as the modified three-dimensional data Sc. A measurement point on the corrected three-dimensional data Sc is a corrected measurement point.
[0061]
Then, using the corrected measurement point as a new measurement point, setting a new virtual light source, the cosine cosθnew formed by the direction vector from the new virtual light source to the corrected measurement point and the normal vector nnew of the corrected measurement point, and the corrected measurement The relationship with the intensity value of the reflected light at the point is plotted, and a new straight line is obtained by the above formulas (7) and (8) to determine the direction of the new virtual light source, and the evaluation function cf is calculated. At this time, the new virtual light source may be set near the temporary light source. Then, it is determined whether or not the value of the evaluation function cf is larger than a predetermined value. If the value exceeds the predetermined value, the z value of the measurement point is corrected as described above to obtain a new corrected measurement point. The calculation of the temporary light source direction and the value of the evaluation function cf for the new corrected measurement point is repeated until the value of the evaluation function cf becomes smaller than the predetermined value. Then, the direction of the virtual light source when the evaluation function cf is equal to or less than a predetermined value is estimated as the direction of the light source, and the three-dimensional shape defined by the corrected measurement point at this time is estimated as the three-dimensional shape of the face. . Here, the estimated three-dimensional shape of the face is the three-dimensional shape of the skin color portion of the face. The above processing is performed by controlling the measurement point setting means 11, the cosine calculation means 12, the plotting means 13, the temporary light source direction estimation means 14, the cosine correction means 15, and the measurement point correction means 16 by the control means 17. Is called.
[0062]
Next, the operation of this embodiment will be described. FIG. 10 is a flowchart showing the operation of this embodiment. First, the alignment unit 1 aligns the two-dimensional image data S0 and the three-dimensional shape data S1 (step S1). The aligned composite data S2 is input to the estimation means 3. The estimation means 3 sets a plurality of measurement points on the three-dimensional shape represented by the three-dimensional shape data S1 ′ after alignment (step S2), and selects an arbitrary virtual light source from the virtual light sources L1 to L5. (Step S3). In this embodiment, the virtual light sources L1 are selected in order. Then, at each measurement point, the cosine formed by the normal vector and the direction vector between the measurement point and the virtual light source is calculated (step S4), and the relationship between the intensity value of the reflected light at each measurement point and the above-mentioned cosine is shown. 6 is plotted (step S5). Then, in the relationship between the intensity value of the reflected light and the cosine, it is determined whether or not the plot is located on a straight line by determining whether or not the evaluation function cf is equal to or less than a predetermined value (step S6) If step S6 is negative, the process returns to step S3, a different virtual light source is selected, and the processing from step S4 to step S6 is repeated. When step S6 is affirmed, the direction of this virtual light source is determined as a temporary light source direction at the time of shooting (step S7).
[0063]
When the temporary light source direction is determined in this way, it is determined whether or not the evaluation function cf when determining the straight line in the temporary light source direction is equal to or smaller than a predetermined value (step S8). If step S8 is negative, the position of the measurement point is corrected as described above (step S9), a new virtual light source is set near the temporary light source direction (step S10), and the process returns to step S3. Thereafter, the processing from step S3 to step S8 is repeated. If step S8 is affirmed, the temporary light source direction is determined as the direction of the light source at the time of photographing, and the three-dimensional shape of the face is determined from the coordinate position of the corrected measurement point (step S11), and the process is performed. finish.
[0064]
As described above, in this embodiment, when the position of the virtual light source is correct, that is, coincides with the light source position at the time of shooting, as shown in Expression (5), the ratio of the intensity values of the reflected light on the face surface is shown. Since the direction of the light source at the time of shooting is estimated based on the feature that the ratio of the cosine and the above cosine ratio coincides, it is only necessary to satisfy only the condition that the value of the cutoff s is not 0 for setting the measurement point. The direction of the light source can be estimated by a relatively simple calculation. Even if the three-dimensional shape of the object is unknown, the three-dimensional shape of the object can be easily estimated based on the estimated direction of the light source as compared with the method in the pond. As a result, when a real image and a virtual image such as a CG are combined, the virtual image is corrected so that it is illuminated from an appropriate light source based on the direction of the light source estimated in the real image and then combined. It is possible to obtain a composite image without any sense of incongruity. Further, in MPEG-4, for example, in an encoding method in which an image is transferred for each object and synthesized at the stage of viewing a moving image, if the object includes a real image and a virtual image, the actual image is illuminated in the illumination direction. Accordingly, the illumination direction of the virtual image can be corrected and synthesized, thereby obtaining a synthesized moving image with no sense of incongruity. Furthermore, since an object included in the real image can be represented as an appropriate three-dimensional image, a three-dimensional image without a sense of incongruity can be generated from the real image. Furthermore, in the case of an image obtained by shooting under sunlight, it is possible to easily perform backlight correction and density correction on the image by estimating the direction of sunlight in the image.
[0065]
In the above embodiment, the virtual light sources L1 to L5 are arranged two-dimensionally for the sake of explanation, but actually, a plurality of virtual light sources are arranged three-dimensionally with respect to the three-dimensional shape data S1. The relationship between the intensity of reflected light and the cosine is plotted for each measurement point to estimate the light source direction and the three-dimensional shape of the object at the time of photographing.
[0066]
Moreover, in the said embodiment, as shown to Formula (6) and (7), although the temporary light source direction is estimated using the evaluation function cf by the least square method, it is not limited to this. The plot shown in FIG. 6 may be displayed on the monitor, and the operator may observe it to obtain the direction of the virtual light source where the plot is located on the straight line as the direction of the temporary light source.
[0067]
Furthermore, the camera used when acquiring the two-dimensional image data S0 has a non-linear relationship between the input light and the output image data. For this reason, it is preferable to take a gray chart with a camera to be used in advance, obtain the relationship between the input and output to the camera, and correct the input / output characteristics of the camera so that this relationship is a straight line.
[0068]
Note that the luminance of the object surface in the actual environment includes specular reflection light in addition to the diffuse reflection light and environment light described above, and can be modeled by a combination of these lights. That is, the brightness of the object surface can be expressed by the following formula (14). Here, the mutual reflection is small as compared with diffuse reflection, specular reflection and ambient light, and can be ignored.
[0069]
I = Id + Is + Ie (14)
Where I: brightness of the object surface
Id: Diffuse reflected light
Is: Specular reflection light
Ie: Ambient light
Here, assuming that there is one light source as in the above embodiment, a case where the specular reflection position is detected from the face image using the three-dimensional shape data S1 ′ will be described.
[0070]
From the above equation (14), the difference ΔI between the intensity values at two arbitrary points on the face surface, that is, the data value, depends on the diffuse reflection light Id and the specular reflection light Is. First, assuming that only the diffuse reflection light Id exists, the intensity difference ΔI at the two measurement points P1 and P2 as shown in FIG. 5 at a place where the surface reflectance is constant is expressed by the following equation (15). As shown, it can be formulated.
[0071]
ΔI = k ・ s1 ・ cosθ1 ・ Lc-k ・ s2 ・ cosθ2 ・ Lc = k (s1 ・ cos1-s2 ・ cos2) Lc （15）
Since it can be assumed that the position and intensity of the light source do not change, the intensity difference ΔI is caused by either a cosine difference at two points, that is, by a normal vector direction difference or blockage. Here, assuming that a reference measurement point (for example, P1) is present at a position where it is diffusely reflected, when the intensity difference ΔI is caused by interruption, the other measurement point P2 is a shadow, so that 2 Even if the position of the point is slightly different, the intensity value changes so as to greatly decrease. On the other hand, when the intensity difference ΔI is caused by a cosine difference, the intensity difference ΔI is approximately proportional to the value of the cosine when the other measurement point P2 is also in a diffuse reflection position, but the other measurement point P2 is a mirror surface. If it is in a reflecting position, the intensity value changes so as to increase greatly even if the positions of the two points are slightly different. In this way, one measurement point P1 is set at the position where the diffuse reflection is performed on the three-dimensional shape data S1 ′ after the alignment, and the data value of the measurement point P1 and an arbitrary measurement point P2 (diffuse reflection) are adjacent thereto. The data value is not limited to the position where the data value is obtained, and the point at which the data value increases rapidly despite the slight difference between the two positions can be obtained as the specular reflection position.
[0072]
On the other hand, the specular reflection equation can be formulated as shown in the following equation (16).
[0073]
Is = k · s · cos ⁿ α ・ Lc (16)
That is, assuming that the light source is located at a position as shown in FIG. 11, the point P is obtained when the photographing means for photographing the object is in the position of the arrow R, that is, in the regular reflection direction at the point P of the light emitted from the light source L. The specular reflection occurs, and the intensity of the specular reflection light decreases as the position of the camera moves to a position V shifted from the angle α by an angle α.
[0074]
Therefore, as shown in FIG. 12, it is determined whether or not there are measurement points that are specularly reflected at the plurality of measurement points set by the measurement point setting means 11, and if there are, the measurement points that are specularly reflected are selected. Before selecting the provisional light source 20 and estimating the direction of the temporary light source (before step S3 in the flowchart shown in FIG. 10), it is determined whether or not there are measurement points that are specularly reflected at a plurality of measurement points. When there is a measurement point to be reflected, as shown in FIG. 11, the normal vector at that position (P) divides the angle formed by the direction vector PL and the direction vector PR into two equal parts. Thus, the direction in which the light source exists can be estimated to some extent. Therefore, even if the virtual light source is not set in any direction as in the above embodiment, if the virtual light source is set only in the direction where the light source is likely to exist, the provisional light source direction can be estimated, thereby reducing the amount of calculation. Thus, processing can be performed at high speed.
[0075]
In the above embodiment, the shape of the person's face is estimated. However, the present invention is not limited to this, and the three-dimensional shape of various objects can be estimated.
[Brief description of the drawings]
FIG. 1 is a schematic block diagram showing a configuration of an estimation apparatus according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining alignment between 2D image data and 3D shape data;
FIG. 3 is a schematic block diagram showing the configuration of estimation means
FIG. 4 is a diagram for explaining the direction of a virtual light source;
FIG. 5 is a diagram for explaining calculation of a cosine at a measurement point
FIG. 6 is a graph showing the relationship between cosine and intensity value of reflected light.
FIG. 7 is a diagram showing a plot used to determine a temporary light source direction.
FIG. 8 is a diagram for explaining correction of θ.
FIG. 9 is a diagram for explaining z value correction;
FIG. 10 is a flowchart showing the operation of the embodiment.
FIG. 11 is a diagram for explaining specular reflection;
FIG. 12 is a schematic block diagram showing a configuration of an embodiment provided with a selection unit.
[Explanation of symbols]
1 Positioning means
2 memory
3 Estimating means
11 Measuring point setting means
12 Cosine calculation means
13 Plotting means
14 Temporary light source direction estimation means
15 Cosine correction means
16 Measuring point correction means
17 Control means
20 selection means

Claims (21)

Based on an image obtained by photographing an object having a three-dimensional shape including a portion having a substantially constant surface reflectance under a substantially single light source, the direction of the light source and the three-dimensional shape of the object An estimation method for estimating
On the approximation object represented by the three-dimensional shape data after aligning the three-dimensional shape data indicating a general three-dimensional shape of the object image data and said object that represents the image of the object on said image A plurality of measurement points are set in a portion corresponding to a portion where the surface reflectance of the object is substantially constant in
Calculating a cosine formed by a direction vector from a virtual light source set in a plurality of directions to the plurality of measurement points and a normal vector at the plurality of measurement points;
The relationship between the cosine and the data value of the object image data corresponding to the plurality of measurement points is plotted for each direction of the plurality of virtual light sources,
Estimating the direction of the virtual light source in which the plot is substantially a straight line as the temporary light source direction,
Modify the cosine in the plot so that the plot for the temporary light source direction is located on the straight line,
Correcting the position of the measurement point so that a cosine formed by the direction vector and the normal vector at the measurement point approaches the corrected cosine, and obtaining a corrected measurement point;
The calculation of the cosine using the corrected measurement point as the new measurement point, the plot, the estimation of the temporary light source direction, the correction of the cosine and the acquisition of the corrected measurement point are repeated a predetermined number of times,
Estimating the temporary light source direction obtained as a result of the repetition process as the direction of the light source;
An estimation method characterized by estimating a shape defined by the position of a corrected measurement point obtained as a result of the repeated processing as a three-dimensional shape of the object . Based on the data value of the object image data at the plurality of measurement points, determine whether there are specular reflection measurement points that are specularly reflected at the plurality of measurement points,
When the specular reflection measurement point exists, based on the specular reflection measurement point, select the direction of the virtual light source that is highly likely to be the direction of the light source,
The virtual light source in said selected direction, estimating method according to claim 1, characterized in that the calculation of the cosine. The direction of estimation of the temporary light source, based on the evaluation function using the least squares method, according to claim 1, wherein said plot and performing by determining whether a substantially linear Estimation method. The estimation method according to claim 3 , wherein the predetermined number of times is a number of times until the evaluation function becomes a predetermined value or less. On the basis of the slope of the line that is set with plots were used to estimate the direction of the light source, estimating method according to any one of the claims 1 to 4, characterized in that to estimate the intensity of the light source. Estimation method according to any one of claims 1 to 5, wherein the object is a human face. Estimation method according to any one of claims 1 6, wherein the three-dimensional shape data is polygon data representing a three-dimensional shape of the object. Based on an image obtained by photographing an object having a three-dimensional shape including a portion having a substantially constant surface reflectance under a substantially single light source, the direction of the light source and the three-dimensional shape of the object An estimation device for estimating
On the approximation object represented by the three-dimensional shape data after aligning the three-dimensional shape data indicating a general three-dimensional shape of the object image data and said object representing the image of the object on the image Measuring point setting means for setting a plurality of measuring points in a portion corresponding to a portion where the surface reflectance of the object is substantially constant;
Cosine calculation means for calculating a cosine formed by a direction vector from the virtual light source set in any plurality of directions to the plurality of measurement points and a normal vector at the plurality of measurement points;
Plotting means for plotting the relationship between the cosine and the data value of the object image data corresponding to the plurality of measurement points for each direction of the plurality of virtual light sources;
A temporary light source direction estimating means for estimating a virtual light source direction in which the plot is substantially a straight line as a temporary light source direction;
Cosine correction means for correcting the cosine in the plot so that the plot for obtaining the temporary light source direction is positioned on the straight line;
Measurement point correcting means for correcting a position of the measurement point so as to obtain a corrected measurement point so that a cosine formed by the direction vector and the normal vector at the measurement point approaches the corrected cosine;
The measurement point setting means for repeating the calculation of the cosine using the corrected measurement point as the new measurement point, the plot, the estimation of the temporary light source direction, the correction of the cosine and the acquisition of the corrected measurement point a predetermined number of times. Control means for controlling the cosine calculating means, the plotting means, the temporary light source direction estimating means, the cosine correcting means, and the measuring point correcting means;
Estimating means for estimating the temporary light source direction obtained as a result of the iterative process as the direction of the light source, and estimating the shape defined by the position of the corrected measurement point obtained as a result of the iterative process as the three-dimensional shape of the object estimating device characterized by comprising and. Based on the data values of the object image data at the plurality of measurement points, it is determined whether or not there are specular reflection measurement points that are specularly reflected at the plurality of measurement points, and when the specular reflection measurement points exist Comprises a selection means for selecting the direction of the virtual light source that is highly likely to be the direction of the light source based on the specular reflection measurement point,
9. The estimation apparatus according to claim 8 , wherein the cosine calculation means is means for calculating the cosine by a virtual light source in the selected direction. The temporary light source direction estimating means is a determination means for performing estimation of the direction of the temporary light source by determining whether or not the plot is substantially a straight line based on an evaluation function using a least square method. estimating apparatus according to claim 8 or 9, characterized in that it comprises. The estimation device according to claim 10 , wherein the predetermined number of times is a number of times until the evaluation function becomes a predetermined value or less. The said estimation means is a means which also estimates the intensity | strength of the said light source based on the inclination of the straight line set by the plot used for estimation of the direction of the said light source, The any one of Claim 8 to 11 characterized by the above-mentioned. The estimation apparatus according to claim 1. Estimating apparatus of any one of 12 claims 8, wherein the object is a human face. We are estimating device according to any one of claims 8 to 13, wherein the three-dimensional shape data is polygon data representing a three-dimensional shape of the object. Based on an image obtained by photographing an object having a three-dimensional shape including a portion having a substantially constant surface reflectance under a substantially single light source, the direction of the light source and the three-dimensional shape of the object A computer-readable recording medium storing a program for causing a computer to execute an estimation method for estimating
The program is represented by the three-dimensional shape data after alignment of object image data representing the image of the object on the image and three-dimensional shape data representing a general three-dimensional shape of the object. A procedure for setting a plurality of measurement points in a portion corresponding to a portion where the surface reflectance of the object on the approximate object is substantially constant;
A procedure for calculating a cosine formed by a direction vector from a virtual light source set in an arbitrary plurality of directions to the plurality of measurement points and a normal vector at the plurality of measurement points;
A procedure for plotting the relationship between the cosine and the data value of the object image data corresponding to the plurality of measurement points for each direction of the plurality of virtual light sources;
A procedure for estimating a virtual light source direction in which the plot is substantially a straight line as a temporary light source direction;
A procedure for correcting the cosine in the plot so that the plot for obtaining the temporary light source direction is positioned on the straight line;
A procedure for obtaining a corrected measurement point by correcting the position of the measurement point so that a cosine formed by the direction vector and the normal vector at the measurement point approaches the corrected cosine;
Calculating the cosine using the corrected measurement point as the new measurement point, the plot, estimating the temporary light source direction, correcting the cosine and obtaining the corrected measurement point a predetermined number of times;
A procedure for estimating the temporary light source direction obtained as a result of the iterative process as the direction of the light source, and estimating a shape defined by the position of the corrected measurement point obtained as a result of the iterative process as a three-dimensional shape of the object; computer readable recording medium characterized by having a. A procedure for determining whether or not there are specular reflection measurement points that are specularly reflected at the plurality of measurement points based on data values of the object image data at the plurality of measurement points;
A step of selecting the direction of the virtual light source having a high probability of being the direction of the light source based on the specular reflection measurement point when the specular reflection measurement point exists;
The computer-readable recording medium according to claim 15 , wherein the procedure of calculating the cosine is a procedure of calculating the cosine by using a virtual light source in the selected direction. Procedure for estimating the direction of the temporary light source, based on the evaluation functions using the least square method, according to claim 15 or 16, characterized in that it comprises a procedure for determining whether the plot is substantially linear The computer-readable recording medium as described. The computer-readable recording medium according to claim 17 , wherein the predetermined number of times is a number of times until the evaluation function becomes a predetermined value or less. The method according to any one of claims 15 to 18 , further comprising a step of estimating the intensity of the light source based on a slope of a straight line set by a plot used for estimating the direction of the light source. Computer-readable recording medium. Computer-readable recording medium of any one of claims 15 to 19, wherein the object is a human face. Computer-readable recording medium of any one of claims 15 to 20, wherein the 3-D shape data is polygon data representing a three-dimensional shape of the object.

Images