JP2017188771A - Imaging system, and display method of image or video image - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging system, and a display method of an image or video image that can perform stable privacy protection regardless of the content of an imaged image or video image.SOLUTION: The imaging system processes an area related to privacy information within an imaged original image for protecting privacy, and synthesizes it with an unmanned image area which does not include the privacy information within the original image, for display or transmission. Then the image or video image after the process for protecting privacy is displayed in the area related to the privacy information in the imaged original image, and the original image is displayed in the area not including the privacy information. Then, as a result, the inside of the original image area where only a portion of a face is displayed is recognized as the inside of the area related to the privacy information, and the image or video image after the process for protecting privacy is displayed.SELECTED DRAWING: Figure 8

Description

  Embodiments described herein relate generally to an imaging system and an image and video display method that can include privacy information within an imaging target.

  Images and videos taken at public facilities and places where people gather often include the faces and shapes of people belonging to privacy information, their possessions and clothes. In particular, it is necessary to consider personal privacy protection when an unspecified number of people can view the images and videos.

  On the other hand, it is possible to identify an individual and an organization to which the person belongs from the characteristics of the person's face and clothes in the image and video. Therefore, from the viewpoint of protecting the privacy of individuals and affiliated organizations, there is a need for a technique that makes it difficult to identify a person from a captured image or video or identify an affiliated organization.

JP 2009-194687 A JP 2004-62560 A

  In Patent Document 1, as a technique for protecting privacy by processing a captured image or video, the location of a face in the image is detected, and privacy mask processing such as mosaic, blurring, and painting is performed on the face or an image area of the background. Has been given. Also in Patent Document 2, the location of a face in an image is detected, mosaic processing is performed only on the face image portion, and a clear image display is performed on clothes and personal belongings other than the face.

  In any of the prior arts, the face image position detection technique is commonly adopted. The face screen position detection technique generally used here often uses an arrangement relationship between the shapes of the captured images and facial parts (eyes, nose, mouth, eyebrows, ears, etc.) in the video.

  Then, in the face image position detection technique, for example, when a part of the face is hidden by an obstacle placed on the front face, the face image is not recognized, and there is a risk that a part of the face is displayed in a clear image state. There is sex. Therefore, in the above-described conventional technology, there are cases where the privacy protection function is insufficient depending on the contents of the image or video.

  Therefore, it is required to provide a photographing system and a method for displaying an image or video that enable stable privacy protection regardless of the content of the photographed image or video.

  As a photographing system in the present embodiment, privacy processing is performed on a region related to privacy information in a photographed original image, and the region is combined with an unmanned image region that does not include privacy information and displayed or transmitted.

  As a method for displaying an image or video in the present embodiment, an image after privacy processing is displayed in an area related to privacy information in a photographed original image, and an original image is displayed in an unmanned image area.

  As a result, the original image area where only a part of the face is displayed is also recognized as the area related to the privacy information, and the privacy processed image is displayed.

The subject example explanatory drawing of a prior art. Explanatory drawing of the area | region in the image | photographed original image. An explanatory view of an example of a generation method of an unattended image. An example explanatory drawing of the area extraction method related to privacy information. Explanatory drawing of an example of the production | generation method of the image by which privacy processing was carried out. FIG. 7 is an explanatory diagram illustrating an example of a display form of an image displayed or transmitted in the present embodiment. Explanatory drawing of the application example expressing the movement in the area | region relevant to privacy information. An application example that expresses the overlap in the area related to privacy information. An application example in which the color of the body area is changed within the area related to privacy information. Explanatory drawing of the structure in the imaging | photography system in this embodiment. Explanatory drawing of an imaging system in the present embodiment via a communication path. Explanatory drawing of the application example of the imaging | photography system via a communication path. FIG. 3 is an explanatory diagram of an architecture in a program executed in the imaging system. Flowchart explanatory drawing which shows the flow which produces | generates an unattended image. Explanatory drawing of a flowchart showing up to generation of an image to be transmitted / displayed. Data structure explanatory drawing which shows an example of the management information regarding a movement information log | history. Data structure explanatory drawing which shows an example of the management information regarding an image history. Data structure explanatory drawing which shows an example of the management information regarding congestion history.

  Hereinafter, embodiments will be described with reference to the drawings. The problems associated with privacy protection for images using conventional technology have been described above. Further, the above-described problem will be specifically described with reference to FIG.

  FIG. 1A shows an example 10 of a captured original image. Here, persons 2, 4, and 6 are displayed in the photographed original image 10. In particular, a part of the face of the photographed person 6 in the original image is hidden under the influence of an obstacle (such as a stick) of the photographed person 4 in the original image.

  FIG. 1B shows the result of the person range detection 22 performed on the photographed original image 10 (FIG. 1A). Here, the portion where the person range detection 22 is performed is indicated by a square frame.

When the person range detection 22 is performed using only the face screen position detection technique as in the prior art, a part of the face of the photographed person 6 in the original image is hidden, so the person 6 in the photographed original image 6 Is not recognized as a person.

  As an example of privacy processing (privacy protection measures for images), an image after blurring 24 is performed on a detection range recognized as a person is shown in FIG. Since the person 6 in the original image captured as described above is not recognized as a person, the face image of that person is clearly displayed. Therefore, it can be seen that privacy protection measures using only the face screen position detection technique as in the prior art are insufficient.

  In this embodiment, a captured image or video is divided into a plurality of areas, and privacy processing is performed on a specific area among them. For convenience of explanation of the present embodiment, a description specific to images will be given below. However, the present embodiment is not limited to the following, and the same embodiment may be applied to video. The definition for each area in this embodiment will be described with reference to FIG.

  An area in which privacy information is not included in the captured original image 30 shown in FIG. On the other hand, an area including an image related to privacy information of an individual or group in the photographed original image 30 is referred to as an area 38 related to privacy information. In the example of FIG. 2, four persons are shown in the original image 30. The privacy information is related to the area where the four people are displayed (the area in the image) and the article related to them (the belongings, etc.) and the living things (for example, plants being transported, animals such as pets). And is identified separately from the unmanned image area 34.

  Here, the person included in the area 38 related to the privacy information is not limited to the person who can be specified most recently, but includes an unspecified majority person. Even if an unspecified majority person who cannot specify an individual immediately after photographing is taken, there is a risk that the individual is specified when the image is viewed by another unspecified majority person via the network.

  An area where all belongings of an arbitrary person or organization / organization are photographed / displayed may be included in the area 38 related to privacy information in the present embodiment. For example, an area where personal belongings (carrying goods and belongings) not shown in the original image 30 are temporarily left at the shooting site and reflected in the original image 30 may be included in the area 38 related to privacy information. . This is because if a close person sees a favorite item such as a bag, hat, or glasses, the owner (belonging to the other party) can be expected, which is related to privacy information.

  As a belonging of any organization or group related to privacy information, it is not limited to national flags, badges, or emblems, but belongings (clothes, hats, shoes, etc.) and personal belongings (organization / group-specific) , And other items).

In the unmanned image area 34,
(1) An image area of the same shooting site taken in an unattended state (2) An image area that does not change within a predetermined time (3) An image area that is not recognized / estimated as a person or its belongings. Here, the predetermined time means within a range of 1 minute to 1 month, preferably within a range of 1 hour to 1 week. Even in an unmanned shooting site, the original image 30 that has been shot changes due to poster replacement, stock change at a free magazine location, or the like. Therefore, the range of the predetermined time is determined in consideration of these image change periods.

  On the other hand, all areas that are not included in the unattended image area 34 in the photographed original image 30 may be regarded as the area 38 related to privacy information.

  In this embodiment, an unmanned image 32 (described later using FIG. 3C) that does not include privacy information is generated in advance. Then, referring to the unmanned image 32, an area 38 related to privacy information in the photographed original image 30 is extracted. Therefore, even if a situation occurs in which some of the people 2, 4, and 6 in the original image overlap (part of the face is hidden), the face (part) is clearly displayed in the image after privacy processing. There is nothing. That is, by adopting the above display method, it is possible to stably protect the privacy without depending on the content of the captured image or video.

  In this embodiment, the unmanned image 32 shown in FIG. 3C is generated using the characteristics of the unmanned image region 34 described above. That is, as a method of “photographing the unmanned state at the same photographing site”, which is the feature of the above (1), for example, an unmanned image 32 is obtained by photographing the site in a busy hour such as at midnight or before dawn. Also good.

  As another method for generating the unmanned image 32, a method for generating the unmanned image 32 in a time zone close to the photographing time of the original image 30 will be described with reference to FIG. First, an example of a method for generating the unmanned image 32 using only the feature described in the above (2) that does not change within a predetermined time will be described.

  FIG. 3A shows an original image 30 taken in a certain time zone. Since the photograph was taken at a certain time zone, four moving persons 42, 44, 46, and 48 are shown in the original image 30. Here, in the original image 30 of FIG. 3 (a) alone, information on the unmanned image 32 (image shown in FIG. 3 (c)) hidden behind the moving persons 42, 44, 46, and 48 cannot be obtained.

  Next, FIG. 3B shows an original image 36 taken a little later than that. In FIG. 3 (b) as compared with FIG. 3 (a), the moving persons 42, 44, 46, and 48 are photographed after moving, and thus are hidden in FIG. 3 (a) (in the unmanned image 32). The corresponding image is visible. Using the feature, it is possible to generate the unmanned image 32 (FIG. 3C) when there is no moving person 42, 44, 46, 48.

  In the area where different images are displayed between the original images 30 and 36 in FIG. 3A and FIG. 3B (that is, the absolute value of the difference between the pixel values is large), One of the images related to the privacy information of 42, 44, 46, and 48 and the “image hidden behind the moving persons 42, 44, 46, and 48 in the unmanned image 32” are shown. Although not shown in FIG. 3, if an original image taken after FIG. 3B is also used, “an image hidden behind the moving persons 42, 44, 46, 48 in the unmanned image 32” Only is easy to extract.

  That is, in many cases, a part of the unattended image 32 is shown at two common pixel positions in the three original images collected by shifting the shooting times. Therefore, if the majority of pixel values is taken for each pixel position, the extraction probability of an image corresponding to the unmanned image 32 is improved.

  Here, the method of generating the unmanned image 32 using three original images having different shooting times has been described. However, the present invention is not limited to this, and the unmanned image 32 may be generated using four or more original images. By using four or more original images, the extraction accuracy of pixels in the unmanned image 32 is improved.

  However, in the method of generating the unmanned image 32 shown in FIG. 3, when the moving person 42, 44, 46, 48 includes a person with extremely slow moving speed, the complete unattended image 32 cannot be obtained. Cases arise. In such a case, the accuracy of the generated unmanned image 32 may be improved by performing an image interpolation process on the portions hidden behind the shadows of the moving persons 42, 44, 46, and 48.

So far, the method of generating the unmanned image 32 using the plurality of original images 30 and 36 having different shooting times has been described. However, the present embodiment is not limited to this. For example, an image analysis is performed on only one original image 30 shown in FIG. May be determined.
Further, in the method for generating the unmanned image 32 shown in FIG. 3, the camera that captures the original image 30 (the imaging unit 112 in FIG. 8) does not necessarily have to be fixed at the shooting site. For example, the unmanned image 32 may be generated using a movable camera (imaging unit 112). When the unmanned image 32 is generated by a movable camera (imaging unit 112), the movable camera (imaging unit 112) is held by hand at a shooting site for a specific time. Recently, since a movable camera (imaging unit 112) has a camera shake correction function, it is possible to accurately collect still images even if the camera (imaging unit 112) is slightly shaken.

  There are places in the site where the original image 30 is photographed where layout changes such as poster replacement and installation / removal of specific equipment are severe. If there is a great discrepancy between the time when the unmanned image 32 was generated and the time when the original image 30 was taken at that location, even unnecessary images related to the replaced posters and the rearranged specific equipment are erroneously detected as areas 38 related to privacy information. There is a high possibility. Therefore, by adopting the unmanned image 32 generation method shown in FIG. 3, there is an effect that the time interval between the unmanned image 32 generation time and the photographing time of the original image 30 can be shortened.

  Typically, posters and the like are often replaced in one to two weeks, so it is desirable to set the above time interval within one week. In addition, the day of the week for changing the poster and the day of the week for changing the layout of specific equipment vary depending on the location. Therefore, even if the above time interval is set within one week, there is a high risk that the poster or layout will be changed the day after the unmanned image 32 generation date. In many cases, it is unlikely that poster replacement or layout change will be performed during crowding. Accordingly, the time interval between the time when the unmanned image 32 is generated and the time when the original image 30 is captured may be set to one day or less. However, when there is a pamphlet installation place that can be taken home on a predetermined pillar or wall, erroneous detection of the area 38 related to privacy information is likely to occur even if the time interval is set to one day or less. Therefore, in such a place, it is desirable to set the time interval to 2 hours or less.

  As another method for generating the unmanned image 32, the above-described technique (3) of “estimating a person area or a person belonging area in an image” may be combined. As described above, in the area where different images are displayed between the original images 30 and 36 in FIGS. 3A and 3B, “related to the privacy information of the moving persons 42, 44, 46, and 48”. Either “Image” or “Image hidden behind moving persons 42, 44, 46, 48 in unmanned image 32” is shown. Then, image components included in the unmanned image 32 may be selected using an image analysis technique.

  That is, the image of a person or animal includes a common shape (consisting of a head, a torso, and a foot (or a hand)). Also, belongings of specific persons such as bags and rucksacks have characteristic shapes (such as rectangular parallelepipeds and trapezoidal columns). Therefore, an image that does not include a characteristic shape in the person or animal or the attributed item may be estimated and selected as an image component included in the unmanned image 32.

  As described above, in this embodiment, a plurality of generation methods for the unmanned image 32 are allowed. In the present embodiment, not only the single unmanned image 32 is generated, but a plurality of unmanned images 32 may be generated simultaneously by the above-described plurality of generation methods.

  FIG. 4A shows a method of extracting an area 38 related to privacy information with reference to the unmanned image 32 generated by the above method. As the simplest extraction method, the difference (FIG. 4A (c)) between the photographed original image 30 (FIG. 4A (a)) and the unmanned image 32 (FIG. 4A (b)) may be calculated. In addition, as a method of referring to the unmanned image 32, the comparison may be made after some editing is performed on both images, or some arithmetic processing may be performed at the time of comparison between the two images.

  Further, as another application example with respect to FIG. 4A, in the present embodiment, the probability related to privacy information is calculated for each area in the original image 30, and the area 38 related to privacy information is extracted based on the calculation result. Also good. For example, as will be described later, in the human probability distribution measurement unit 160 in FIG. 8, characteristic shapes and temporal changes in the region 38 related to privacy information are extracted, and the value of the human probability is calculated for each extracted region. To do.

  In the same manner, characteristic shapes and temporal changes from all regions in the original image 30 in FIG. 4A (a) (and the original image 36 taken slightly after FIG. 3B) (FIG. 5B). Movement information 88) is extracted. Next, by using a pattern matching method or the like, the degree of coincidence with a human or animal body shape or a human belonging shape (such as a bag or a rucksack) is detected, and a probability related to privacy information is calculated. At the same time, the degree of coincidence between the contour shape change (movement information 88 or its detailed information) corresponding to the time change and the library data stored in advance is detected and added to the probability related to the privacy information. Furthermore, the arrangement relationship between the shapes of the components in the face (eyes, nose, mouth, hair, ears) may be examined to check the degree of coincidence with the face, and may be added to the probability related to privacy information.

  An area where the probability related to the privacy information calculated in this way exceeds a predetermined determination value may be extracted as the area 38 related to the privacy information. Here, when the predetermined determination value is set to 50% (preferably 70%), high extraction accuracy for the area 38 related to privacy information can be ensured.

  The calculation of the probability related to the privacy information, the determination of the calculation result, and the extraction of the area 38 related to the privacy information may be performed in the person probability distribution measurement unit 160 described above.

  FIG. 4B shows a method for generating an image 40 privacy-processed using the area 38 related to privacy information extracted in this way. That is, privacy processing is performed only on the region corresponding to the region 38 (FIG. 4B (b)) related to the privacy information in the photographed original image 30 (FIG. 4B (a)) (FIG. 4B (c)).

  In FIG. 4B (c), a blurring process is performed as the privacy-processed image 40. However, the present invention is not limited to this, and as other privacy processing methods, changes such as mosaic processing, painting, shape change, color tone change, density change, etc. may be added, or conversion (replacement) to a completely different image may be performed.

  For example, as a specific example of privacy processing by a shape change, for example, only the width in the horizontal direction may be narrowed like a match stick. For example, moving people 42 to 48 overlap each other in a crowded place, and it may be difficult to grasp the number of people photographed by a viewer of an image after privacy processing. By narrowing only the lateral width of the area 38 related to privacy information with respect to the original image 30 obtained by photographing such a crowded place, there is an effect that the viewer can easily grasp the number of people. In this embodiment, since the unmanned image 32 itself is not processed, even if only the width in the horizontal direction of the area 38 related to privacy information is narrowed, there is an effect that the uncomfortable feeling given to the viewer can be minimized.

  As will be described later with reference to FIG. 6, in this embodiment, it is possible to predict and extract only an area where a person is shown in the area 38 relating to privacy information. Therefore, as an example of image conversion (replacement) for an area in which a person is shown, the person may be converted (replacement) into another image such as an animal such as a cat or a panda or a car or an airplane. If an unspecified number of viewers for the privacy-processed image are interested only in the crowded situation at the shooting site, the privacy-processed person image is not very interested. On the other hand, there are many women who are interested in images of small animals such as cats and pandas and boys who are interested in mechanical devices such as cars and airplanes. Therefore, by performing image conversion (replacement) as described above, there is an effect of giving a favorable impression to the viewer and giving it an interest.

  A method for creating an image to be displayed or transmitted to the viewer is shown in FIG. 4C. By superimposing the privacy-processed image 40 (FIG. 4C (a)) obtained by the method described above with reference to FIG. 4B on the unattended image 32 (FIG. 4C (b)) collected in advance, FIG. 4C (c) Can be generated. In this case, the privacy-processed image 40 is superimposed on the upper layer (near side) of the unmanned image 32. In this way, the privacy processed image 40 itself may have a certain degree of transparency so that the background unmanned image 32 can be seen. For example, when the shooting site is very crowded with moving persons 42 to 48, the background unmanned image 32 is difficult to see, making it difficult for the viewer to identify the shooting site. If the background unmanned image 32 is seen with transparency as described above, an effect is obtained that makes it easy for the viewer to identify the shooting site, no matter how crowded.

  Application examples for the image display method in the present embodiment described above will be described with reference to FIGS.

  The privacy processing method for the area 38 related to the privacy information is not limited to the above-described mosaic processing, blurring, painting, shape change, color change, density change and image conversion (replacement), but the movement information of the area 38 itself related to the privacy information. FIG. 5 shows a method for providing both of them simultaneously.

  In the original image example 82 in FIG. 5A, the left person moves from left to right, and the right person squats (ie, sinks down). In this case, the original image 82 is collected in time series, and the movement direction of the person can be extracted by detecting the movement of the area 38 related to the privacy information corresponding to the time lag.

  The movement information example 88 extracted by the above method is indicated by an arrow in FIG. FIG. 5C shows an example 52 in which the entire original image 82 is privacy-processed using the movement information 88. In the example of FIG. 5C, “blurring” is applied as a privacy processing method before the movement information 88 is used. However, the present invention is not limited to this, and privacy processing may be performed by any method described above.

  In the example of FIG. 5C, as the method of adding the movement information 88 to the privacy processing, the image in the area 38 related to the privacy information photographed a predetermined time before is displayed in an overlapping manner. In addition, the transparency when overlaid is displayed with gradation, and the direction of movement is easy to see. That is, as shown in FIG. 5C, the moving direction of the moving person is easily seen by increasing the transparency in the order of the photographed image.

  Further, as can be seen from FIG. 5C, the moving speed may be expressed based on the “deviation amount” with respect to the person position displayed a predetermined time before overlapping. That is, it can be seen that the left person has a large “deviation amount” and is moving at a relatively high speed. On the other hand, the “deviation amount” of the right person is small and it can be seen that the person is moving slowly.

  In this way, by overlapping and displaying a part of images taken in the past, the moving direction and moving speed of the person can be expressed, and there is an effect that the viewer (unspecified majority) can easily grasp the movement state of the person. The display method of the moving direction and moving speed of the moving person is not limited to the above, and any other method may be used. For example, the speed and direction of movement may be indicated by arrows having different lengths.

  The color for displaying the moving person may be changed according to the moving speed of the moving person. In general, there is a characteristic that the red color is more conspicuous than the blue color. Therefore, the conversion color may be changed from a blue color to a red color as the moving speed increases. Further, the converted color may be expressed deeply according to the moving direction. In this way, when the moving direction and moving speed of the moving person are changed by the color change or the density change, it is not necessary to superimpose a part of the image taken in the past, and the area 38 related to the privacy information is narrowed. There is an effect that can be expressed.

  FIG. 5D shows an example region 54 related to privacy information extracted by the method described with reference to FIG. 4A. FIG. 5E shows an example of the unmanned image 56 generated by the method described in FIG. A diagram generated by combining FIG. 5C and FIG. 5B using FIG. 5D corresponds to FIG. The image generated in FIG. 5F is viewed by an unspecified majority as an image 58 to be transmitted / displayed.

  For example, FIG. 6 shows a display example when the shooting site is in a congested state and a region 38 related to privacy information is configured in a state where a plurality of persons overlap. That is, in the original image example 84 shown in FIG. 6A, two people are photographed on the right side in an overlapping manner. When this original image 84 is subjected to privacy processing, as shown in an example 62 in which the entire privacy processing in FIG. Although not shown, a plurality of privacy-processed images 62 having different privacy processing strengths (for example, “bokeh amount” strength) are prepared.

  Then, as will be described later using the person probability distribution measurement unit 160 of FIG. 8, the probability of person-likeness for each area in the area 64 related to the extracted privacy information is estimated and calculated. As a result, as shown in FIG. 6C, area division corresponding to different person probability distributions is made in the area 64 related to privacy information. Here, a part of the body contour of the back person (or part of the face) of the two people photographed on the right side is hidden by the front person. As a result, it is considered that the person probability is low in the area where the person on the back side is shown.

  FIG. 6E shows the result of overlaying the plurality of privacy-processed images 62 obtained in FIG. 6B and the unmanned image 66 shown in FIG. 6D using the information in FIG. 6C. . As shown in FIG. 6E, the transparency of an area having a low human character (person probability) in the area 64 related to privacy information is increased, and the privacy processing strength (for example, “blur amount”) is increased.

  In this way, display of unnecessary image areas can be achieved by increasing or hiding the transparency (or increasing the strength of privacy processing) of the low-person-like (person probability) area in the area 64 related to privacy information. Can be limited. As a result, the display of an area that seems to be a person in the area 64 related to the privacy information (high probability of person) is relatively emphasized, and an image that is easy to view for an unspecified number of viewers can be provided.

  In particular, when the display method of the present embodiment is used as is apparent from a comparison between the image of FIG. 6E and FIG. 1C obtained using the conventional technique, a part of the image is hidden by an obstacle or the like. The privacy protection function for a person (the face is clearly displayed in FIG. 1C is not displayed in FIG. 6E) is significantly improved.

  FIG. 7 shows an application example in which the display method (privacy processing method) is changed depending on the region position in the region 74 related to privacy information.

  When an image of a specific person is disclosed to an unspecified majority, there is a high risk that the face (head) of the specific person is recognized and specified as an individual. Therefore, it is difficult to identify individuals by applying privacy processing (mosaic, painting, blurring, color conversion, density conversion, shape conversion, replacement with another image, etc.) to the display area of the face (head).

  On the other hand, by recognizing the display area of the body excluding the face, there is a feature that the organization or group to which the person belongs can be easily specified. For example, the name of the junior high school or high school where students on their way home from club activities can be inferred from the shape and color of uniforms worn by junior high school students and high school students. Therefore, for example, by changing the color or shape of clothes in the photographed original image 86, it is difficult to specify the organization or group to which the person belongs.

  Corresponding to the above situation, the head range and body range in the region 74 related to the privacy information extracted in FIG. 7C using the head range detection unit 152 and the body range detection unit 154 in FIG. May be automatically recognized individually. Based on the recognition result, the color and shape of the clothes are changed in FIGS. 7B and 7E.

  In the privacy processing method for the original image 86 in the present embodiment, a wide variety of processing methods such as mosaic, painting, blurring, color conversion, density conversion, transparency change, shape conversion, and replacement with another image are allowed. In the application example shown in FIG. 7 (e), privacy-processed images using different methods for the head (including the face) within the same person and at least a part of the body region other than the head (for example, the clothing part). Is sent or displayed. Here, the “head” that performs privacy processing different from the body other than the head (or a part of the body such as the clothing part) does not necessarily need the entire head, for example, a part of the head such as a face or a hairstyle. But it ’s okay. As another example of privacy processing, the strength of privacy processing such as changing the “blur amount” may be changed.

  A specific example in which different privacy processing is performed in the above-mentioned head and at least a part of the body region other than the head (at least a part thereof) is described below. As an example of privacy processing on the area of the head (or part of the head such as the face or hairstyle), it may be difficult to identify individuals by powerfully mosaicing, blurring, or painting, or only the head may be a cat or It may be transformed into the head or vehicle of an animal such as a dog to give a favorable impression to the viewers of women and children. On the other hand, as an example of privacy processing to be performed on (at least a part of) a body region other than the head, processing such as mosaic, blurring, and painting is performed lightly, and the color and shape of the clothing part and hat are changed, It may be difficult to identify the organization.

  As described above, the most suitable privacy processing method applied to the head (or face or hairstyle) to make it difficult to identify individuals, and the body other than the head to make it difficult to identify the organization or group to which the person belongs. The optimal privacy processing method applied to (part of) is often different. Therefore, in the image to be transmitted or displayed shown in FIG. 7E, the privacy processing method for the head (or face or hairstyle) and the body (part of the body) other than the head is individually optimized. As a result, it is difficult to identify individuals and organizations / organizations at the same time, and the effect of enhancing the privacy protection function is produced.

  The image and video display method in the above-described embodiment will be described again collectively. The photographed original image 30 (FIG. 2) and transmitted / displayed image examples 68 and 78 (FIG. 6 (e) and FIG. 7 (e)) are both unmanned image areas 34, 66, and 76 that do not include privacy information. The area can be divided (separated) into areas 38, 64, and 74 related to privacy information. Then, the image examples 68 and 78 after privacy processing in the areas 38, 64 and 74 related to the privacy information are transmitted / displayed.

  Here, with reference to the unmanned image area 32 (FIG. 3C) generated in advance, an area 38 related to privacy information in the captured original image 30 is extracted. As a result, there is an effect that privacy processing is stably performed even on an image in which only a part of a person's eyelid is reflected. Looking at the above description from another angle, the image examples 68 and 78 to be transmitted / displayed in the present embodiment include unmanned image areas 66 and 76 that do not include privacy information and image areas after privacy processing. It can be said. In addition, it can be said that privacy information is included in the image area after the privacy processing, and a part of a person's eyelid can be included as the privacy information.

  FIG. 8 shows the structure of an imaging system that implements the display methods shown in FIGS. The original image 30 captured by the imaging unit (camera) 112 is processed and generated in the image processing unit 104 and then transmitted from the image transmission unit 120 to the display unit 122. The image processing unit 104 includes an unmanned image generation unit 130, a privacy processing unit 116, and an image composition unit 118.

  In the unmanned image generation unit 130, an unmanned image 32 (FIG. 3C) that does not include privacy information is generated using the captured original image 30. On the other hand, the privacy processing unit 116 performs privacy processing on the image existing in the area 38 (FIG. 2) related to the privacy information in the photographed original image 30. Further, the image synthesizing unit 118 synthesizes the privacy-processed image 40 (FIG. 4B (c)) generated by the privacy processing unit 116 and the unmanned image 32.

  A method for generating the unmanned image 32 in the photographing system of FIG. 8 will be described with reference to FIG. The original image 30 obtained at the shooting site is shot by the imaging unit (camera) 112. The original image 30 is transferred to the image storage unit 132 via the image / image acquisition unit 114 and then stored in the image history storage unit 134.

  The unmanned image generation unit 130 compares the original image 30 (FIG. 3A) stored in the image history storage unit 134 with the original image 36 (FIG. 3B) taken after that. Then, a region hidden behind the persons 42 to 48 of the original image 30 in the unmanned image 32 (FIG. 3C) is extracted. As described above, the images used in the unmanned image generation unit 130 are not only the two images shown in FIGS. 3A and 3B, but also a plurality of images taken before and after that time. The unmanned image 32 can be generated using the image. If a part of the unmanned image 32 cannot be generated due to a moving person who moves slowly, an automatic interpolation process is performed.

  As described above, the unmanned image generation unit 130 may simultaneously generate the unmanned image (1) 32-1 and the unmanned image (2) 32-2 having different creation methods. As a specific example, the unmanned image (1) 32-1 transferred to the image compositing unit 118 is acquired by photographing the scene in a busy time zone (such as at midnight or before dawn). On the other hand, the unmanned image (2) 32-2 used in the head range extraction unit 152, the body range extraction unit 154, and the person probability distribution control unit 160 is an original image 30, 36 ( (See the description in FIG. 3). Thus, a plurality of unmanned images (1) 32-1 and unmanned images (2) 32-2 having different creation methods are created at the same time, and the optimum unmanned images (1) 32-1 and unmanned images ( 2) By transmitting 32-2, the effect of transmitting / displaying an image with very high accuracy and expressiveness is produced.

  That is, in the image 68 to be transmitted / displayed, it is strongly prohibited to display a clear person from the viewpoint of privacy protection. In addition, the photographer can confirm a situation in which no person appears in the unmanned image (1) 32-1 at the stage of photographing the site in a time zone where there is no traffic and obtaining the unmanned image (1) 32-1. Accordingly, it is possible to guarantee a situation in which an image related to privacy information is not included in the image 68 to be transmitted / displayed created using the unmanned image (1) 32-1.

  On the other hand, when the difference in the shooting time between the unmanned image (1) 32-1 acquired in the time zone without traffic and the original image 30 (FIG. 4A (a)) is severe, it occurred due to the difference in the shooting time. The extraction accuracy of the area 38 (FIG. 4C) related to the privacy information is lowered due to an unnecessary shadow or the like. Therefore, the shooting time of the unmanned image (2) 32-2 used by the head range extraction unit 152, the body range extraction unit 154, and the human probability distribution control unit 160 is related to privacy information when it is closer to the shooting time of the original image 30. The extraction accuracy of the region 38 to be improved improves.

  With reference to the unmanned image (2) 32-1, the region 38 related to the privacy information in the original image 30 transferred from the imaging unit 112 output from the unmanned image generation unit 130 via the image acquisition unit 114 is displayed. Extract (refer to the description using FIG. 4A).

  Further, in the case of the application example of FIG. 6, the individuality (person probability) is evaluated for each area in the area 38 related to privacy information, and the person-like area (person probability is high) and the person-like area (person probability is high). Change the privacy processing method between the (low) areas. A person probability necessary for this is calculated in the person probability distribution measurement unit 160.

  In the person probability distribution measurement unit 160, the person probability is calculated by using the shape of each region in the region 38 related to privacy information and its time change (based on the shape change). That is, the person in an upright state is arranged in the order of the head at the top, then the chest, belly, and feet. Accordingly, the degree of coincidence is calculated by using a pattern matching method for the outer shape of each region in the region 38 related to the human body shape and privacy information. That is, it is considered that the human probability is high in the area having a shape of the human body. On the other hand, the person moves by moving both feet and moves by moving both arms. Therefore, it is considered that the person probability is high in the area where the change state according to the time change of the contour shape of the area 38 related to the privacy information matches the movement of the foot or the movement of the hand. On the other hand, a region that moves without changing its shape is regarded as a personal property such as a bag or a car, and the value of the human probability is reduced.

  Privacy processing methods include blurring, mosaic, painting, shape changes to other creatures, shape changes that change the aspect ratio, shape changes, color changes, brightness changes, etc. The processing of the area related to information is not one type, and a plurality of types of privacy processing methods can be used. Then, as described with reference to FIG. 4C, the image 40 privacy-processed by the image combining unit 118 and the unmanned image 32 are combined. Prior to this image composition, the person probability distribution measuring unit 160 calculates the person probability for each area in the area 38 related to the privacy information. Based on the person probability value calculated for each area, the optimum image is selected from the plurality of types of privacy-processed images 40 and synthesized. As a result, a privacy-processed image (FIG. 6 (e)) that differs depending on the character of the person is displayed.

  The head range detection unit 152 extracts a characteristic shape in the region 38 related to the extracted privacy information. A person's head is composed of a face area and a hairstyle area. In the face area, eyes, nose, mouth, and eyebrows (and ears) are arranged at predetermined positions while forming unique shapes. By extracting the arrangement status of each of these areas using a pattern matching method or the like, the range of the head in the area 38 related to the privacy information is detected.

  For example, it is difficult to analyze the positional relationship between the eyes, nose, mouth, and eyebrows (and ears) in the face of a small person such as the moving person 44 in FIG. In such a case, it may be estimated that a vertically elongated human body shape is extracted and the upper part corresponds to the head.

  Similarly, the body range detection unit 154 extracts a characteristic shape in the region 38 related to the extracted privacy information. And a human body area | region is estimated by the pattern matching process etc. with the human body shape prepared previously.

  In this manner, the head estimated (detected) by the head range detection unit 152 and the body range detection unit 154 and a body region other than the head are extracted. Then, using the extraction result, different privacy processing is performed between the head and the body region other than the head (FIG. 7E).

  The image 58 representing the movement of the person shown in FIG. 5F is created using the image history storage unit 134, the movement information extraction unit 136, the movement information history storage unit 138, and the movement information aggregation unit 140. The images accumulated from the image acquisition unit 114 via the image accumulation unit 132 are sequentially stored in the image history storage unit 134.

  An example of the management information related to the image history stored in the image history storage unit 134 is shown in FIG. Individual images are stored in the image column in FIG. In addition, identification information (camera ID) of the imaging unit (camera) 112 that has captured the image and a shooting time are recorded as part of the management information.

The movement information extraction unit 136 compares the image acquired in the past obtained from the image history storage unit 134 with the original image 82 currently acquired. The difference information obtained there is sequentially stored in the movement information history storage unit 138 as movement information 88 (FIG. 5B).

  An example of management information relating to the history of the movement information 88 stored in the movement information history storage unit 138 is shown in FIG. The identification information (camera ID) of the imaging unit (camera) 112 that captured the original image 82 that is the basis of the movement information 88 is described. Further, the time information at which the movement information 88 is calculated in the movement information extraction unit 136 is recorded as the measurement time.

  As shown in FIG. 5B, the movement information 88 is managed as displacement vector information. That is, in the difference information (movement information 88) obtained as a result of comparing the image acquired in the past with the original image 82 currently acquired, the pixel position where the difference has occurred is set as “region coordinates in the image” (starting point coordinates of the displacement vector). Managed. The movement amount is represented by the length of the displacement vector, and the movement direction is represented by the direction of the displacement vector. Therefore, the coordinates representing the end point (tip) position of the displacement vector are recorded as the coordinates of the “movement vector”.

  Then, the movement information 88 stored in the movement information history storage unit 138 is totaled in the movement information totaling unit 140, and based on the result, the privacy processing unit 116 expresses the movement of the person and privacy-processed image. 52 (FIG. 5C) is generated.

  The images 58, 68, and 78 to be transmitted / displayed obtained as a result of the series of processing are transferred from the image transmission unit 120 to the display unit (personal computer, smartphone, electronic bulletin board, etc.) 122 and displayed on the display unit. .

  The photographing system 102 according to the present embodiment has a data processing / database management unit 106 therein, and data storage / statistical processing related to privacy information (and personal traffic) can be performed in parallel. As described above, the images 58, 68, and 78 transmitted / displayed on the display unit 122 are displayed so that they can be viewed by an unspecified number of people, and the original image 30 captured by the imaging unit (camera) 112 is used at the same time. Data processing / database management. As described above, since the original image 30 captured by the imaging unit (camera) 112 can be used for a plurality of purposes, there is an effect of improving the use efficiency of the original image 30.

  Specifically, the number of people reflected in the photographed original image 30 is estimated in the number of people counting unit 162 using the person probability for each region in the image calculated by the person probability distribution measuring unit 160. The result is sequentially stored in the congestion history storage unit 164 and statistically processed in the congestion status statistical processing system 168, so that the temporal change of the congestion status in the photographing site can be known.

  An example of management information relating to the congestion history stored in the congestion history storage unit 164 is shown in FIG. The identification information of the imaging unit (camera) 112 that captured the original image 82 that is the basis of the congestion history information is described as the camera ID. Also, the calculation date / time information of the congestion history information obtained by estimation in the number of persons counting unit 162 is recorded as the measurement time. Similarly to the area coordinates in the image in FIG. 14, a particularly congested location in the original image 82 is recorded as coordinates, and the congestion status at that location is recorded as a congestion value. Here, the congestion value represents the number of people in the target area specified by coordinates, the population density, and the like.

  In FIG. 8, the imaging unit (camera) 112 and the image processing unit 104 are drawn close to each other. However, the present invention is not limited thereto, and as shown in FIG. 9, the image pickup device (camera) 252 and the computer 256 that generates an image to be transmitted / displayed may be arranged at a distance.

  The imaging device (camera) 252 includes an imaging unit 264, a semiconductor memory unit 266 that temporarily stores captured images, and a communication unit 262. The communication unit 262 is connected to the Internet 200 through a wired or wireless line, and the original image 30 captured by the imaging unit 264 is transmitted to the communication unit 272 in the computer 256 via the wired or wireless line via the Internet 200. . A dedicated network line may be used for transmitting the original image 30 between the image pickup device (camera) 252 and the computer 256 instead of using the line of the Internet 200.

  The computer 256 that generates the image 68 to be transmitted / displayed is not limited to a large computer but may be a medium-sized general-purpose computer or a personal computer. The original image 30 transferred from the image pickup device (camera) 252 is temporarily stored in the memory unit 276. The original image 30 displayed on the display unit 274 from the memory unit 276 generates an image 68 to be transmitted / displayed by operating the input unit 278. Here, a series of these processes is performed by the control unit 270. That is, the processing in the image processing unit 104 and the data processing / database management unit 106 in FIG. 8 is advanced according to a program executed by the control unit 270.

  The image 68 to be transmitted / displayed created by the processing in the computer 256 is stored in the database 226 in the web server 202 via the Internet 200. The portable terminal 250 used for browsing by an unspecified majority person browses the image 68 transmitted / displayed from the web server 202 via the Internet 200. The access destination for browsing the image 68 transmitted / displayed by the portable terminal 250 is not limited to the above, and may be the database server 204 having a predetermined database 228.

  In the present embodiment, not only the image 68 to be transmitted / displayed but also the unmanned image 32 may be transmitted in advance to the database 226 in the web server 202 or the database 228 in the database server 204. When the user operates the display switching designation area 288 displayed in the display unit 284 of the portable terminal 250, the image displayed on the display unit 284 can be switched to the unmanned image 32. Some of the users who use the portable terminal 250 are interested in the congestion situation at the shooting site, and browse the image of the congestion situation with privacy protection. However, depending on the user who uses the portable terminal 250, there are cases where the user is interested in the unmanned image 32 displaying the background scenery. Thus, by transmitting the unmanned image 32 together with the image 68 to be transmitted / displayed in advance, there is an effect that various images corresponding to various users can be displayed.

  As another application example of the present embodiment, as shown in FIG. 10, a control unit 240 and an imaging unit (camera unit) 246 that execute processing corresponding to the image processing unit 104 and the data processing / database management unit 106 of FIG. May be built in the portable terminal 206 together. In this case, the original image 30 may be captured and the image 68 to be transmitted / displayed may be generated in the same portable terminal 206.

  In this case, the graphics controller 248 may have a part of the functions in the image processing unit 104 in FIG. However, most of the image processing unit 104 is executed by a program (software) executed in the control unit 240 (and the graphics controller 248).

  In the portable terminal 206 of FIG. 10, the imaging unit (camera unit) 246 related to high-speed image processing, the graphics controller 248, the semiconductor memory unit 236, and the control unit 240 are connected by a north bridge 242 for high-speed transfer. On the other hand, the south bridge 244 is used for connection at a portion that allows data transfer at a relatively low speed such as the display unit 234 and the communication unit 232.

  When viewing the image 68 to be transmitted / displayed or the unmanned image 32 created in the portable terminal 206, the portable terminal 210 in which the communication unit 222 and the display unit 224 are built may be used as in FIG. . However, the present invention is not limited thereto, and may be displayed on the fixed stationary Internet connection television 212, the personal computer 214, or the electronic bulletin board 216 via the router 208.

  FIG. 11 shows a basic architecture of a program (software) executed in the control unit 270 of FIG. 9 and the control unit 240 (or graphics controller 248) of FIG.

  The mobile terminal control main program (class) 308 performs shooting control using the imaging unit (camera) 112 in FIG. 8 and external communication control in the image transmission unit 120. Here, when the program is executed in the control unit 270 built in the computer 256 of FIG. 9, a main program (class) corresponding to the main program (class) 308 for controlling the mobile terminal is used.

  A camera control subprogram / method (camera controller) 316 located in a lower program (subprogram or method) in the main program (class) 308 for mobile terminal control controls the imaging unit (camera unit) 246.

  Further, when the original image 30 photographed here is temporarily stored in the semiconductor memory unit 236, a memory control subprogram / method (memory controller) 318 is used. In the imaging system application example already described with reference to FIG. 10, the captured original image 30 is temporarily stored in the semiconductor memory unit 236 in the portable terminal 206. On the other hand, the temporary storage destination of the original image 30 captured in the imaging system of FIG. 9 is the memory unit 276 in the computer 256. Therefore, in this case, the memory control subprogram / method (memory controller) 318 and a dedicated memory memory control subprogram / method (dedicated memory controller) described later control (drive) the memory unit 276 in the computer 256. .

  The functions in the image processing unit 104 and the data processing / database management unit 106 in FIG. 8 are handled by the privacy protection-compatible image editing subprogram / method 300. That is, the above-mentioned privacy protection-compatible image editing is performed by using a dedicated API (Application Programming Interface) 304 from the privacy protection-compatible control subprogram / method (privacy protection commander) 314 to be called in the mobile terminal control main program (class) 308. A predetermined processing instruction is issued to the subprogram / method 300.

  On the other hand, in the privacy protection compliant image editing subprogram / method 300, an interpreter subprogram / method (interpreter) 324 receives a command using the dedicated API 304 and interprets the contents of the instruction command. Based on the result, the privacy protection compatible integrated program / class (integrator) 330 calls each subprogram / method set in the privacy protection compatible image editing subprogram / method 300 and follows the above instructions. Perform execution processing.

  Here, the information stored in the image history storage unit 134, the movement information history storage unit 138, and the congestion history storage unit 164 of FIG. 8 is stored in different areas in the semiconductor memory unit 236. As described above, in this embodiment, since the semiconductor memory unit 236 is frequently accessed, the memory memory control subprogram / method (dedicated memory controller) 328 responsible for the access control to the dedicated semiconductor memory unit 236 is assigned to each subprogram. / Located directly under the method. This has the effect of speeding up the access control to the semiconductor memory unit 236 within each subprogram / method.

  11 is responsible for the functions of the image storage unit 132, the image history storage unit 134, and the unmanned image generation unit 130 in FIG. Here, the detailed execution flow in the unattended image creation subprogram / method (background generator) 332 corresponds to the flowchart shown on the right side in FIG.

  Further, the movement information extraction / management subprogram / method (moving part extractor) 334 in FIG. 11 includes an image history storage unit 134, a movement information extraction unit 136, a movement information history storage unit 138, and movement information in FIG. The function of the totaling unit 140 is assumed. The detailed execution flow in the movement information extraction / management subprogram / method (moving part extractor) 334 corresponds to the flowchart shown on the left side in FIG.

  Further, the transmission / display image generation subprogram / method (privacy protector) 336 in FIG. 11 includes a head range detection unit 152, a body range detection unit 154, a person probability distribution measurement unit 160, and a privacy processing unit 116 in FIG. Each function of the image composition unit 118 is assumed. The detailed execution flow in the transmission / display image generation subprogram / method (privacy protector) 336 corresponds to the flowchart shown in FIG.

  Then, the subprogram / method (data manager) 338 for controlling the accumulation / statistical processing of data related to the privacy information (or person traffic) in FIG. 11 performs the functions in the data processing / database management unit 106 in FIG. Bear. A detailed flowchart in the control subprogram / method (data manager) 338 for storing / stating statistical data related to this privacy information (or person traffic) shows a person for each pixel described on the left side in FIG. Step 38 for calculating the degree of traffic, Step 51 for calculating the number of people in the person range, and Step 52 for storing the congestion information correspond to each other.

  FIG. 12 shows a flowchart of a program executed in the control unit 270 of FIG. 9 or the control unit 240 of FIG. 10 (and a part of the graphics controller 248). When the photographing system is started in Step 1, image acquisition (Step 2) continues until the photographing system in Step 3 is completed.

  When an instruction to start generation of an unmanned image is received (Step 11), the image acquired in Step 2 is appropriately stored in the image history storage unit 134 (corresponding to Step 13 in FIG. 8) (Step 12). Thereafter, in Step 14, the image stored as appropriate is read from the image history storage unit 134 as appropriate, and when the generation of the unmanned image 32 is completed, the processing of Step 15 is completed.

  In Example 1, as an unattended image generation method at Step 14, an image taken in a time zone where there is no person, such as at night, is used. In another example 2, it may be generated by synthesizing a portion having no motion from an image at regular intervals.

  Next, when the movement information of a person is displayed in the image 58 to be transmitted / displayed shown in FIG. 5F, an instruction to start generation of movement information is received as shown in Step 21. Then, the image is compared with the past image, the motion in the image is calculated (Step 22), and the result is appropriately stored in the movement information history storage unit 138 (corresponding to Step 23 in FIG. 8). After that, at Step 24, the movement information for a predetermined time is totaled, and the movement information 88 obtained as a result is stored in the movement information history storage unit 138. When the generation process of the movement information is completed, the end process is performed (Step 25).

  In the first example, the movement information may be averaged as a specific method of counting the movement information for a certain time in Step 24 described above. In another example 2, the moving direction of the person may be learned for each place (area) in the image, and the result may be used.

  FIG. 13 shows a procedure for generating images 50, 58, 68, and 78 to be transmitted / displayed in the present embodiment. When an instruction to start the imaging system is received at Step 31, a series of loop processing up to Step 44 displayed on the receiving side immediately before the end of the imaging system at Step 45 is repeated at arbitrary intervals (Step 32).

  Immediately after the image acquisition (Step 33) in the above loop processing, the calculation of the degree that the body is reflected for each pixel (Step 34), the calculation that the head is reflected for each pixel (Step 35), The calculation of the degree of reflection (Step 38) is executed in parallel.

  In Example 1 showing the specific method of the above processing, the degree of being a person may be considered to be higher (the probability is higher) as the absolute value of the difference between the pixel values of the unmanned image and the current image is larger.

  In Example 2 showing another specific method, a plurality of images in which a person (head or body) is shown in advance are learned, and the range in which the current image is close to the learning result is shown as a person. It may be considered that the degree of (or head or body) is large (the probability is high).

  In yet another example 3, a plurality of methods such as example 1 and example 2 described above are performed in parallel, and the degree (probability) of a person in terms of the total value or average value of the obtained degrees (probabilities). May be determined.

  As described above with reference to FIG. 5, when it is desired to change the color of the body (or clothes or portable items) according to the moving speed of the person (Step 36), the movement information 88 generated in FIG. 12 and Step 34 are used. The calculation result of the degree that the body is reflected for each pixel is used.

  As a specific example 1 in which the color of the body (or clothing or portable items) is changed according to the moving speed of the person in Step 36, the moving person (the whole or a part of the body such as the clothing part) is grayscale. And the density (degree of black) may be changed according to the moving speed.

  As another example 2, the color may be changed according to a moving person and a stopped person.

  Further, as described above with reference to FIG. 7, when “blurring” is applied only to the face area (face area) (privacy processing) or “blur” is applied to the entire image (privacy processing) (Step 37), The movement information 88 generated in FIG. 12 and the calculation result of the degree that the head is reflected for each pixel obtained in Step 35 are used.

  Further, as described in FIG. 5, as another method for displaying the moving direction of the person, Step 37 may apply “blur” in accordance with the moving direction.

  Then, the privacy-processed video image obtained in Step 37 and the unmanned image 32 obtained in FIG. 12 are synthesized in Step 39. Further, as described above with reference to FIG. 6, when different privacy processing is performed for each region having different human probabilities in the image, as described in Step 41, “blurring processing” has been completed according to the degree (probability) of being a person. The pixel values of the image and the unmanned image 32 may be weighted and averaged to change the transparency in the area 38 related to privacy information.

  Thereafter, an image to be transmitted / displayed is generated (Step 42), and the generated image is transmitted (Step 43).

  An example of data accumulation / processing (or database management and statistical processing using the data) such as taking statistics of temporal changes in the congestion situation at the shooting site, for example, using the captured original image 30 will be described below.

  For the original image 30 acquired in Step 33, in Step 38, the degree (person probability) in which a person is shown is calculated for each pixel. Then, it is determined (estimated) that the person is reflected in the original image 30 in a portion where the calculated degree (person probability) of the person exceeds a predetermined determination value (for example, 50% or 70%).

  Next, the pixels determined that the person is shown are totaled, and the number of people shown in the original image 30 is calculated (Step 51). In general, the same person often appears across a plurality of pixels. Accordingly, the continuity between pixels whose degree of human appearance (person probability) exceeds a predetermined determination value (for example, 50% or 70%) is determined to estimate the segmentation area between a plurality of persons, and within the original image 30 Predict the number of people.

  The number of persons (number of people) shown in the same original image 30 obtained in Step 51 is sequentially stored in the congestion history storage unit 164 (Step 52). The information stored in the congestion history storage unit 164 records a congestion value for each number of people calculation time (measurement time) as already described with reference to FIG.

  Thereafter, the congestion status statistical processing system 168 in FIG. 8 may statistically process the congestion history information (FIG. 16) in the congestion history storage unit 164, and analyze the congestion situation at the photographing site and the time change situation with respect to the congestion location.

  In the description of the above-described embodiment, the privacy processing target is described as if it is all persons. However, the above-described system can provide a specific person as a clear image without processing privacy for the specific person as necessary. For example, when the imaging signal is used for an emergency report or used for news in the field, the image of the reporter or the announcer may be clear. When such a necessity arises, for example, the original image 30 output from the image acquisition unit 114 shown in FIG. 8 is input to the display unit 274 of the personal computer 256 and is confirmed by the operator. The operator designates, for example, the face or the entire body of an image of a specific person on the display unit 274, for example. As a designation method, for example, a frame is drawn with a mouse pointer that is a part of the input unit 278, and for example, a face or the whole body of an image of a specific person is surrounded, and a determination operation is performed. As a result, when the head range detection unit 152 and / or the body range detection unit 154 detects a specific person (preferably performing face recognition processing), the privacy processing unit does not process the privacy for the person. Control information is transmitted to 116. The specific person detection method is not limited to the method described above, and various methods are possible. For example, a specific person may wear a specific easily-identifiable costume or hat, and the head range detection unit 152 and / or body range detection unit 154 may detect the costume or hat.

  Although several embodiments have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

2, 4, 6 Person in original image taken 10 Example of original image 12, 14 Region indicating person range detected in photographed original image 22 Person range detection 24 Blur processing to detection range 30 Original image 32 Uninhabited Image 34 Unattended image region 36 Original image taken a little later 38 Region related to privacy information 40 Privacy processed image 42, 44, 46, 48 Moving person 50 Image to be transmitted / displayed 52, 62, 72 Privacy processing Example 54, 64, 74 Example of region related to privacy information 56, 66, 76 Example of unmanned image 58 Example of transmission / display image 68, 78 Example of image to be transmitted / displayed 82 Example of original image 84, 86 Example of original image 88 Example of movement information 102 Imaging System 104 Image Processing Unit 106 Data Processing / Database Management Unit 112 Imaging Unit (Camera)
114 Image acquisition unit 116 Privacy processing unit 118 Image composition unit 120 Image transmission unit 122 Display unit (PC, smartphone, electronic bulletin board, etc.)
DESCRIPTION OF SYMBOLS 130 Unattended image production | generation part 132 Image storage part 134 Image history preservation | save part 136 Movement information extraction part 138 Movement information history preservation | save part 140 Movement information total part 152 Head range detection part 154 Body range detection part 160 Person probability distribution measurement part 162 Person totalization Unit 164 congestion history storage unit 168 congestion status statistical processing system 200 internet 202 web server 204 database server 206, 210, 250 portable terminal 208 router 212 internet connection television 214 personal computer 216 electronic bulletin board 222, 232, 262, 272 communication unit 224, 234, 274, 284 Display unit 236, 266 Semiconductor memory unit 240, 270 Control unit 242 North bridge 244 South bridge 246 Imaging unit (camera unit)
248 Graphics Controller 252 Imager (Camera)
256 Computer 264 Imaging unit 274, 284 Display unit 276 Memory unit 278 Input unit 288 Display switching designation area 300 Privacy protection compatible image editing subprogram / method 304 Dedicated API
308 Main program (class) for mobile terminal control
314 Privacy protection control subprogram / method (privacy protection commander)
316 Camera Control Subprogram / Method (Camera Controller)
318 Memory control subprogram / method (memory controller)
324 interpreter subprogram / method (interpreter)
328 Dedicated memory memory control subprogram / method (dedicated memory controller)
330 Integrated Program / Class for Privacy Protection (Integrator)
332 Subprogram / method (background generator) for unattended image creation
334 Subprogram / method for moving information extraction / management (moving part extractor)
336 Subprogram / method for image generation for transmission / display (privacy protector)
338 A subprogram / method (data manager) for controlling accumulation / statistical processing of data related to privacy information (and personal traffic).

Claims (6)

An unmanned image generation unit that generates an unmanned image that does not include privacy information from the captured image or video;
A privacy processing unit that processes privacy regarding the area related to privacy information in the captured original image;
Consists of an image synthesis unit that synthesizes the image after privacy processing generated by the privacy processing unit and the unmanned image,
An imaging system for extracting an area related to the privacy information with reference to the unmanned image. Extract movement information in the image by comparing with past images,
A movement information totaling unit that calculates the movement direction by totaling the movement information for a certain period of time;
The imaging system according to claim 1, wherein the privacy processing is performed based on information obtained from the movement information aggregation unit. A head range detection unit for detecting a head range of a person from the area related to the privacy information;
A body range detection unit that detects the body range of a person from the area related to the privacy information;
The imaging system according to claim 1, wherein different privacy processing is performed in at least a part of the body range other than the head range and the head range. A privacy processing unit that extracts a region related to privacy information from the captured image or video, and processes the privacy in the region;
A head range detection unit for detecting a head in an area related to the privacy information;
A body range detection unit that detects a body region within a region related to the privacy information;
Photography in which privacy processing is performed in the privacy processing unit, which is different between at least a part of the body region other than the head based on the detection result of the head range detection unit and the detection result of the body range detection unit. system. At least a part of the original image captured by the imaging unit is
After being processed and generated in the image processing section,
It is a method for displaying images or videos sent from the image sending unit,
The image or video includes an unattended image area that does not include privacy information and an image area after privacy processing,
Privacy information is included in the image area after the privacy processing,
In addition, the privacy information may include an image or video display method that can include a part of a person's face. At least a part of the original image captured by the imaging unit is
After being processed and generated in the image processing section,
It is a method for displaying images or videos sent from the image sending unit,
The image or video includes an unattended image area that does not include privacy information and an area related to privacy information,
An image or video display method in which display based on different privacy processing is performed on at least a part of a person's head in an area related to the privacy information and a body area other than the head.

Images