KR20210019645A - Apparatus for monitoring passengers in autonomous vehicles - Google Patents

Abstract

Disclosed are an apparatus and a method for monitoring a passenger in an autonomous vehicle, which are able to monitor the status of the passenger in a shuttle-type autonomous vehicle, and to, when a dangerous situation occurs or is expected, notify such fact to the outside. The apparatus comprises: a camera which photographs the inside of the autonomous vehicle in real time; a passenger and article processing unit which recognizes and tracks one or more of the passenger and articles from an image from the camera; a posture and face direction estimation unit which estimates the posture and face direction of the passenger based on the results of processing by the passenger and article processing unit; a status determination unit which determines whether the passenger is in danger or not based on output from the posture and face direction estimation unit; and a notification unit which, when the passenger is determined to be in danger by the status determination unit, generates a corresponding notification message, and transmits the message to an external center.

Description

Apparatus for monitoring passengers in autonomous vehicles

The present invention relates to an apparatus for monitoring occupants in an autonomous vehicle, and more particularly, to an apparatus for monitoring occupants in an autonomous vehicle that monitors a state of an occupant in an autonomous vehicle in the form of a shuttle.

A vehicle refers to a means of transportation that can move people or luggage by using kinetic energy.

For the safety and convenience of users who use the vehicle, various sensors and devices are provided in the vehicle, and functions of the vehicle are diversified.

The vehicle functions may be divided into a convenience function for promoting the driver's convenience, and a safety function for promoting the safety of the driver and/or pedestrian.

First, the convenience function has a development motivation related to the driver's convenience, such as giving the vehicle an infotainment (information + entertainment) function, supporting a partial autonomous driving function, or helping to secure the driver's vision such as night vision or blind spots. . For example, adaptive cruise control (ACC), smart parking assist system (SPAS), night vision (NV), head up display (HUD), and around view monitor (around view monitor, AVM) and adaptive headlight system (AHS) functions.

Safety functions are technologies that ensure the safety of drivers and/or pedestrians, such as lane departure warning system (LDWS), lane keeping assist system (LKAS), and automatic emergency braking (autonomous emergency). braking, AEB) functions, etc.

In order to support and increase the function of the vehicle, it may be considered to improve the structural part and/or the software part of the vehicle control device.

Thanks to these improvements, autonomous vehicles are being developed that can automatically drive to their destination without driver intervention.

Autonomous driving is defined as that at least one of acceleration, deceleration, and driving direction is controlled by a preset algorithm even if the driving operation device is not operated by the driver.

If autonomous vehicles are commercialized, drivers can use the time spent driving for other tasks. For example, you can read a book, watch a video, or sleep.

In autonomous vehicles, the driving and stopping of the vehicle is determined by software, not by humans. Accordingly, there is a need to develop an autonomous vehicle capable of automatically performing many tasks previously performed by a driver.

Accordingly, various algorithms related to autonomous driving are being developed. For example, an algorithm that determines the possibility of a collision with an object outside the vehicle and avoids a collision, an algorithm that recognizes a signal and performs an action corresponding to the signal, adjusts the distance between the vehicle in front and the vehicle behind, and adjusts the speed. Algorithms and the like are being developed.

As described above, in the related art, research on autonomous vehicles capable of automatically responding to various situations occurring outside the vehicle has been active.

However, research on autonomous vehicles taking into account the state of the occupants in the vehicle is still insufficient.

Prior Art 1: Korean Patent Application Publication No. 10-2018-0053081 (autonomous driving vehicle and its control method) Prior Art 2: Republic of Korea Patent Publication No. 10-2014-0020230 (System and method for predicting the behavior of the detected object) Prior Art 3: Korean Patent Laid-Open Publication No. 10-2019-0078553 (Vehicle control method and intelligent computing device for controlling the vehicle)

The present invention has been proposed in consideration of the above-described conventional circumstances, and is to notify the outside when a dangerous situation occurs or a dangerous situation is expected by monitoring the status of the occupant in a shuttle-type autonomous vehicle. It is an object of the present invention to provide an occupant monitoring device in an autonomous vehicle.

In order to achieve the above object, an apparatus for monitoring occupants in an autonomous vehicle according to a preferred embodiment of the present invention includes: a camera for photographing the interior of the autonomous vehicle in real time; An occupant and object processing unit for recognizing and tracking at least one of an occupant and an object in the input image from the camera; A posture and face direction estimating unit for estimating a posture and face direction of the occupant based on a result of processing by the occupant and object processing unit; A state determination unit determining whether the occupant is in danger based on the output from the posture and face direction estimation unit; And a notification unit that generates a notification message corresponding to the status determination unit as it is determined as a dangerous occupant and sends the notification message to the external center.

In recognizing one or more of the occupant and the object, the occupant and object processing unit recognizes the occupant in the input image from the camera and identifies the gender of the recognized occupant, and if there is an object carried by the occupant, the corresponding It is possible to recognize an object and perform tagging between the occupant and the object.

The posture and face direction estimation unit detects landmarks of the body and face of the occupant recognized by the occupant and object processing unit, respectively, and tracks the detected landmarks of the body and face of the corresponding occupant, respectively, and Face orientation can be estimated.

The state determination unit determines the current state of the corresponding occupant based on the output from the posture and face direction estimation unit, tracks the current state of the corresponding occupant in real time to determine the change in behavior of the occupant, and Based on the identification of behavioral changes, it is possible to determine whether the occupant is at risk.

According to the present invention with this configuration, when a problem occurs between passengers in a shuttle-type autonomous vehicle or a passenger who is expected to be dangerous is found, a corresponding notification message can be transmitted to the external center. Accidents can be quickly responded to or can be prevented from occurring.

1 is a block diagram showing the configuration of an occupant monitoring apparatus in an autonomous vehicle according to an embodiment of the present invention.
2 is a diagram showing an example of an image captured by the camera shown in FIG. 1.
3 is a view showing an example of a landmark of a body of a passenger processed by the posture and face direction estimation unit shown in FIG. 1.
4 is a diagram illustrating an example of a landmark of a passenger's face processed by the posture and face direction estimation unit shown in FIG. 1.
5 is a basic example of body and face landmarks processed by the posture and face direction estimation unit shown in FIG. 1.
6 and 7 are flowcharts illustrating a method for monitoring occupants in an autonomous vehicle according to an embodiment of the present invention.

In the present invention, various modifications may be made and various embodiments may be provided, and specific embodiments will be illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to a specific embodiment, it is to be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. In describing the present invention, in order to facilitate an overall understanding, the same reference numerals are used for the same elements in the drawings, and duplicate descriptions for the same elements are omitted.

1 is a block diagram showing the configuration of an occupant monitoring device in an autonomous vehicle according to an embodiment of the present invention, FIG. 2 is a view showing an example of an image captured by the camera shown in FIG. 1, and FIG. 3 is 1 is a view showing an example of a landmark of the occupant's body processed by the posture and face direction estimating unit shown in FIG. 1, and FIG. 4 is a landmark of the occupant's face processed by the posture and facial direction estimating unit shown in FIG. 1 It is a diagram showing an example, and FIG. 5 is a basic example of body and face landmarks processed by the posture and face direction estimation unit shown in FIG. 1.

The occupant monitoring device in an autonomous vehicle according to an embodiment of the present invention includes a camera 10, a passenger and object processing unit 12, a posture and face direction estimation unit 14, a state determination unit 16, and a notification unit 18. ), the learning unit 20, and the control unit 22 may be included.

The camera 10 may be installed on the ceiling of an autonomous vehicle (eg, a shuttle) to capture the interior of the autonomous vehicle in real time. For example, the camera 10 may take a picture of the interior of the autonomous vehicle as illustrated in FIG. 2.

The camera 10 has a visual angle of about 110 to 130 degrees (more preferably about 120 degrees) in the horizontal direction, and about 70 to 80 degrees (more preferably about 73 degrees) in the vertical direction. It is a wide-angle camera with an angle) (also called an angle of view).

In addition, the camera 10 may have a tilting angle of approximately 70 degrees in order to better capture occupants in the autonomous vehicle.

The occupant and object processing unit 12 processes the recognition and tracking of occupants and objects in the input image from the camera 10. In recognizing the occupant and the object, the occupant and object processing unit 12 may recognize the occupant and the object by using the first learning model of the learning unit 20 to be described later.

That is, the occupant and object processing unit 12 recognizes the occupant from the input image from the camera 10, identifies the gender of the recognized occupant, and identifies the object carried by the occupant (e.g., bag, cart, umbrella, mobile phone, gun). , Knife, etc.)

Here, the gender of the passenger will be divided into male and female, but more specifically, it may be subdivided into children, adolescents, and adults.

Recognition of the occupant and the object in the occupant and object handling unit 12 described above can be sufficiently understood by a well-known technology for those who are engaged in the same industry, and thus a detailed description thereof will be omitted.

In addition, the occupant and object processing unit 12 may tag objects recognized for each occupant. Here, tagging can be understood as matching passengers and objects. Of course, tagging would not be necessary if the occupants had no items (e.g. bags, carts, umbrellas, cell phones, guns, knives, etc.).

Further, the occupant and object processing unit 12 may track the occupant and the object, respectively, and may update the tracking of the occupant and the object.

The posture and face direction estimating unit 14 may detect landmarks of the passenger's body and face recognized by the occupant and object processing unit 12, respectively, and track the detected landmarks of the body and face of the corresponding occupant. .

For example, the posture and face direction estimation unit 14 may detect the landmark of the occupant's body in 3D as illustrated in FIG. 3, and the landmark of the occupant's face as illustrated in FIG. 4. Can be detected.

The posture and face direction estimating unit 14 may detect landmarks of the body and face of the occupant in 3D using the second learning model of the learning unit 20 to be described later.

The posture and face direction estimation unit 14 detects landmarks of the occupant's body and face in 3D. In the case of the front of the occupant's face, the four feature points, that is, both ears and both ears, as illustrated in (a) of FIG. The nose and chin feature points may be determined and detected as landmarks for a corresponding occupant's face. Meanwhile, in the case of the side of the passenger's face, three feature points, that is, the feature points of one ear, nose, and chin, as illustrated in FIG. 5B may be determined and detected as landmarks for the face of the passenger. Meanwhile, in the case of the occupant's body, 13 characteristic points may be determined and detected as landmarks for the occupant's body as illustrated in FIG. 5C. In particular, when detecting a landmark on the body of the occupant, in the related art, both knees were connected from the torso to each of the knees with curved connecting lines, but in the embodiment of the present invention, both knees from the torso were connected. ) Are connected with straight-line connecting lines.

On the other hand, the posture and face direction estimation unit 14, when tracking the landmarks of the body and face of the corresponding occupant, uses, for example, an Extended Kalman Filter (EKF) to determine the landmarks of the body and face, respectively. Track.

As the body and face landmarks are tracked using the extended Kalman filter as described above, the posture and face direction estimating unit 14 may estimate a pose and a head orientation of a corresponding occupant.

The state determination unit 16 determines and classifies the current state of a corresponding occupant based on the output from the posture and face direction estimation unit 14 (eg, a result of estimating the occupant's posture and face direction). For example, in the case of the passenger illustrated in FIG. 2, the state determination unit 16 may determine that an adult man is sitting and using a mobile phone. And, the status determination unit 16 for the passenger "passenger ID: 1", "gender: male, adult", "time: 16:47", "detected object: mobile phone", "state: sit down and use a mobile phone. It can be classified using a classification code that includes content such as ".

In addition, the state determination unit 16 may track the current state of the corresponding occupant in real time.

As the state determination unit 16 continuously tracks the current state of the occupant in real time, it is possible to grasp a change in the behavior of the occupant based on the tracking result.

For this reason, the state determination unit 16 may determine whether the occupant has a dangerous object or is performing a dangerous action (eg, fighting, arson, etc.), or whether a dangerous action has been taken.

Then, the state determination unit 16 transmits the determination result (that is, including information indicating that the occupant has a dangerous object or is performing a dangerous behavior or has performed a dangerous behavior) to the controller 22.

The notification unit 18 generates a predetermined notification message under the control of the control unit 22, which will be described later, and sends it to an external center (eg, customer center or 112, 119, etc.). Here, the notification message is, for example, "A fight has occurred between two passengers A and B as of $$ minutes at &&.", "Board A is holding a dangerous knife.", "Get off at && at @@ minutes. One passenger A got off with another passenger B's bag." It may be a voice and/or text message such as.

Meanwhile, when transmitting a voice and/or text message, the notification unit 18 may transmit an image of a corresponding passenger causing a problem.

The learning unit 20 provides a first learning model that enables the occupant and object processing unit 12 to recognize occupants and objects, and the posture and face direction estimation unit 14 identifies landmarks of the occupant's body and face. It provides a second learning model that enables detection of each.

The learning unit 20 may train the first learning model and the second learning model based on data from the passenger and object processing unit 12 and the posture and face direction estimation unit 14.

The control unit 22 controls the overall operation of the occupant monitoring device in the autonomous vehicle according to the embodiment of the present invention.

In particular, if a dangerous condition occurs in the autonomous vehicle or a passenger who is expected to be dangerous is found on the basis of the determination result of the state determination unit 16, the control unit 22 communicates with the external center (eg, Control so that a corresponding notification message is transmitted to the customer center or 112, 119, etc.).

That is, the control unit 22 can sufficiently grasp how the passengers in the autonomous vehicle are behaving based on the determination result of the state determination unit 16, and thus, whether a dangerous state has occurred in the autonomous vehicle or is expected to be dangerous. You can know who is.

In FIG. 1 described above, the state determination unit 16 and the control unit 22 are configured separately, but the state determination unit 16 and the control unit 22 may be viewed as integral modules.

6 and 7 are flowcharts illustrating a method for monitoring an occupant in an autonomous vehicle according to an embodiment of the present invention.

First, the camera 10 installed on the ceiling of the autonomous vehicle captures the interior of the autonomous vehicle in real time (S10).

Subsequently, the occupant and object processing unit 12 processes the recognition, tagging, and tracking of the occupant and object in the input image from the camera 10. That is, the occupant and object processing unit 12 recognizes the occupant from the input image from the camera 10 and distinguishes the recognized gender of the occupant (S12). Then, the occupant and object handling unit 12 recognizes the object (eg, bag, cart, umbrella, mobile phone, gun, knife, etc.) carried by the occupant and tags the recognized object for each occupant (S14). ). Of course, you won't need tagging for passengers who don't have things they have (e.g. bags, carts, umbrellas, cell phones, guns, knives, etc.). Next, the occupant and object processing unit 12 tracks the occupant and the object, respectively, and updates the tracking of the occupant and the object (S16).

Thereafter, the posture and face direction estimation unit 14 detects landmarks of the body and face of the occupant recognized by the occupant and object processing unit 12 (S18).

Then, the posture and face direction estimation unit 14 tracks the detected landmarks of the body and face of the corresponding occupant, respectively. As such, by tracking the landmarks of the body and face of the corresponding occupant, the posture and face direction estimation unit 14 may estimate the pose and the head orientation of the corresponding occupant (S20). The posture and face direction estimating unit 14 sends the estimation result to the state determining unit 16.

Subsequently, the state determination unit 16 determines and classifies the current state of the corresponding occupant based on the output from the posture and face direction estimation unit 14 (eg, a result of estimating the occupant's posture and face direction) (S22).

Then, the state determination unit 16 continues to track the current state of the corresponding occupant in real time (S24). As such, if the current status of the occupant is continuously tracked in real time, the change in behavior of the occupant can be identified. For this reason, the state determination unit 16 may determine whether the occupant has a dangerous object or is performing a dangerous action (eg, fighting, arson, etc.), or whether a dangerous action has been taken. Then, the state determination unit 16 transmits the state determination result of the corresponding occupant to the control unit 22.

Thereafter, the controller 22 determines whether a dangerous state has occurred in the autonomous vehicle or a passenger who is expected to be dangerous is found based on the determination result of the state determination unit 16 (S26).

The determination result of the state determination unit 16 indicates that a dangerous condition (e.g., fighting, arson, etc.) has occurred in the autonomous vehicle or that a passenger (e.g., a passenger with a gun, knife, etc.) is expected to be dangerous. If it is ("Yes" in S26), the control unit 22 transmits a corresponding notification message to an external center (eg, customer center or 112, 119, etc.) through the notification unit 18 (S28). For example, when a fight occurs in an autonomous vehicle, the notification unit 18 generates a voice and/or text message stating "A fight has occurred between two passengers A and B as of $$ minute &&". After that, it is wirelessly transmitted to an external center (eg, customer center or 112, 119, etc.). For example, if one of the passengers is holding a knife, the notification unit 18 generates a voice and/or text message stating "Board A is holding a dangerous knife." To the center or 112, 119, etc.). For example, if passenger A gets off with another passenger B's bag, a voice and/or text message is generated that says, "Occupant A, who got off at && at @@ minutes, got off with another passenger B's bag." After that, it is wirelessly transmitted to an external center (eg, customer center or 112, 119, etc.). Of course, when the notification unit 18 transmits a voice and/or text message, it may also transmit an image of a corresponding occupant causing a problem.

In addition, the above-described method for monitoring occupants in an autonomous vehicle according to the present invention can be implemented as a computer-readable code on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tapes, floppy disks, and optical data storage devices. In addition, the computer-readable recording medium is distributed over a computer system connected through a network, so that computer-readable codes can be stored and executed in a distributed manner. In addition, functional programs, codes and code segments for implementing the method can be easily inferred by programmers in the art to which the present invention belongs.

As described above, an optimal embodiment has been disclosed in the drawings and specifications. Although specific terms have been used herein, these are only used for the purpose of describing the present invention, and are not used to limit the meaning or the scope of the present invention described in the claims. Therefore, those of ordinary skill in the art will understand that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical scope of the present invention should be determined by the technical spirit of the appended claims.

10: camera 12: passenger and object handling unit
14: posture and face direction estimation unit 16: state determination unit
18: notification unit 20: learning unit
22: control unit

Claims (4)

A camera that photographs the interior of the autonomous vehicle in real time;
An occupant and object processing unit for recognizing and tracking at least one of an occupant and an object in the input image from the camera;
A posture and face direction estimating unit for estimating a posture and face direction of the occupant based on a result of processing in the occupant and object processing unit;
A state determination unit determining whether the occupant is in danger based on the output from the posture and face direction estimation unit; And
And a notification unit that generates a notification message according to the determination as a dangerous occupant by the state determination unit and sends the notification message to the external center.
The method according to claim 1,
The passenger and object handling unit,
In recognizing at least one of the occupant and the object, the occupant is recognized in the input image from the camera, the gender of the recognized occupant is identified, and if there is an object carried by the occupant, the object is recognized, and the occupant An occupant monitoring device in an autonomous vehicle, characterized in that tagging between the and the object.
The method according to claim 1,
The posture and face direction estimation unit,
Each of the landmarks of the body and face of the occupant recognized by the occupant and object processing unit is detected, and the detected landmarks of the body and face of the occupant are respectively tracked to estimate the posture and face direction of the occupant. Occupant monitoring device in autonomous vehicle.
The method according to claim 1,
The state determination unit,
Based on the output from the posture and face direction estimation unit, the current state of the corresponding occupant is determined, the current state of the corresponding occupant is tracked in real time to determine the change in behavior of the corresponding occupant, and based on the identification of the behavior change of the corresponding occupant An occupant monitoring device in an autonomous vehicle, characterized in that it determines whether the corresponding occupant is dangerous.

Images