WO2024213062A1

WO2024213062A1 - Control method for autonomous mobile device, autonomous mobile device, and storage medium

Info

Publication number: WO2024213062A1
Application number: PCT/CN2024/087292
Authority: WO
Inventors: 于航; 汪圆圆; 吴俊杰; 郑思远; 潘勋; 张兆龙
Original assignee: 苏州宝时得电动工具有限公司
Priority date: 2023-04-11
Filing date: 2024-04-11
Publication date: 2024-10-17

Abstract

The present application discloses a control method for an autonomous mobile device, an autonomous mobile device, and a storage medium. The method comprises: obtaining the region type of each region in an acquired image, each region at least comprising: a region for traveling and a target region, wherein the region for traveling is a part of a working region, and the working region is a region adjacent to the region for traveling; and when the region type of the target region is a first region type, controlling an autonomous mobile device to change a traveling mode, such that the coverage range of a working unit comprises a part of the target region. In this way, the autonomous mobile device can process the boundary of the working region while the safety is guaranteed, the processing coverage rate at the boundary of the working region is improved, and the use experience of a user is improved.

Description

Self-equipping device control method, self-equipping device and storage medium

Technical Field

The present invention relates to the field of intelligent control technology, and in particular to a control method for a self-moving device, a self-moving device and a storage medium.

Background Art

As the trend of intelligent development is getting faster and faster, autonomous mobile devices have been widely used in various industries and people's daily lives. Autonomous mobile devices can walk according to pre-set paths or areas and perform related tasks.

Commonly used self-moving equipment includes automatic lawn mowers, automatic sprinklers, etc. Taking automatic lawn mowers as an example, there may be objects that cannot be cut outside the boundary of the working area, such as walls, roads, flowers, etc. For safety reasons, automatic lawn mowers usually only cut inside the boundary. At this time, the cutting range of the body shell and the blade disc are both inside the boundary, or there is a certain distance from the inside of the boundary, resulting in the grass at the boundary cannot be cut cleanly.

Summary of the invention

In order to overcome the problems existing in the related art, the present disclosure provides a control method of a self-moving device, a self-moving device and a storage medium.

According to a first aspect of an embodiment of the present disclosure, a control method for a self-moving device is provided, wherein the self-moving device includes a working part, the working part is configured to perform a work operation, and the self-moving device further includes: an image acquisition device, the image acquisition device is configured to acquire an image in front of the self-moving device, and the self-moving device is configured to travel and/or work in a working area, wherein the method includes:

Acquire the area type of each area in the acquired image, wherein each area at least includes: an area to be driven and a target area, wherein the area to be driven is a part of the working area, and the target area is an area adjacent to the area to be driven;

When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a part of the target area.

In one embodiment, the method further includes: inputting the collected image into a trained image recognition model to perform semantic segmentation on the image to obtain the region type of each region.

In one embodiment, the method further comprises:

When the working mode is a preset working mode, when the area type of the target area is a first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a part of the target area. The preset working mode is an edge working mode or a working mode preset by the user. The edge working mode is a mode for controlling the self-moving device to move and work along the boundary of the working area.

In one embodiment, the first area type includes one or more of roads, manhole covers, stone roads, hard roads, fallen leaves and mud, or the first area type is obtained according to a user instruction.

In one embodiment, the method further comprises: when a contact collision is detected during the process in which the working part covers a portion of the target area, controlling the self-moving device to change a driving path so as to drive toward the inside of the working area.

In one embodiment, obtaining the region type of each region in the acquired image further includes:

Performing depth estimation processing on the image to determine depth information corresponding to the target area, the depth information being used to represent the distance between each position of the target area and the image acquisition device;

Correspondingly, when the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode, including include:

The self-moving device is controlled to change the driving mode according to the area type of the target area and the depth information, so that when the area type of the target area is the first area type and the height difference between the target area and the area to be driven is less than a height threshold, the coverage range of the working unit includes a part of the target area.

In one embodiment, semantic segmentation is performed on the image to obtain the region type of each region, the region type of the to-be-traveled region is a grassland type, and the region type of the target region is a first region type; accordingly,

Controlling the coverage of the working portion to include a portion of the target area comprises:

Marking the area type of the portion of the target area adjacent to the area to be driven as a grassland type;

The self-moving device is controlled to travel and/or operate along a boundary between an area defined by the grass type and an area defined by the first area type.

In one embodiment, when the area type of the target area is the second area type, the self-moving device is controlled to travel and/or work in the working area.

In one embodiment, the second area type includes one or more of a wall, a dirt pit, a pool, a fence, a flower bed, soil with a hardness less than a hardness threshold, an irrigation facility area, and a movable life form area.

In one embodiment, controlling the self-moving device to travel and/or work in the working area includes:

Get the distance information along the edge;

The self-moving device is controlled to travel and/or work in the working area according to the edge distance information.

In one embodiment, obtaining edge distance information includes:

Get user command information;

The edge distance information is determined according to the user instruction information.

In one embodiment, the method further includes: when the area type of the target area is a first area type, controlling the self-moving device to change the driving mode so that the self-moving device avoids the area where the second area type is located, wherein the area type of the target area includes a first area type and a second area type, and the area where the first area type is located and the area where the second area type is located are both adjacent to the area to be driven.

In one embodiment, the method further includes: when the area type of the target area is a first area type, controlling the self-moving device to change the driving mode so that the driving distance of the working unit is less than the boundary length between the area where the first area type is located and the area to be driven, wherein the driving distance is the distance traveled by the self-moving device when the coverage range of the working unit includes a part of the target area.

In one embodiment, before obtaining the area type of each area in the acquired image, the self-moving device travels along a first path; when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode includes:

The self-moving device is controlled to continue traveling along the first path for a period of time, and after traveling for a period of time, the traveling path is changed so that the coverage range of the working unit includes a part of the target area.

According to a second aspect of an embodiment of the present disclosure, an embodiment of the present application provides a self-moving device, the self-moving device comprising:

A working unit configured to perform a work operation;

An image acquisition device, wherein the image acquisition device is configured to acquire an image in front of the mobile device;

A controller is connected to the working part and the image acquisition device by signal, and the controller is configured to control the self-moving device to travel and/or work in the working area, including:

The controller obtains the area type of each area in the collected image, and each area at least includes: a waiting area and a target area, wherein the waiting area is a part of the working area, and the target area is adjacent to the waiting area. area;

According to a third aspect of an embodiment of the present disclosure, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: obtaining the area type of each area in the captured image, and when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode so that the coverage of the working part includes a part of the target area. The self-moving device can process the boundary of the working area while taking safety into consideration, thereby improving the processing coverage rate of the boundary of the working area and enhancing the user experience. Furthermore, the self-moving device can quickly identify the area type of each area in the image, that is, it can quickly identify a variety of scenes, so that different working strategies can be selected for different scenes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic flow chart of a method for controlling a self-moving device provided by an embodiment of the present invention;

FIG2 is a schematic diagram of a working area according to an embodiment of the present invention;

FIG3 is a second schematic diagram of a working area in an embodiment of the present invention;

FIG4 is a third schematic diagram of a working area in an embodiment of the present invention;

FIG5 is a schematic diagram of the offset between each boundary of the working area and the charging station in an embodiment of the present invention;

FIG6 is a schematic diagram of a scenario in which an obstacle area is detected by a mobile device according to an embodiment of the present invention;

FIG7 is a first image captured by a mobile device according to an embodiment of the present invention;

FIG8 is a second image provided by an embodiment of the present invention after marking the area type of the portion of the first image adjacent to the area to be driven as a grassland type;

FIG9 is a schematic diagram of a scenario in which a mobile device is traveling along a boundary of a working area within a working area according to an embodiment of the present invention;

FIG10 is a schematic diagram of a scenario in which a mobile device performs a cross-edge action according to an embodiment of the present invention;

11-12 are schematic diagrams of another scenario in which a mobile device performs a cross-border action according to an embodiment of the present invention;

13-14 are schematic diagrams of another scenario in which a mobile device performs a cross-border action according to an embodiment of the present invention;

FIG15 is a schematic diagram of the structure of a self-moving device provided in an embodiment of the present invention;

FIG. 16 is a schematic diagram of a working module of a self-moving device provided in an embodiment of the present invention.

DETAILED DESCRIPTION

Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same names in different embodiments of the present application may have the same meaning, and also It may have different meanings, and its specific meaning needs to be determined by its explanation in this specific embodiment or further combined with the context in this specific embodiment.

It should be understood that, although the terms first, second, third, etc. may be used to describe various information in this article, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this article, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "at the time of..." or "when..." or "in response to determination". Furthermore, as used in this article, the singular forms "one", "one" and "the" are intended to also include plural forms, unless there is an opposite indication in the context. It should be further understood that the terms "comprising", "including" indicate that there are described features, steps, operations, elements, components, projects, kinds, and/or groups, but do not exclude the existence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and/or groups. The terms "or" and "and/or" used herein are interpreted as inclusive, or mean any one or any combination. Thus, “A, B, or C” or “A, B and/or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition will occur only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

It should be understood that, although the various steps in the flowchart in the embodiment of the present application are displayed in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and it can be performed in other orders. Moreover, at least a portion of the steps in the figure may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

It should be noted that, in this article, step codes such as S101, S102, etc. are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the sequence. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.

It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

Considering that the processing coverage rate at the boundary of the working area is low when the self-moving device is working in the working area defined by the boundary. Taking the self-moving device as an automatic lawn mower as an example, the cutting coverage rate at the boundary of the working area of the automatic lawn mower is low when cutting. The main reasons include three aspects: one is that the outside of the boundary may be flowers planted by the user, or obstacles such as fences, walls, and roads. In order to avoid cutting ornamental flowers or avoiding collisions with obstacles, the general automatic lawn mower cuts inside the boundary when working, and the entire body (including the shell and the cutting range of the blade) is located inside the boundary, and even has a certain distance from the boundary. Another aspect is that the automatic lawn mower based on visual products needs to rely on visual cameras to move. In the scene where the boundary is identified by the camera and the cutting is moved along the boundary, when the lateral distance between the boundary and the body is determined in real time by vision, the distance generally has an error, resulting in a distance between the body and the boundary. In addition, for safety reasons, the cutting range of the blade in the general automatic lawn mower will not exceed the body, which may cause missed cutting areas between the blade and the boundary. At least due to the above three reasons, after the machine completes the cutting work in the working area, it will still identify the existence of uncut grass in the working area, and most of the uncut grass is concentrated near the boundary.

However, users may have the following requirements: they hope to cut the boundary between the road and the lawn cleanly; or, they hope to cut the boundary between the mud and the lawn cleanly; or, because the visual recognition model only divides the image into two categories: grass and non-grass, a pile of yellow fallen leaves on the lawn will be identified as non-grass, and the boundary between grass and non-grass will make the automatic lawn mower unable to cut the area around the fallen leaves cleanly, but the user actually hopes to cut it cleanly. In other words, it is necessary to solve the problem of how existing self-moving equipment can handle the grass at the boundary of the working area while taking safety into consideration. In addition, in actual application, there will be a variety of scenes such as roads, mud, fences, walls, etc., and the users Users may have different requirements for whether different scenes are cut cleanly. In other words, the self-moving device is also required to be able to quickly identify multiple scenes, and select different working strategies for different scenes. Based on this, as shown in Figure 1, the present application proposes a control method for a self-moving device. It should be noted that although the present disclosure provides method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on routine or no creative labor. In the steps where there is no necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiments of the present disclosure.

Referring to FIG. 1 , a control method of a self-moving device provided in an embodiment of the present application is shown. For example, the self-moving device includes a working part and the self-moving device is configured to travel and/or work in a working area. The method provided in this embodiment includes:

The area type of each area is obtained, and each area at least includes: a to-be-traveled area and a target area, wherein the target area is an area adjacent to the working area.

This embodiment provides a self-moving device, as shown in Figures 1, 15 and 16, the self-moving device includes a body 27, an imaging sensor 200 (hereinafter also referred to as a visual sensor, or an image acquisition device 28), a position sensor 500 and a control circuit 600.

Specifically, the body 27 includes a driving device 700, which is used to drive the body 27 to move on the work surface according to the received driving instruction, and generally includes a roller and a motor that drives the roller to rotate. The roller may include a driving wheel and a driven wheel. The rollers 211 and 212 may be distributed on both sides of the body 27, and the number of rollers on each side may be one or two.

The body 27 also includes a working part, which is used to perform specific work tasks. For example, if the self-moving device is an automatic lawn mower, the working part includes a mowing blade 221, a cutting motor 222, etc., and may also include auxiliary components such as a mowing height adjustment mechanism to optimize or adjust the mowing effect; for example, if the self-moving device is an automatic vacuum cleaner, the working part includes a vacuum motor, a vacuum port, a vacuum pipe, a vacuum chamber, a dust collection device, etc., which are used to perform vacuum tasks.

The body 27 may also include an energy module, which is used to provide energy for various operations of the self-moving device. The energy module may include a rechargeable battery and a charging connection structure, wherein the charging connection structure is usually a charging electrode sheet, which can be used in conjunction with a charging electrode sheet set at the docking station to charge the self-moving device.

The body 27 also includes a memory 400, which is used to store data generated by sensors or control circuits, or to pre-store data for use by the control circuits.

The fuselage 27 also includes a position sensor 500 , which may include a satellite positioning sensor, an inertial sensor IMU, or an odometer ODO installed on the driving device 700 , etc., for obtaining a relative position according to the movement of the fuselage 27 .

In addition to the above modules, the body 27 may also include a shell for accommodating and installing each module, a control panel for user operation, etc. It may also include various environmental sensors, such as humidity sensors, temperature sensors, acceleration sensors, light sensors, etc. The above sensors can assist the self-mobile device in determining the working environment so as to execute the corresponding program.

The control circuit 600 (also known as the controller) is the core component of the self-moving device, which is used to control the automatic movement and operation of the self-moving device. The functions it performs include controlling the working module to start or stop working, generating a moving path and controlling the drive device 700 to move along the path, judging the power of the energy module and timely controlling the self-moving device to return to the docking station for automatic docking and charging, and executing corresponding programs in combination with the data of the environmental sensor.

1 and 15 and 16, the self-moving device includes an imaging sensor 200, which is connected to the fuselage 100 and is used to collect images in the forward direction of the fuselage 27, and the image is at least partially an image of the working surface in the forward direction. The collected image is located within the field of view 210 of the imaging sensor 200. The imaging sensor 200 can be a camera or a laser radar commonly used in the industry.

Generally, the imaging sensor 200 is installed at the upper front part of the fuselage 27, preferably in the center, with the viewing angle facing the front and bottom to collect images of the work surface. The size of its field of view can be adjusted according to actual needs. The larger the field of view 210, the more images collected in the forward direction of the fuselage 27, and vice versa. The forward direction of the fuselage 27 can have multiple directions, such as normal forward movement, backward movement, turning, etc. In this embodiment, the forward direction of the fuselage refers to the normal forward direction, that is, the direction of the central axis of the fuselage. The imaging sensor 200 in this application The middle one can be a camera.

In this embodiment, the working area may be an area where the self-moving device travels and/or works, and the working area is defined by a boundary. Of course, the self-moving device may also travel outside the working area to perform work. The area to be traveled may be a part of the working area, for example: it may be a working area within a specified range in front of the moving direction of the self-moving device, or an area marked on the working area where the self-moving device is located. For example, taking the self-moving device as an automatic lawn mower as an example, the working area may be a grass area in front of the moving direction of the self-moving device and within 3 meters of the self-moving device. It can be understood that the area to be traveled may be a partial area in the working area, that is, any area in the working area, and the target area may be any area outside the working area and adjacent to the area to be traveled.

It is understandable that there may be only one area adjacent to the area to be driven, or there may be multiple areas, that is, there may be one or more target areas. It should be noted that the area to be driven and the target area may both be located in the working area, or only the area to be driven may be located in the working area, while the target area is not in the working area.

Among them, the area type can be used to indicate the attributes of the area. Optionally, the area type can be the name of the area, such as road, stone road, hard road surface, manhole cover, earth pit, feces, fallen leaves, soil, wall, tree, fence, flower bed, stone, step, pool, soft soil, irrigation facility area, movable life area, etc.; wherein the soil hardness of soft soil is less than the hardness threshold, and the soil hardness of mud is higher than that of soft soil; soft soil, such as granular soil, may cause the self-moving device to sink and be trapped when driving on the soft soil; the irrigation facility area can be the area where the irrigation facilities (such as faucets) are located; the movable life can be a person or an animal, and the movable life can move by itself; the movable life area is the area where the movable life is located.

In a possible implementation, the self-moving device includes an image acquisition device configured to acquire images in front of the self-moving device, the images including images corresponding to the to-be-traveled area and images corresponding to the target area. The area type of each area in the image is determined by processing the image.

Specifically, the self-moving device can trigger the image acquisition device to acquire images in real time, irregularly or periodically to obtain images corresponding to the area to be driven and the target area acquired by the image acquisition device, and then perform preset processing on the acquired images to obtain the area type of the target area. It can be understood that the self-moving device can also trigger the image acquisition device to acquire images only when it is close to the boundary of the area to be driven to save energy consumption, and the self-moving device can determine whether it is close to the boundary of the area to be driven based on the relationship between its own position and the position of the boundary of the area to be driven.

Among them, the image acquisition device can be specifically a camera or other device, and the image acquisition range corresponding to the image acquisition device can be set according to the actual needs, and can usually include the designated range in front of the moving direction of the self-moving device and on both sides. Take the self-moving device as an automatic lawn mower as an example, as shown in Figure 2, the dotted box in Figure 2 represents the image acquisition range corresponding to the image acquisition device of the automatic lawn mower. When the automatic lawn mower moves close to the boundary of the lawn, the acquired image includes both images corresponding to the lawn area and images corresponding to the non-grass area, such as images corresponding to a pool or a wall. Step S101 can be performed according to the actual needs: obtaining the area type of each area in the image, each area at least includes: a to-be-traveled area and a target area, wherein the to-be-traveled area is a part of the working area, and the target area is an area adjacent to the to-be-traveled area. The method of obtaining the type includes but is not limited to semantic segmentation processing, classification processing, etc., which is not specifically limited here.

In the disclosed embodiment, the mobile device collects images corresponding to the area to be driven and the target area through an image acquisition device, and performs preset processing on the images to obtain the area type of the target area. The operation is convenient and the accuracy is high.

In one possible implementation, the image may include a two-dimensional image. The captured image is input into an image recognition model trained in advance, and each pixel in the image is classified to determine the category of each pixel (such as grass, cobblestone road, or fence, etc.), thereby achieving the goal of semantic segmentation of the image (semantic segmentation is also referred to as image segmentation below) to identify the area type of each area. The image recognition model can be a multi-classification model, that is, it can identify more than two area types, such as whether the area type is grass, road, or wall. It can be understood that the regional classification model established based on the classification algorithm can be trained in advance with training samples, The training samples may include image samples of different regions and corresponding region type labels. For example, the image recognition model may be a neural network model. By performing semantic segmentation on the image, compared to other methods such as using depth information to only identify scenes with obstacles, the present application can quickly identify a variety of richer scenes such as roads, mud, fences, walls, etc., making it easier for subsequent users to set different cutting requirements for different scenes.

In one embodiment of the present application, semantic segmentation of an image results in that the area type of the area to be driven is a grassland type, and the type of the target driving area is a first area type or a second area type, which may also be collectively referred to as a target type. As described above, the area type may be used to indicate the attributes of an area, and the grassland area in the image represents the area where the space where the grassland is located is mapped onto the image. The attributes or area types of the grassland area may be referred to as grassland attributes, grassland types, or simply grassland. The target area in the image represents the area where the space where the target object is located is mapped onto the image, and the attributes or area types of the target area are the target type, which may also be the first area type or the second area type.

Taking the self-mobile device as an automatic lawn mower as an example, referring to FIG2 , after obtaining an image corresponding to any virtual frame, the automatic lawn mower performs semantic segmentation processing on the image to determine the presence of walls, pools, fallen leaves, etc. in the image.

Exemplarily, the user can also set identification devices (such as identification signs, identification lines, etc.) corresponding to the area type in each area. The mobile device can detect and identify the identification device based on the image, determine the identification information corresponding to the identification device, and identify the area type of the target image based on the identification information.

In the disclosed embodiment, a mobile device performs semantic segmentation processing on an image corresponding to the area to be driven and an image corresponding to the target area to obtain a corresponding area type without user involvement in the setting and with a fast processing speed.

In one possible implementation, the image includes a depth image, and the image is processed to determine the area type of each area in the image, including: performing depth estimation processing on the image to determine depth information corresponding to the target area, the depth information is used to represent the distance between each position of the target area and the self-mobile device; based on the depth information, determining the area type of the target area.

Specifically, depth estimation processing is performed on the depth image including the image corresponding to the area to be driven and the image corresponding to the target area to determine the depth information corresponding to the target area, and the depth information is used to represent the distance between each position of the target area and the self-moving device. Then, the area type of the target area is determined based on the change amplitude information of the depth information.

Among them, performing depth estimation processing on an image refers to estimating the distance between the object represented by each pixel in the image and the self-mobile device, which can be specifically processed by a depth estimation network model or the like. Since different objects (i.e., areas) are photographed at the same time, the pixels of different objects in the depth image have different change characteristics. For example, the farther away from the image acquisition device of the self-mobile device, the adjacent pixels of normal grass will gradually increase, the adjacent pixels of the wall will not change, and the adjacent pixels of the depression will increase suddenly. Therefore, the area type of the target area can be determined based on the change amplitude information of the depth information corresponding to the target area. It should be noted that the specific process of performing depth estimation processing on an image can refer to the prior art and will not be repeated here. In addition, the depth information includes pixel information.

In one possible implementation, the region type of the target region is determined based on the change amplitude information of the depth information, including: when the change amplitude information represents that the depth information is continuously changed, determining the region type of the target region as the first region type; and/or, when the change amplitude information represents that the depth information is discontinuously changed, determining the region type of the target region as the second region type.

It can be understood that since the change amplitude information of the depth information corresponding to different areas has different characteristics, for example, adjacent pixels in areas such as flat ground will change continuously, while adjacent pixels in areas such as concave ground and walls will change discontinuously. At the same time, areas where adjacent pixels change continuously can be considered safe for self-equipped equipment, while areas where adjacent pixels change discontinuously can be considered dangerous for self-equipped equipment. Therefore, the area type of the target area can be determined based on the change amplitude information of the depth information, that is, when the change amplitude information represents that the depth information is continuously changed, the area type of the target area is determined to be the first area type, and/or, when the change amplitude information represents that the depth information is discontinuously changed, the area type of the target area is determined to be the second area type.

In a possible implementation, obtaining the region type of each region includes: detecting a boundary signal and determining a detection result, The boundary signal includes a magnetic field signal generated by a boundary line set at the boundary of the area to be driven; and the area type of each area is determined according to the detection result.

It can be understood that whether to set a boundary line on the target boundary can be determined in advance according to the area type of the target area adjacent to the area to be traveled, so as to generate a boundary signal through the boundary line. For example, when the target area adjacent to the area to be traveled is a dangerous area such as a pool or a wall, a boundary line can be set on the target boundary between the area to be traveled and the target area, so as to generate a boundary signal for indicating that the target area is a dangerous area through the boundary line; and when the target area adjacent to the area to be traveled is a safe area such as soil or fallen leaves, a boundary line may not be set on the target boundary between the area to be traveled and the target area, so as not to generate a boundary signal.

Here, the boundary line is a conductor that forms a loop after being energized. When a constant current is passed through the boundary line, a constant magnetic field surrounding the boundary line is generated around it, and the self-moving device can identify the magnetic field signal of the constant magnetic field, i.e., the boundary signal, during the working and walking process. The boundary line can be specifically a geomagnetic line, etc., and the geomagnetic line can be specifically a permanent magnet, a energized conductor, etc.

Among them, the self-moving device may be provided with a signal detection device for detecting boundary signals. The self-moving device may detect boundary signals in real time, irregularly or periodically and determine the detection results, and then determine the area type of the target area according to the detection results. For example, if the detection result shows that no boundary signal is detected, it means that the area type of the target area or the boundary type of the target boundary is safe; if the detection result shows that a boundary signal is detected, it means that the area type of the target area or the boundary type of the target boundary is dangerous, etc. It can be understood that the self-moving device may also trigger the detection boundary signal only when it is close to the boundary of the area to be driven to save energy consumption, and the self-moving device may judge whether it is close to the boundary of the area to be driven based on its own position and the position of the boundary of the area to be driven.

In a possible implementation, the region type of the target area or the boundary type of the target boundary is determined according to the detection result, including: when no boundary signal is detected, the region type of the target area is determined to be the first region type; when a boundary signal is detected, the region type of the target area is determined to be the second region type, or the boundary type of the target boundary is determined to be the second boundary type. Specifically, when it is determined according to the detection result that no boundary signal is detected, it means that it is safe for the self-moving device or the working part to move and work in the target area or at the target boundary, and the region type of the target area is determined to be the first region type. For example, if the area to be driven is a grassland, and the target area is fallen leaves, a boundary line may not be set at the boundary between the grassland and the fallen leaves, so that when the self-moving device moves to the vicinity of the boundary between the grassland and the fallen leaves, the boundary signal cannot be detected. When it is determined according to the detection result that a boundary signal is detected, it means that it is dangerous for the self-moving device or the working part to move and work in the target area or at the target boundary, and the region type of the target area is determined to be the second region type. For example, if the area to be driven is a grassland and the target area is a pond, a boundary line may be set at the boundary between the grassland and the pond, so that a boundary signal is detected when the mobile device moves near the boundary between the grassland and the pond.

Take the self-moving device as an automatic lawn mower as an example, refer to Figure 3. When the automatic lawn mower moves to a puddle, it may cause the automatic lawn mower to fall, and when it works close to a wall, the cutter disc may be damaged, that is, the puddle and the wall belong to the dangerous area. Therefore, geomagnetic lines are set at the boundaries between the grass and the puddle and the wall respectively, so that when the automatic lawn mower moves close to the boundary between the grass and the puddle or the boundary between the grass and the wall, a boundary signal will be detected, thereby determining that the area type of the target area is a dangerous type. In other words, a boundary line is set only at the boundary between the grass area and the dangerous type area. If a boundary signal is detected, it means there is danger, and the automatic lawn mower cannot cross the boundary. If no boundary signal is detected, it means there is no danger, and the automatic lawn mower can cross the boundary.

In a possible implementation, the region type of each region is obtained, including:

Acquire a boundary signal, where the boundary signal is a magnetic field signal generated by a boundary line set at the boundary of the area to be driven;

Acquire preset mapping information, where the preset mapping information represents a correspondence between the strength of the boundary signal and the area type;

The area type of each area is determined according to the strength of the boundary signal and the preset mapping information.

It can be understood that, for the area type of the target area adjacent to the area to be driven, the parameters of the boundary line of the boundary between the area to be driven and the target area can be set or controlled accordingly, so that when the area type of the target area is different, the strength of the magnetic field signal generated by the corresponding boundary line is also different accordingly. For example, when the area type of the target area is a safe area, the magnetic field signal generated by the corresponding boundary line can be controlled. The intensity of the field signal is a first preset threshold, and when the area type of the target area is a dangerous area, the intensity of the magnetic field signal generated by the corresponding boundary line can be controlled to be a second preset threshold, and the second preset threshold is greater than the first preset threshold, etc. After the mobile device obtains the boundary signal generated by the boundary line set at the boundary of the area to be driven, the correspondence between the strength of the predetermined boundary signal and the area type or boundary type can be queried according to the strength of the boundary signal, thereby determining the area type of the target area or the boundary type of the target boundary. Among them, for different area types corresponding to different areas adjacent to the area to be driven, the intensity of the magnetic field signal generated by the corresponding boundary line can be different by controlling the input current to the boundary line of the boundary between the area to be driven and the adjacent area. For example, when the target area adjacent to the area to be driven is soil, the input current of the boundary line of the boundary between the area to be driven and the soil area can be controlled to be a first current value, and when the target area adjacent to the area to be driven is stone, the input current of the boundary line of the boundary between the area to be driven and the stone area can be controlled to be a second current value, and the second current value is greater than the first current value, so as to increase the intensity of the magnetic field signal generated by the boundary line.

In a possible implementation, obtaining the area type of each area includes: obtaining the type of the boundary between the area to be driven and the target area, and determining the area type according to the type of the boundary. Specifically, obtaining map information, the map information including the map, the position information of each boundary in the map, and the type information of each boundary in the map; determining the first position information of each boundary; and determining the boundary type according to the first position information of each boundary and the map information.

Among them, the map information can be obtained by mapping based on satellite positioning sensors, inertial sensors and odometer sensors. Taking the self-mobile device as an automatic lawn mower as an example, refer to Figure 4. When the automatic lawn mower drives along the boundary of the grass area for the first time, the offset of each boundary (a, b, c, d, e) of the grass area and the charging station can be recorded according to the satellite positioning sensor, inertial sensor and odometer sensor, as shown in Figure 5; then, the user can set the type of each boundary (a, b, c, d, e) through APP, etc.; finally, the offset of each boundary of the grass area and the charging station and the type of each boundary are stored in the map. Alternatively, the map information is obtained by mapping based on visual real-time positioning and map construction technology.

It is understandable that the user can set the attributes of each boundary of the work area on the map in advance, and the attributes of the boundary include the location information and type information of the boundary, and mark and save the attribute information of each boundary in the map. For example, assuming that the shape of the work area is a rectangle, and the areas adjacent to the work area are stone, road, mud and pool, respectively, the attribute information of the boundary corresponding to the work area can be set in the map as stone, road, mud and pool, and the location information of each boundary can be marked in the map.

In a possible implementation, the user can pre-divide the regions on the map, including the location of each region and the corresponding region type, and mark and save the region division results including the region type and the corresponding location in the map. For example, assuming that the map includes three adjacent regions, namely, grass, mud, and a pool, the attribute information corresponding to each region can be set in the map, namely, grass, mud, and a pool, and the location information of each region can be marked in the map, so that the mobile device can obtain the region type of the to-be-traveled region and the target region according to the region type marked in the map during operation.

In this embodiment, the area type of each area in the captured image is obtained, and the mobile device can quickly identify various scenes such as roads, mud, fences, walls, etc., so as to select different working strategies for different scenes.

Step S102: When the area type of the target area is the first area type, control the self-mobile device to change the driving mode so that the coverage of the working unit includes a part of the target area.

In one embodiment of the present application, after obtaining the area type of each area in the captured image, the self-moving device can be controlled to travel and/or work based on the area type of the target area, so that when the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage of the working part includes a part of the target area; when the area type of the target area is identified as the second area type, the self-moving device is controlled to travel and/or work along the boundary of the area to be driven. Changing the driving mode may include: changing the driving path or changing the driving direction.

In order to enable the self-moving device to process the boundary of the working area while taking into account safety, The position of the working unit can be adjusted according to the area type so that the coverage of the working unit meets the preset conditions, so as to improve the processing coverage rate at the boundary of the working area and enhance the user experience. It can be understood that the preset conditions can be set according to actual needs. For example, taking the self-moving device as an automatic lawn mower as an example, if it is determined according to the area type that the automatic lawn mower can cross the boundary or the working unit can work at the boundary, then the coverage of the working unit is set to include a part of the target area, that is, it can cross the boundary, so that the automatic lawn mower can cut the grass at the boundary of the working area; if it is determined according to the area type that the automatic lawn mower cannot cross the boundary, then the working unit is set to travel within the working area, which can be along the boundary of the area to be traveled, that is, the working unit cannot cross the boundary.

In an embodiment of the present application, if the area type of the target area is safe or the user expects the machine to travel in the first area type, the coverage of the working part can be controlled to include a part of the target area. At this time, there will be no collision, entanglement, jamming, excessive tilting, falling, etc. that hinder the movement of the self-moving device, and there will be no harm to the safety of pedestrians or animals; if the area type of the target area is unsafe or the user does not expect the machine to travel in the second area type, the machine can be controlled not to travel to the target area during the driving process to ensure the safety of the machine. Exemplarily, if the target area is a cobblestone road, the self-moving device can cross the boundary between the cobblestone road and the working area to safely travel and/or work on the cobblestone road. If the target area is a pool, the mobile device cannot cross the boundary between the pool and the working area. It should be noted that the self-moving device can keep working during the adjustment of the working part, or it can work after the adjustment of the working part is completed, which is not specifically limited here.

In the present application, when the working mode is a preset working mode, when the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage of the working part includes a part of the target area. The preset working mode is an edge working mode or a working mode preset by the user. The edge working mode is a mode for controlling the self-moving device to move and work along the boundary of the working area.

Specifically, the self-moving device at least includes: a random working mode and an edge working mode. In the random working mode, the self-moving device randomly cuts or drives along the planned path in the working area; in the edge working mode, the self-moving device drives along the boundary of the working area. In the process of driving along the boundary, the machine can execute the steps of obtaining the area type of each area in the collected image proposed in this application, and controlling the movement of the self-moving device according to the area type. When the conditions of the first area type are met, the working part of the machine can cover the part of the target area beyond the working area, which is referred to as performing a cross-edge action. That is, in the edge mode, the machine can perform a cross-edge action during the process of driving along the boundary, for example: driving astride the boundary of the working area. In addition, optionally, the self-moving device may also include a cross-edge working mode. In the cross-edge working mode, the coverage range of the working part of the self-moving device includes a part of the target area; at this time, the edge working mode may only control the machine to drive along the boundary inside the working area. In addition, in the random working mode, when the self-moving device works near the boundary of the working area, in order to enable the self-moving device to cover more working ranges, it can be controlled to trigger the cross-edge action when the image meets the area type. Of course, in the present application, the cross-border action can be triggered only when the cross-border working mode is turned on, thereby improving the flexibility and convenience of controlling the self-mobile device.

As shown in FIG9 , the self-moving device drives along the boundary of the working area inside the working area. The self-moving device can move and work in the working area surrounded by the boundary 10, and can move and work along the boundary 10. When the machine moves along the boundary 10, there will be a certain distance d between the machine and the boundary 10, so as to prevent the machine from going out of the boundary and preventing the machine from being damaged. The machine can determine the actual distance d between the machine and the boundary 10 based on the image.

FIG10 is a schematic diagram of the machine performing a crossing-edge action. The target area 70 is an area where the self-moving device can safely travel, such as a stone slab area, a sidewalk area, a dirt area, etc.; when the self-moving device is located on the side of the target area 70, the target area exists in the image taken by the self-moving device, and the to-be-marked area of the target area can be marked as grass attribute, thereby changing the boundary between the grass attribute area and the target attribute area. For example, FIG7 is a first image taken by the self-moving device, and FIG8 is a second image after the area type of the part adjacent to the to-be-traveled area in the first image is marked as grass type. It can be seen that the position of the boundary 50 between the grass type area and the first area type in the first image and the boundary 60 between the grass type area and the first area type in the second image. is different; the self-mobile device moves along the boundary 60 between the grass type area and the first area type in the second image, so that it can move across the boundary 10 of the actual space, achieve cutting to the edge, and cleanly cut the grass between the working area and the slate area.

In a possible implementation, the self-mobile device may provide multiple working modes for the user to choose from, or the self-mobile device may automatically select the desired working mode during operation. The machine may, according to the working mode selected by the user, execute the steps of obtaining the region type of each region in the acquired image and controlling the movement of the self-mobile device according to the region type when the working mode is the edge working mode. Alternatively, the corresponding process may be executed according to the execution flow planned within the machine. For example, the random working mode may be executed first, and after the work corresponding to the random working mode is completed, the machine may switch to the edge working mode for work.

In one embodiment of the present application, when the area type of the target area is identified as the first area type, the self-moving device is controlled so that the coverage of its working part includes a part of the target area. In one possible implementation, the first area type includes one or more of roads, manhole covers, stone roads, hard roads, fallen leaves and soil. Among them, roads include but are not limited to cement ground, asphalt ground, roads, sidewalks, etc. The first area type is not limited to this, as long as the self-moving device can safely drive and/or work on it. The first area type can also be set according to the received user instructions, so that the machine can perform cross-edge in certain scenarios according to user needs. The self-moving device can identify multiple types of objects or objects in the image based on the image recognition model, for example, it can simultaneously identify grass, ditches, puddles, fences, stone slabs, fallen leaves, walls and other non-grass; the user can select one or more types as the first area type, for example, the user can select stone slabs and fallen leaves as the first area type.

In one embodiment of the present application, when the area type of the target area is the second area type, the self-moving device is controlled to travel and/or work along the boundary of the area to be driven. The second area type may include one or more of a wall, a pit, a pool, a fence, a flower bed, soil with a soil hardness less than a hardness threshold, an irrigation facility area, and a movable life area. Further, when the second area type is met, if the height difference between the area where the second area type is located and the area to be driven is equal to or greater than the height threshold (for example, 10 cm), the machine is controlled to travel along the boundary of the area to be driven to ensure the safety of the machine. When it is identified that the area type of the target area is the second area type or an area with a higher height difference, it is necessary to control the machine to keep driving within the working area to prevent rollover or crash, or unsafe phenomena such as collision of the working part.

For example: In combination with FIG6 , in one embodiment, the image captured by the image acquisition device can be detected based on the image recognition model. When an obstacle area 30 (i.e., the area where the second area type in the present application is located) is detected, the self-moving device is controlled to stay away from the boundary between the obstacle area 30 (such as a pool) and the working area, and the self-moving device is controlled to maintain a safety distance f from the boundary, and the machine moves and works while maintaining the safety distance f to avoid damage to the machine. The value of the safety distance f can be fixed or adjustable. The value of the safety distance f can be automatically adjusted by the machine, or it can be set or adjusted by the user.

In one embodiment of the present application, as shown in Figures 7 and 8, the self-moving device is controlled to travel and/or work based on the area type of the target area, so that when the area type of the target area is the first area type, the coverage of the working unit includes a part of the target area, which may include: marking the area type of the part of the target area adjacent to the area to be traveled as a grassland type; controlling the self-moving device to travel and/or work along the boundary between the area determined by the grassland type and the area determined by the first area type.

Specifically, as shown in FIG7 , the area to be traveled is a grassland type. When the area type of the target area is the first area type, the area to be traveled can be expanded as shown in FIG8 , and the intersection area between the grassland area obtained after the expansion and the target area is marked as a grassland attribute (i.e., a grassland type) to obtain a second image. Alternatively, the area to be marked can be determined in the target area, and the part of the target area close to the grassland area can be taken as the area to be marked, and then the area to be marked can be marked as a grassland attribute to obtain a marked grassland area to obtain a second image. In this way, the self-moving device can be controlled to move along the boundary between the grassland area and the target area in the second image. In an embodiment of the present application, since the boundary position between the grassland area and the target area is adjusted, the self-moving device can move along the adjusted boundary, and the boundary is basically consistent with the physical boundary in the actual space. Therefore, the self-moving device can move across the physical boundary in the actual space. The boundary moves and cuts during the movement, so that the grass at the boundary is cut cleanly, achieving the effect of cutting to the edge, improving the cutting coverage rate and user satisfaction.

Furthermore, obtaining the area type of each area in the captured image may also include: performing depth estimation processing on the image to determine the depth information corresponding to the target area, the depth information is used to represent the distance between each position of the target area and the image acquisition device; accordingly, controlling the self-moving device to travel and/or work based on the area type of the target area may include: controlling the self-moving device to travel and/or work according to the area type and depth information of the target area, so that when the area type of the target area is the first area type, and the height difference between the target area and the area to be driven is less than the height threshold, the coverage of the working part includes a part of the target area. That is, in addition to meeting the area type requirement, the target area also needs to meet the height requirement. When the height difference between the target area that meets the first area type and the area to be driven is less than the height threshold, for example, the height threshold can be set to 3 cm, 5 cm, etc. For example: the target area is a manhole cover with a height difference of 3 cm with the working area, then the self-moving device can be controlled to perform a cross-edge action.

In one embodiment of the present application, as shown in Figures 13 and 14, when the self-moving device enters the edge-along working mode and identifies the process of the stone slab 70 crossing the edge, when a collision with the stone 81 is detected, the self-moving device can be controlled to change the driving direction to drive inside the working area as shown in Figure 14. Driving inside the working area may include: driving along the boundary of the working area, and may also include controlling the projection of the working part in the working area to be inside the working area. For example: when the lawn mower is driving across the edge, when the camera is detected to be displaced, it can be controlled to drive within the working area.

Furthermore, when a collision is detected, the time or continuous distance of the machine driving into the working area is recorded. If it exceeds 5 meters, the machine can be controlled to restart to identify whether the first area type exists. If it exists, the machine continues to drive across the boundary. If no stone slab is detected during the edge-traveling process, the machine is controlled to drive into the working area. If the driving distance exceeds 5 meters, the machine can be controlled to restart the driving across the boundary.

In an application scenario of the present application, during the process of moving along the edge, when the machine walks to the end of the long strip of stone road, only part of the stone road remains in the picture. Since the machine uses single-frame recognition, the blocky stone road in the image will be considered as a stepping stone. Stepping stones refer to stone slabs that are relatively small and high. Machines cannot pass through. Therefore, due to recognition errors, the machine will not be able to continue along the edge. In response to this problem, it can be determined whether the machine is already in a stable crossing state. If the machine is in a stable crossing state, the stepping stone is used as a stone road and the crossing action is performed; if it is not in a stable crossing state, the processing logic of avoiding the stepping stone, that is, controlling the machine to travel into the working area, is executed. Among them, if the stone road boundary is seen for a continuous period of time (such as: within 2S), the stable crossing state is entered, and if the stone road boundary is not seen for a continuous period of time (such as: within 5S), the stable crossing state is exited.

In one embodiment, the first image captured by the image acquisition device may contain one or more of a grass area, a target area of a first area type, and a target area of a second area type (hereinafter also referred to as an obstacle area).

In one embodiment of the present application, when there are multiple area types in the target area of the first image, the self-moving device can be controlled to travel and/or work based on the area type of the target area, so that the self-moving device avoids the area where the second area type is located during driving, and prevents the self-moving device from traveling along the boundary between the area determined by the grass type and the area determined by the first area type, and part of the machine body travels to the area where the second area type is located, which may cause safety problems. The area types of the target area include the first area type and the second area type, and the area where the first area type is located and the area where the second area type is located are both adjacent to the area to be driven.

In one embodiment of the present application, the self-moving device is controlled to travel and/or work based on the area type of the target area so that the travel distance of the working unit is less than the boundary length between the area where the first area type is located and the area to be traveled, wherein the travel distance is the distance traveled by the self-moving device when the coverage of the working unit includes a part of the target area. That is, the travel distance of the self-moving device in response to identifying the first area type is less than the actual physical boundary length adjacent to the first area type. In the case where there are multiple area types in the target area type, the control machine only performs part of the cross-border action to prevent safety problems.

In one embodiment of the present application, an image is collected while the mobile device is traveling along the first path, and each The area type of the area controls the self-moving device to continue to travel along the first path for a period of time, and after traveling for a period of time, changes the travel path so that the coverage range of the working part includes a part of the target area. It can also be controlled to travel to a position where the distance from the pool is the length of the body of the lawn mower according to the length of the body, and then the controller performs the cross-edge action.

Taking the scenes shown in FIG. 11 and FIG. 12 as an example, when the lawn mower is driving in the current direction, the area type of each area in the image is obtained, and the area type may include: grass 90, stone road 70 and pool 80. The stone road is adjacent to the pool, and the stone road and the pool as a whole are adjacent to the working area. The lawn mower is controlled to start the cross-edge mode, the camera captures the image, and the grass, stone road and pool in the image are semantically identified, the grass in the image is expanded, and the part of the expanded image that overlaps with the stone road is obtained, and the part is modified to grass, and the stone road and the pool in the non-overlapping part are marked as non-grass. As shown in the figure, if the machine directly drives to the stone position PP adjacent to the pool, its tail may drive into the pool. Therefore, the machine can be controlled to continue driving in the current direction for a period of time, and then the lawn mower can be controlled to change the driving direction to drive along the boundary between grass and non-grass in the image. By delaying the response to the image recognition result, the lawn mower will not fall into the pool when driving to the grass area in the image to ensure the safety of the lawn mower.

In the present application, the area type of each area in the captured image is obtained, and when the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage of the working part includes a part of the target area. The self-moving device can process the boundary of the working area while taking safety into consideration, thereby improving the processing coverage rate of the boundary of the working area and enhancing the user experience. Furthermore, the self-moving device can quickly identify the area type of each area in the image, that is, it can quickly identify a variety of scenes, so that different working strategies can be selected for different scenes.

In another embodiment of the present application, the self-mobile device collects images during driving and establishes a local map of its surrounding area based on the images. When a scenario is identified in which the area where the first area type and the area where the second area type are located are adjacent to the area to be driven, the driving path of the self-mobile device can be planned according to the local map to control the vehicle to avoid the area where the second area type is located after driving according to the adjusted image.

In one embodiment, after obtaining an image acquired by an image acquisition device (hereinafter also referred to as a first image), marking the area type of a portion of the target area adjacent to the area to be driven as a grass type to obtain a second image includes:

Traversing all pixels in the first image to determine pixels in the area to be marked;

The pixels in the area to be marked are marked as pixels corresponding to grass, thereby obtaining a second image.

Exemplarily, the color of the pixels in the area to be marked may be modified to green, or the pattern shape of the area to be marked may be modified to the shape of grass.

The area to be marked may be a partial area of the target area with a preset width adjacent to the grass area.

In one embodiment, after obtaining an image acquired by the image acquisition device (hereinafter also referred to as the first image), marking the area type of a portion of the target area adjacent to the area to be driven as a grassland type includes:

rasterizing the first image to generate a raster image, wherein the raster image includes a grassland raster corresponding to the grassland area and a target raster corresponding to the target area, the attributes of the grassland raster are grassland attributes, and the attributes of the target raster are the first area type or the second area type;

Perform expansion processing on the grassland raster, and mark the attributes of the expanded raster as grassland attributes;

Get the overlapping part of the target grid with the expanded grid;

If the attribute of the overlapping grid in the target grid is the first area type, the attribute of the overlapping grid is modified to the grassland attribute;

If the attribute of the overlapping grid in the target grid is the second area type, the attribute of the overlapping grid is modified to a non-grassland attribute to obtain a second image.

Alternatively, after the grid image is generated, the grid to be marked may be determined according to the position of the grassland grid and the position of the target grid; the attributes of the grid to be marked are marked as grassland attributes to obtain the second image.

The standard for rasterizing an image is: a grid contains 25 pixels, each pixel is marked with a corresponding attribute, and each grid attribute is based on the attribute of a larger number of pixels. The first image can be rasterized based on the camera's internal parameters, external parameters, distortion parameters, and image pixels to obtain a raster image corresponding to the first image. During the rasterization process, the grassland area can be converted into a grassland grid, and the attributes of the grassland grid are grassland attributes; the target area can be converted into a target grid, and the attributes of the target grid are target attributes, that is, the first region type or the second region type. Non-grass attributes include the first region type or the second region type. In this embodiment, the target attribute of the grid to be marked can be modified to the grassland attribute to obtain the second image.

In one embodiment, the neighboring grid of the grid to be marked includes a grassland grid. The grid to be marked may be a grid adjacent to the grassland grid. Exemplarily, if any grid among the neighboring grids around the grid to be marked is a grassland grid, the grid to be marked may be regarded as a grassland grid.

In one embodiment, when the self-moving device moves along the boundary of the working area based on a counterclockwise direction, the left grid of the grid to be marked is a grass grid.

When the self-moving device moves along the boundary of the working area in a counterclockwise direction, the boundary is on the right side of the self-moving device and the grassland is on the left side of the self-moving device. Therefore, it can be determined whether the left grid of the target grid is a grassland grid. If it is a grassland grid, the target grid can be used as the grid to be marked.

When the mobile device moves along the boundary of the working area in a clockwise direction, it is determined whether the right grid of the target grid is a grassland grid. If it is a grassland grid, the target grid can be used as a grid to be marked.

In one embodiment, the lower grid of the grid to be marked is a grass grid.

For example, the stone slab may be located directly in front of the machine, and the camera does not capture the side of the stone slab parallel to the machine's forward direction, but only captures the side of the stone slab perpendicular to the machine's forward direction. In this case, since the left grid of all target grids is the target area, it can be determined whether the grid below the target grid is a grass grid. If it is a grass grid, the target grid can be used as a grid to be marked. Alternatively, in another embodiment, when there is no area to be driven in the image, it means that the machine is at a right angle to the boundary. When the machine is driving counterclockwise along the boundary, the machine is controlled to turn left or an alarm is issued.

Since the position and shape of the boundary of the machine are varied, the position and shape of the boundary in the image captured by the machine are also varied. You can set specific conditions according to actual needs and use the target grid that meets the conditions as the grid to be marked.

In one embodiment, determining the grid to be marked according to the positions of the grassland grid and the target grid includes:

The grid to be marked is determined according to the position of the grassland grid, the position of the target grid and the cross-edge distance conditions.

For example, the span distance condition may represent the width of a portion of a machine body that exceeds an actual boundary, or the width of a portion of a machine working portion that exceeds a boundary.

According to the position of the grassland grid and the position of the target grid, the target grid adjacent to the grassland grid can be determined as the first column of grids to be marked, and further according to the cross-edge distance condition, the X columns of target grids located on the side of the first column of grids to be marked can be used as other columns of grids to be marked. The X value can be determined according to the cross-edge distance condition.

For example, if the width of the portion of the machine body that exceeds the actual boundary is required to be 10 cm, and the width of a grid is 5 cm, two columns of target grids can be used as grids to be marked, and X is 1.

For example, if the control logic of the machine is that the fuselage and the corresponding position of the boundary in the image need to maintain a certain distance, such as 10 cm, and the cross-edge distance condition requires that the width of the part of the machine fuselage that exceeds the actual boundary is 10 cm, and the width of a grid is 5 cm, then the four-column target grid that meets the conditions can be used as the grid to be marked, and X is 3.

In one embodiment, after converting the first image to obtain the first raster image, all the raster in the first raster image can be traversed to determine the raster to be marked in the first raster image, mark it as grass attribute, and obtain a second raster image; then determine the raster to be marked in the second raster image, mark it as grass attribute, and obtain a third raster image; then mark the raster to be marked in the third raster image The grid is marked as grass attribute, and the fourth grid image is obtained; then the grids to be marked in the fourth grid image are marked as grass attribute, and the fifth grid image, i.e., the second image, is obtained. Thus, all grids in the image are traversed four times, and four columns of target grids are used as grids to be marked.

In one embodiment, in the process of traversing the grids, the grass grid originally having grass attributes will not be modified, and the obstacle grid originally having obstacle attributes will not be modified.

In one embodiment, controlling the self-moving device to travel and/or work according to the second image includes:

According to the second image, the self-moving device is controlled to travel and/or work in the space corresponding to the grass attribute area, avoid the space corresponding to the non-grass attribute area, and control the coverage range of the working part of the self-moving device to meet preset conditions; the preset conditions include that the coverage range of the working part includes a part of the target area, or the coverage range of the working part intersects with the boundary of the working area, or the distance between the coverage range of the working part and the boundary of the working area is less than a first threshold.

Exemplarily, in the process of moving along the border in a counterclockwise direction, when the self-mobile device detects the target area based on the image, the self-mobile device can be controlled to turn right and move according to the processed second image until the self-mobile device drives and/or works in the space corresponding to the grass attribute area of the second image, and drives along the position corresponding to the boundary between the grass attribute area and the target attribute area, thereby achieving cross-edge cutting.

As shown in Figure 10, in the actual space, the self-moving device is driving and cutting across the edge; but in the internal control logic of the self-moving device, it still thinks that it is driving and cutting along the boundary; this is because the grass attribute area and the target attribute area in the second image are re-marked based on the first image. This method is also more compatible with other control logics of the machine.

Based on the second image, a map of the working area is generated, the map including a drivable area and a non-drivable area, the drivable area corresponds to the grass attribute area of the second image, and the non-drivable area corresponds to the non-grass attribute area of the second image; based on the map, the self-moving device is controlled to drive and/or work so that the self-moving device drives and/or works in the space corresponding to the drivable area and avoids the space corresponding to the non-drivable area.

A map of a local area in the working area is generated based on the second image, and the map can reflect the environment around the mobile device, and reflect the grass and non-grass around the mobile device. The drivable area corresponds to the grass attribute area of the second image, including the real grass area and the target area whose attributes are changed to grass attributes. The non-drivable area corresponds to the non-grass attribute area of the second image, including the real obstacle area, the target area whose attributes are not changed, and the obstacle area.

The self-moving device can plan the path according to the map, so as to move along the boundary between the drivable area and the non-drivable area, and realize cross-edge cutting in the actual space.

In one embodiment, the height difference between the target area and the grass area is less than a second threshold.

In this embodiment, the target area is a safe area, so the height difference between the two cannot be too large.

As shown in FIG9 , in one embodiment, the method further includes:

When the region type of the target region in the first image is the second region type, obtaining edge distance information;

According to the edge distance information, the distance d between the mobile device and the boundary 10 of the working area is controlled.

If only the second area type is identified in the image collected by the self-moving device during the movement along the edge, it means that the outside of the boundary between the to-be-traveled area and the target area may be dangerous, so it cannot move across the boundary. In order to ensure safety, at this time, a certain distance d can be maintained between the machine and the boundary 10, and the distance can be fixed or adjustable.

The actual distance d may be adjusted according to the acquired edge distance information so that the actual distance d is equal to the distance value represented by the edge distance information.

In one embodiment, obtaining the edge distance information includes: obtaining user instruction information; and determining the edge distance information according to the user instruction information. The edge distance information can be determined according to the user instruction. For example, if the user selects 10 cm, the edge distance information is 10cm, the self-moving device can adjust the moving direction, thereby adjusting the distance d between itself and the boundary 10 to 10cm.

In one embodiment, the area type of the portion of the target area adjacent to the area to be driven is marked as a grass type to obtain a second image, including: determining the area to be marked; and when no collision is detected, marking the attributes of the area to be marked in the first image as grass attributes to obtain a second image.

In one embodiment, the attributes of the area to be marked in the first image are marked as grass attributes to obtain a second image, including: determining the area to be marked; in the case of a collision being detected, determining a first area and a second area of the area to be marked, the distance between the first area and the grass area being smaller than the distance between the second area and the grass area; marking the attributes of the first area as grass attributes to obtain a second image.

Exemplarily, the machine is provided with collision detection sensors.

For example, when the machine does not have the cross-edge cutting function (that is, the cross-edge action described above) turned on, the control logic of the machine is that the fuselage needs to maintain a certain distance from the position corresponding to the boundary in the image, such as 15 cm. When the machine turns on the cross-edge cutting function and the machine does not detect a collision, the control logic of the machine is that the fuselage needs to maintain a certain distance from the position corresponding to the boundary in the image, such as 10 cm. The cross-edge distance condition requires that the width of the part of the machine body that exceeds the actual boundary is 10 cm, and the width of a grid is 5 cm. In this case, the four-column target grid that meets the conditions can be used as the grid to be marked.

When the collision sensor of the machine detects a collision, the machine can be controlled to move away from the collision object, increase the distance between the machine and the collision object, and move. For example, when the machine turns on the cross-edge cutting function and the collision sensor of the machine detects a collision, the control logic of the machine is that the fuselage and the corresponding position of the boundary in the image need to maintain a certain distance, such as 10cm, and the cross-edge distance condition can be changed to require that the width of the part of the machine body that exceeds the actual boundary is 0cm, and the two columns of target grids that meet the conditions can be used as grids to be marked. In other words, when cross-edge cutting is turned on and a collision is detected, the number of columns of grids to be marked can be reduced so that the number of columns is less than the number of columns of grids to be marked when cross-edge cutting is turned on and no collision is detected.

In one embodiment, the area type of the portion of the target area adjacent to the area to be driven is marked as a grass type, including: determining the area to be marked; determining a first area and a second area of the area to be marked, wherein the distance between the first area and the grass area is smaller than the distance between the second area and the grass area; marking the attributes of the first area and the second area as grass attributes; and in the event of a collision being detected, marking the attributes of the second area as target attributes to obtain a second image.

When cross-edge cutting is enabled and a collision is detected, all grids to be marked may be marked as grass grids, and then a portion of the grids to be marked may be marked back to the target attribute, thereby obtaining a second image.

In one embodiment, when the self-moving device does not detect a collision, the step of marking the attributes of the area to be marked in the first image as grass attributes to obtain a second image is executed; the control method also includes: when the self-moving device detects a collision, marking the attributes of the area to be selected in the first image as target attributes to obtain a third image, and the area to be selected is an area in the grass area close to the target area; according to the third image, the self-moving device is controlled to travel and/or work.

After the cross-edge cutting function is turned on, if no collision is detected, the attribute of the area to be marked can be directly marked as a grass attribute to obtain a second image.

After the cross-edge cutting function is turned on, if a collision is detected, the grid near the target area in the grass area can be marked as a target attribute to obtain a third image, and then the self-moving device can be controlled to travel and/or work according to the third image, so that the boundary is retracted toward the grass, and the self-moving device is away from the collision object to avoid collision. The position of the retracted boundary can be directly analyzed according to the third image, and then the machine can be controlled. A grid map can also be generated according to the third image, and the machine can be controlled according to the boundary information in the map.

In one implementation manner, for a first image acquired within a preset time period after a collision is detected, an attribute of a to-be-selected area in the first image is marked as a target attribute to obtain a third image.

For example, within 7 seconds after the collision, the step of marking the attributes of the area to be selected in the first image as the target attributes is performed to obtain the third image. After 7 seconds after the collision, the step of marking the attributes of the area to be marked in the first image as the grass attributes is performed to obtain the second image. steps.

For example, when the cross-edge cutting function is not turned on, the control logic of the machine is that the machine body needs to maintain a certain distance from the position corresponding to the boundary in the image, such as 15 cm.

When the machine turns on the cross-edge cutting function and does not detect a collision, the machine's control logic is that the machine body needs to maintain a certain distance from the corresponding position of the boundary in the image, such as 10 cm. The cross-edge distance condition requires that the width of the part of the machine body that exceeds the actual boundary is 10 cm, and the width of a grid is 5 cm. In this case, the four-column target grid that meets the conditions can be used as the grid to be marked.

When the machine turns on the cross-edge cutting function and the machine's collision sensor detects a collision, the machine's control logic is that the machine body needs to maintain a certain distance from the corresponding position of the boundary in the image, such as 10 cm, and the cross-edge distance condition is changed to require the machine body to retract 10 cm inward from the actual boundary. In this way, the two columns of target grids in the grass area can be used as candidate grids.

That is to say, when the cross-edge cutting function is turned on and a collision is detected when crossing the edge, the self-moving device can be controlled to move inward along the edge within a period of time after the collision, that is, the self-moving device can be controlled to move along the edge inside the grass, and the distance between the self-moving device and the boundary can be appropriately increased to avoid collision.

It should be noted that different embodiments provided in the embodiments of the present disclosure can be combined with each other, but the manner of combination is not specifically limited.

An embodiment of the present disclosure also provides a self-moving device, comprising: a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the control method of the self-moving device described in any one of the embodiments of the present disclosure is implemented.

Based on the same inventive concept as the above-mentioned embodiment, this embodiment further provides a computer storage medium, in which a computer program is stored. The computer storage medium may be a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); or various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer storage medium is executed by a processor, the control method of the self-moving device applied to the above-mentioned device is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to the description of the embodiment shown in Figure 1, which will not be repeated here.

The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

In this document, the terms "comprises," "comprising," or any other variations thereof, are intended to cover a non-exclusive inclusion of elements other than those listed and may also include additional elements not expressly listed.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

A control method for a self-moving device, wherein the self-moving device comprises a working part, wherein the working part is configured to perform a work operation, and the self-moving device further comprises: an image acquisition device, wherein the image acquisition device is configured to acquire an image in front of the self-moving device, and the self-moving device is configured to travel and/or work in a working area, wherein the method comprises:

Acquire the area type of each area in the acquired image, wherein each area at least includes: an area to be driven and a target area, wherein the area to be driven is a part of the working area, and the target area is an area adjacent to the area to be driven;

When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a part of the target area.
The method according to claim 1, characterized in that the method further comprises:

The collected image is input into the trained image recognition model to perform semantic segmentation on the image to obtain the region type of each region.
The method according to any one of claims 1 to 2, characterized in that the method further comprises:

When the working mode is a preset working mode, the step of controlling the self-moving device to change the driving mode so that the coverage range of the working unit includes a part of the target area is executed when the area type of the target area is the first area type. The preset working mode is an edge working mode or a working mode preset by the user. The edge working mode is a mode for controlling the self-moving device to move and work along the boundary of the working area.
The method according to any one of claims 1 to 3 is characterized in that the first area type includes one or more of roads, manhole covers, stone roads, hard pavement, fallen leaves and mud, or the first area type is obtained according to user instructions.
The control method according to any one of claims 1 to 4, characterized in that the method further comprises:

When a contact collision is detected during the process in which the working part covers a portion of the target area, the self-moving device is controlled to change a driving path so as to drive toward the inside of the working area.
The method according to any one of claims 1 to 5, characterized in that obtaining the region type of each region in the acquired image further comprises:

Performing depth estimation processing on the image to determine depth information corresponding to the target area, the depth information being used to represent the distance between each position of the target area and the image acquisition device;

Correspondingly, when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode includes:

The self-moving device is controlled to change the driving mode according to the area type of the target area and the depth information, so that when the area type of the target area is the first area type and the height difference between the target area and the area to be driven is less than a height threshold, the coverage range of the working unit includes a part of the target area.
The method according to any one of claims 1 to 6 is characterized in that semantic segmentation is performed on the image to obtain the region type of each region, the region type of the to-be-traveled region is a grassland type, and the region type of the target region is a first region type; accordingly,

Controlling the coverage of the working portion to include a portion of the target area comprises:

Marking the area type of the portion of the target area adjacent to the area to be driven as a grassland type;

The self-moving device is controlled to travel and/or operate along a boundary between an area defined by the grass type and an area defined by the first area type.
The method according to claim 1 is characterized in that when the area type of the target area is the second area type, the self-moving device is controlled to travel and/or work in the working area.
The method according to claim 8, wherein the second area type includes one or more of a wall, a pit, a pool, a fence, a flower bed, soil with a hardness less than a hardness threshold, an irrigation facility area, and an area of movable life.
The control method according to claim 8 or 9, characterized in that controlling the self-moving device to travel and/or work in the working area comprises:

Get the distance information along the edge;

The self-moving device is controlled to travel and/or work in the working area according to the edge distance information.
The control method according to claim 10, wherein obtaining the edge distance information comprises:

Get user command information;

The edge distance information is determined according to the user instruction information.
The control method according to any one of claims 1 to 11, characterized in that the method further comprises:

When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the self-moving device avoids the area where the second area type is located, wherein the area type of the target area includes the first area type and the second area type, and the area where the first area type is located and the area where the second area type is located are both adjacent to the area to be driven.
The control method according to any one of claims 1 to 12, characterized in that the method further comprises:

When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the driving distance of the working unit is less than the boundary length between the area where the first area type is located and the area to be driven, wherein the driving distance is the distance traveled by the self-moving device when the coverage range of the working unit includes a part of the target area.
The control method according to any one of claims 1 to 13, characterized in that before obtaining the area type of each area in the acquired image, the self-moving device travels along a first path;

When the area type of the target area is the first area type, controlling the self-moving device to change the driving mode includes:

The self-moving device is controlled to continue traveling along the first path for a period of time, and after traveling for a period of time, the traveling path is changed so that the coverage range of the working unit includes a part of the target area.
A self-moving device, the self-moving device comprising:

A working unit configured to perform a work operation;

An image acquisition device, wherein the image acquisition device is configured to acquire an image in front of the mobile device;

a controller, connected to the working part and the image acquisition device by signal, the controller being configured to control the self-moving device to travel and/or work in a working area,

The invention is characterized by comprising:

The controller acquires the area type of each area in the acquired image, wherein each area at least includes: an area to be driven and a target area, wherein the area to be driven is a part of the working area, and the target area is an area adjacent to the area to be driven;

When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a part of the target area.
A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and the computer program is loaded and executed by a processor to implement the control method of a self-moving device as described in any one of claims 1 to 14.