JP2024073232A - Image processing apparatus, image processing method, and control program - Google Patents

Abstract

To provide an image processing apparatus, an image processing method, and a control program capable of determining whether absence of a corner area of a medium has occurred in an image, simply with high accuracy.SOLUTION: An image processing apparatus includes: an acquisition unit which acquires an input image obtained by imaging a medium; a detection unit which detects a plurality of corner areas each estimated to include corners of the medium in the input image; a determination unit which determines whether absence of image has occurred in each of the corner areas by comparing the corner areas detected by the detection unit; and an output unit which outputs information on a result determined by the determination unit.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image processing device, an image processing method, and a control program.

Conventionally, image reading devices such as scanners are used to capture an image of a medium and generate an image. In such image reading devices, there are cases where the medium is placed outside the imaging range, or where a corner of the medium being imaged is folded by the user. In such cases, an image is generated by the image reading device, resulting in a missing image in which the corner of the medium is not included.

An image forming device that forms an original image on recording paper has been disclosed (see Patent Document 1). This image forming device detects each edge of the original image, and detects the presence or absence of an inclined edge at the corner of the effective area where image formation is performed in the original image. When the image forming device detects an inclined edge, it determines whether the inclined edge is present within the effective area, and if it is determined that the inclined edge is present within the effective area, it prohibits the saving of the original image or the formation of the original image.

An image reading device is disclosed that detects edge pixels from an input image, detects straight lines from the edge pixels, and determines whether edge pixels that are not within a specified distance from the straight lines represent the edge of a document (see Patent Document 2).

Patent No. 6561952 JP 2014-147046 A

Image processing devices are required to easily and accurately determine whether image loss has occurred in the corner areas of the medium in an image.

The object of the present invention is to provide an image processing device, an image processing method, and a control program that can easily and accurately determine whether or not image loss has occurred in the corner areas of a medium in an image.

An image processing device according to one aspect of the present invention has an acquisition unit that acquires an input image of a medium, a detection unit that detects multiple corner areas in the input image that are estimated to each include a corner of the medium, a determination unit that determines whether or not image loss occurs in each of the multiple corner areas by comparing the multiple corner areas detected by the detection unit, and an output unit that outputs information related to the determination result by the determination unit.

An image processing device according to one aspect of the present invention has an acquisition unit that acquires multiple input images of a medium, a detection unit that detects corner areas in each of the multiple input images that are estimated to include a corner of the medium, a determination unit that determines whether or not an image is missing in the corner area by comparing the corner areas detected by the detection unit, and an output unit that outputs information related to the determination result by the determination unit.

In an image processing method according to one aspect of the present invention, an image processing device acquires an input image of a medium, detects multiple corner areas within the input image that are estimated to each contain a corner of the medium, compares the detected multiple corner areas to determine whether or not image loss has occurred in each of the multiple corner areas, and outputs information related to the result of the determination.

In an image processing method according to one aspect of the present invention, an image processing device acquires multiple input images of a medium, detects corner areas in each of the multiple input images that are estimated to contain the corners of the medium, compares the detected corner areas to determine whether image loss has occurred in the corner areas, and outputs information related to the results of the determination.

A control program according to one aspect of the present invention is a control program for an image processing device, which causes the image processing device to acquire an input image of a medium, detect multiple corner areas within the input image that are estimated to each include a corner of the medium, compare the detected multiple corner areas to determine whether or not image loss has occurred in each of the multiple corner areas, and output information related to the result of the determination.

A control program according to one aspect of the present invention is a control program for an image processing device, which acquires multiple input images of a medium, detects corner areas in each of the multiple input images that are estimated to include the corners of the medium, compares the detected corner areas to determine whether image loss has occurred in the corner areas, and outputs information related to the result of the determination.

According to the present invention, an image processing device, an image processing method, and a control program are capable of easily and accurately determining whether or not image loss has occurred in a corner area of a medium in an image.

FIG. 1 is a diagram illustrating an example of a configuration of an image processing system. 2 is a diagram for explaining a transport path inside the image reading device 100. FIG. 3 is a schematic diagram for explaining each sensor for detecting a medium. FIG. 1 is a block diagram showing a schematic configuration of an image reading device 100 and the like. 1A is a diagram showing a schematic configuration of a first processing circuit 140 and the like, and FIG. 1B is a diagram showing a schematic configuration of a second processing circuit 220 and the like. 10 is a flowchart showing an example of an operation of an image reading process. 13 is a flowchart showing an example of the operation of a determination process. FIG. 2 is a schematic diagram showing an example of an input image N. 13 is a flowchart showing an example of the operation of an image loss determination process. 13 is a flowchart showing an example of the operation of a content determination process. Graphs (A), (B), and (C) show an example of the gradation values of pixels on the edge of a medium, and graph (D) is a histogram of the graph shown in (A). 13 is a flowchart illustrating an example of an operation of a content correction process. 10 is a flowchart showing an example of the operation of another image loss determination process. 13 is a flowchart showing an example of the operation of still another image loss determination process. FIG. 11 is a schematic diagram for explaining a corner distance. 13 is a flowchart showing an example of the operation of still another image loss determination process. 13 is a flowchart showing an example of another operation of a content determination process. 13 is a flowchart showing an example of the operation of still another content determination process. 13 is a flowchart showing an example of the operation of another content correction process. 13 is a flowchart showing an example of the operation of still another content correction process. 1A is a diagram showing a schematic configuration of another first processing circuit 340, and FIG. 1B is a diagram showing a schematic configuration of another second processing circuit 420. FIG.

The following describes an image processing device, an image processing method, and a control program according to one aspect of the present invention, with reference to the drawings. However, please note that the technical scope of the present invention is not limited to these embodiments, but extends to the inventions described in the claims and their equivalents.

Figure 1 is a configuration diagram of an example of an image processing system 1 according to an embodiment.

The image processing system 1 has an image reading device 100 and an information processing device 200. The image reading device 100 and the information processing device 200 are each an example of an image processing device. The image reading device 100 and the information processing device 200 are connected to each other for communication. The image reading device 100 is an image scanner that conveys a medium, which is a document, and captures an image. The medium is paper, thin paper, thick paper, a card, a booklet, a passport, or the like. The image reading device 100 may be a facsimile, a copier, a printer multifunction machine (MFP, Multifunction Peripheral), or the like. The image reading device 100 may also be a flatbed type image scanner, a facsimile, a copier, an MFP, or the like that captures an image without conveying a medium. The image reading device 100 may also be a mobile phone, a smartphone, a tablet computer, a notebook personal computer, or the like that captures an image of a person, an object, a landscape, or the like. The information processing device 200 is a personal computer, or the like. The information processing device 200 may be a mobile phone, a smartphone, a tablet computer, a notebook personal computer, or the like. The information processing device 200 may also be a server. In this case, a plurality of image reading devices 100 may be communicatively connected to one information processing device 200.

In FIG. 1, arrow A1 indicates the media transport direction, and arrow A2 indicates the width direction perpendicular to the media transport direction. In the following, upstream refers to the upstream side of the media transport direction A1, and downstream refers to the downstream side of the media transport direction A1.

The image reading device 100 includes a lower housing 101, an upper housing 102, a loading platform 103, an ejection platform 104, a first operation device 105, and a first display device 106.

The upper housing 102 is positioned to cover the top surface of the image reading device 100, and engages with the lower housing 101 by a hinge so that it can be opened and closed when removing media, cleaning the inside of the image reading device 100, etc.

The mounting table 103 engages with the lower housing 101. The mounting table 103 has a mounting surface 103a on which the medium is placed, and the medium to be fed and transported is placed thereon. On the mounting surface 103a, a side guide 103b is provided movably in a width direction A2 perpendicular to the medium transport direction. The side guide 103b is positioned according to the width of the medium placed on the mounting table 103, and regulates the width direction of the medium. In the example shown in FIG. 1, two side guides 103b are movably arranged with a gap between them so that the medium is positioned in the center in the width direction A2. Only one side guide 103b may be movably arranged so that the medium is positioned at one end in the width direction A2.

The ejection tray 104 engages with the upper housing 102 and places the ejected media thereon. The ejection tray 104 may also be provided to engage with the lower housing 101.

The first operating device 105 has an input device such as a button and an interface circuit that acquires signals from the input device, accepts input operations by a user, and outputs an operation signal corresponding to the user's input operation.

The first display device 106 is an example of an output unit. The first display device 106 has a display including a liquid crystal, an organic EL (Electro-Luminescence), or the like, and an interface circuit that outputs image data to the display, and displays the image data on the display. Note that the first display device 106 and the first operating device 105 may be provided integrally using a touch panel type input device.

Figure 2 is a diagram for explaining the transport path inside the image reading device 100.

The transport path inside the image reading device 100 includes a first media sensor 111, a feed roller 112, a separation roller 113, a second media sensor 114, a first transport roller 115, a second transport roller 116, a third media sensor 117, an imaging device 118, a third transport roller 119, and a fourth transport roller 120.

The number of each of the feed roller 112, separation roller 113, first conveyor roller 115, second conveyor roller 116, third conveyor roller 119 and/or fourth conveyor roller 120 is not limited to one, and may be multiple. In this case, the multiple feed rollers 112, separation rollers 113, first conveyor roller 115, second conveyor roller 116, third conveyor roller 119 and/or fourth conveyor roller 120 are arranged at intervals in the width direction A2.

The image reading device 100 has a so-called straight path. The top surface of the lower housing 101 forms a lower guide 101a of the medium transport path, and the bottom surface of the upper housing 102 forms an upper guide 102a of the medium transport path.

The first media sensor 111 is disposed upstream of the feed roller 112 and the separation roller 113. The first media sensor 111 has a contact detection sensor and detects whether or not a medium is placed on the placement table 103. The first media sensor 111 generates and outputs a placement signal whose signal value changes depending on whether or not a medium is placed on the placement table 103. Note that the first media sensor 111 is not limited to a contact detection sensor, and any other sensor capable of detecting the presence or absence of a medium, such as an optical detection sensor, may be used as the first media sensor 111.

The feed roller 112 is provided in the lower housing 101, and separates and feeds the media placed on the mounting table 103 from the bottom up. The separation roller 113 is a so-called brake roller or retard roller, and is provided in the upper housing 102, and is arranged opposite the feed roller 112, and is rotatable or stoppable in the direction A4 opposite to the media feed direction.

The first transport roller 115 and the second transport roller 116 are disposed downstream of the feed roller 112, facing each other, and transport the medium fed by the feed roller 112 and the separation roller 113 to the imaging device 118. Either the first transport roller 115 or the second transport roller 116 may be a driven roller that rotates following the rotation of the other roller.

The imaging device 118 is disposed downstream of the first conveying roller 115 and the second conveying roller 116, and captures an image of the medium conveyed by the first conveying roller 115 and the second conveying roller 116. The imaging device 118 includes a first imaging device 118a and a second imaging device 118b disposed opposite each other across the medium conveying path. The first imaging device 118a has a line sensor using a CIS (Contact Image Sensor) of a 1:1 optical system type having imaging elements using CMOS (Complementary Metal Oxide Semiconductor) linearly arranged in the main scanning direction. The first imaging device 118a also has a lens that forms an image on the imaging element, an A/D converter that amplifies the electrical signal output from the imaging element and performs analog/digital (A/D) conversion, and a backing (reference member) that faces the line sensor of the second imaging device 118b. The first imaging device 118a generates and outputs an input image that captures the surface of the medium being conveyed according to control from a processing circuit described later.

Similarly, the second imaging device 118b has a line sensor using a CIS of the same magnification optical system type having CMOS imaging elements arranged in a line in the main scanning direction. The second imaging device 118b also has a lens that forms an image on the imaging element, an A/D converter that amplifies and analog-to-digital (A/D) converts the electrical signal output from the imaging element, and a backing (reference member) that faces the line sensor of the first imaging device 118a. The second imaging device 118b generates and outputs an input image of the back side of the medium being transported according to control from a processing circuit described below.

The image reading device 100 may have only one of the first imaging device 118a and the second imaging device 118b arranged, and may read only one side of the medium. Also, instead of a CIS line sensor of an equal-magnification optical system type having a CMOS imaging element, a CIS line sensor of an equal-magnification optical system type having a CCD (Charge Coupled Device) imaging element may be used. Also, a reduction optical system type line sensor having a CMOS or CCD imaging element may be used.

The third transport roller 119 and the fourth transport roller 120 are arranged facing each other downstream of the imaging device 118, and discharge the medium transported by the first transport roller 115 and the second transport roller 116 and imaged by the imaging device 118 to the discharge tray 104. Either the third transport roller 119 or the fourth transport roller 120 may be a driven roller that rotates following the rotation of the other roller.

The medium placed on the loading platform 103 is transported between the lower guide 101a and the upper guide 102a in the medium transport direction A1 by the rotation of the feed roller 112 in the direction of the arrow A3 in FIG. 2, i.e., the medium feed direction. When transporting the medium, the separation roller 113 rotates or stops in the direction of the arrow A4, i.e., the opposite direction to the medium feed direction. This limits the transport of media other than the separated media (preventing double feeding).

The medium is guided by the lower guide 101a and the upper guide 102a and sent between the first transport roller 115 and the second transport roller 116. The medium is sent between the first imaging device 118a and the second imaging device 118b by the first transport roller 115 and the second transport roller 116 rotating in the directions of the arrows A5 and A6, respectively. The medium read by the imaging device 118 is discharged onto the discharge tray 104 by the third transport roller 119 and the fourth transport roller 120 rotating in the directions of the arrows A7 and A8, respectively.

Figure 3 is a schematic diagram explaining each sensor for detecting the medium.

In the example shown in FIG. 3, the number of each of the feed rollers 112, second conveying rollers 116, and fourth conveying rollers 120 is two. In this case, the number of each of the separation rollers 113, first conveying rollers 115, and third conveying rollers 119 arranged opposite the feed rollers 112, second conveying rollers 116, and fourth conveying rollers 120, respectively, is also two. In addition, the number of the second media sensors 114 is two, and the number of the third media sensors 117 is one. Note that the number of the second media sensors 114 and/or third media sensors 117 may be any number.

The second media sensor 114 detects the media transported to its position. In the example shown in FIG. 3, two second media sensors 114 are arranged side by side with a gap in the width direction A2. The second media sensor 114 is arranged downstream of the nip portion N1 between the feed roller 112 and the separation roller 113 in the media transport direction A1, and on each of both sides of the media transport path in the width direction A2 perpendicular to the media transport direction. In particular, the second media sensor 114 is arranged upstream of the imaging device 118 in the media transport direction A1, and particularly upstream of the nip portion N2 between the first transport roller 115 and the second transport roller 116. In addition, the second media sensor 114 is arranged inside (to the center) the side wall 101b of the media transport path in the width direction A2. The second media sensor 114 is positioned in the width direction A2 outside (towards the side wall 101b) the maximum size media supported by the image reading device 100 when placed on the placement table 103, that is, outside the position of the inner side surface of the side guide 103b when placed at the outermost position. The second media sensor 114 is also positioned outside the end position of the imaging range of the imaging device 118 in the width direction A2.

The second media sensor 114 includes a light emitter and a light receiver provided on one side of the media transport path, and a light guide tube provided at a position facing the light emitter and the light receiver across the media transport path. The light emitter is an LED (Light Emitting Diode) or the like, and irradiates light toward the media transport path. On the other hand, the light receiver is a photodiode or the like, and receives the light irradiated by the light emitter and guided by the light guide tube. When a medium is present at a position facing at least one of the light emitter and the light receiver, the light irradiated from the light emitter is blocked by the medium, so the light receiver does not detect the light irradiated from the light emitter. The second media sensor 114 generates and outputs a second media signal whose signal value changes depending on whether a medium is present or not at the position of the second media sensor 114 based on the intensity of the light received by the light receiver.

The third media sensor 117 is disposed downstream of the nip portion N1 between the feed roller 112 and the separation roller 113 in the media transport direction A1, and at the center of the media transport path in the width direction A2 perpendicular to the media transport direction. In particular, the third media sensor 117 is disposed downstream of the second media sensor 114 in the media transport direction A1, particularly downstream of the nip portion N2 between the first transport roller 115 and the second transport roller 116 and upstream of the imaging device 118. In addition, the third media sensor 117 is disposed between the multiple feed rollers 112 (nip portion N1) and/or between the multiple first transport rollers 115 (nip portion N2) in the width direction A2.

The third media sensor 117 includes a light emitter and a light receiver provided on one side of the media transport path, and a light guide tube provided opposite the light emitter and the light receiver across the media transport path. The light emitter is an LED or the like, and emits light toward the media transport path. On the other hand, the light receiver is a photodiode or the like, and receives the light emitted by the light emitter and guided by the light guide tube. When a medium is present in a position opposite at least one of the light emitter and the light receiver, the light emitted from the light emitter is blocked by the medium, and the light receiver does not detect the light emitted from the light emitter. The third media sensor 117 generates and outputs a third media signal whose signal value changes depending on whether a medium is present or not at the position of the third media sensor 117, based on the intensity of the light received by the light receiver.

In the second media sensor 114 and/or the third media sensor 117, a reflective member such as a mirror may be used instead of a light guide tube. In the second media sensor 114 and/or the third media sensor 117, the light emitter and the light receiver may be arranged opposite each other across the media transport path. The second media sensor 114 and/or the third media sensor 117 may detect the presence of the media using a contact detection sensor that passes a predetermined current when the media is in contact or when the media is not in contact. The second media sensor 114 and/or the third media sensor 117 may detect the presence of the media using an ultrasonic sensor.

Figure 4 is a block diagram showing the schematic configuration of the image reading device 100 and the information processing device 200.

In addition to the above-mentioned configuration, the image reading device 100 further includes a motor 121, a first communication device 122, a first storage device 130, and a first processing circuit 140.

The motor 121 includes one or more motors. The motor 121 rotates the feed roller 112, the separation roller 113, the first conveyor roller 115, the second conveyor roller 116, the third conveyor roller 119, and/or the fourth conveyor roller 120 in response to a control signal from the first processing circuit 140 to convey the medium.

The first communication device 122 is an example of an output unit and a communication unit. The first communication device 122 has an antenna for transmitting and receiving wireless signals, and a wireless communication interface circuit for transmitting and receiving signals through a wireless communication line according to a communication protocol such as a wireless LAN (Local Area Network). The first communication device 122 can communicate with the information processing device 200, and is communicatively connected to the information processing device 200 to transmit and receive various images and information. The first communication device 122 may have a wired communication interface circuit according to a communication protocol such as TCP/IP (Transmission Control Protocol/Internet Protocol), and may be connected to the information processing device 200 via a network. The first communication device 122 may also have an interface circuit conforming to a serial bus such as a USB (Universal Serial Bus), and may be connected to the information processing device 200 via a wired cable such as a USB cable.

The first storage device 130 includes a memory device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), a fixed disk device such as a hard disk, or a portable storage device such as a flexible disk or an optical disk. The first storage device 130 also stores computer programs, databases, tables, and the like used for various processes of the image reading device 100. The computer programs may be installed in the first storage device 130 from a computer-readable portable recording medium using a known setup program or the like. The portable recording medium is, for example, a CD-ROM (compact disc read only memory), a DVD-ROM (digital versatile disc read only memory), etc.

The first processing circuit 140 operates based on a program previously stored in the first storage device 130. The processing circuit is, for example, a CPU (Central Processing Unit). The first processing circuit 140 may be a DSP (digital signal processor), an LSI (large scale integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or the like.

The first processing circuit 140 is connected to the first operating device 105, the first display device 106, the first medium sensor 111, the second medium sensor 114, the third medium sensor 117, the imaging device 118, the motor 121, the first communication device 122, the first storage device 130, and the like, and controls each of these components. Based on the signals received from each sensor, the first processing circuit 140 performs drive control of the motor 121, imaging control of the imaging device 118, and the like, acquires an input image from the imaging device 118, and transmits it to the information processing device 200 via the first communication device 122.

The information processing device 200 has a second display device 202, a second operation device 201, a second communication device 203, a second storage device 210, a second processing circuit 220, etc.

The second operation device 201 has input devices such as a keyboard and a mouse, and an interface circuit that acquires signals from the input devices, accepts operations by a user, and outputs a signal corresponding to the user's input to the second processing circuit 220.

The second display device 202 is an example of an output unit. The second display device 202 has a display made of liquid crystal, organic EL, or the like, and an interface circuit that outputs image data to the display, and displays image data on the display according to instructions from the second processing circuit 220. Note that the second display device 202 and the second operating device 201 may be provided integrally using a touch panel type input device.

The second communication device 203 is an example of an output unit and a communication unit. The second communication device 203 has an interface circuit similar to that of the first communication device 122. The second communication device 203 is capable of communicating with the image reading device 100, and is communicatively connected to the image reading device 100 to transmit and receive various images and information.

The second storage device 210 is an example of a storage unit, and includes a memory device such as a RAM or ROM, a fixed disk device such as a hard disk, or a portable storage device such as an optical disk. The second storage device 210 also stores computer programs, databases, tables, and the like used for various processes of the information processing device 200. Computer programs may be installed in the second storage device 210 from a computer-readable portable recording medium such as a CD-ROM or DVD-ROM using a known setup program, etc.

The second processing circuit 220 operates based on a program previously stored in the second storage device 210. The processing circuit is, for example, a CPU. A DSP, an LSI, an ASIC, an FPGA, etc. may also be used as the second processing circuit 220.

The second processing circuit 220 is connected to the second operation device 201, the second display device 202, the second communication device 203, the second storage device 210, etc., and controls each of these components. The second processing circuit 220 controls each device, acquires an input image from the image reading device 100, and performs image processing on the acquired input image.

Figure 5 (A) is a diagram showing the schematic configuration of the first storage device 130 and the first processing circuit 140 of the image reading device 100.

As shown in FIG. 5(A), the first storage device 130 stores a control program 131, a transmission program 132, and the like. Each of these programs is a functional module implemented by software running on a processor. The first processing circuit 140 reads each program stored in the first storage device 130 and operates according to the read program. In this way, the first processing circuit 140 functions as a control unit 141 and a transmission unit 142.

Figure 5 (B) is a diagram showing the schematic configuration of the second storage device 210 and the second processing circuit 220 of the information processing device 200.

As shown in FIG. 5(B), the second storage device 210 stores an acquisition program 211, an extraction program 212, a detection program 213, an image loss determination program 214, a content determination program 215, a correction program 216, a notification program 217, an output control program 218, and the like. Each of these programs is a functional module implemented by software that runs on a processor. The second processing circuit 220 reads each program stored in the second storage device 210 and operates according to each program that has been read. As a result, the second processing circuit 220 functions as an acquisition unit 221, an extraction unit 222, a detection unit 223, an image loss determination unit 224, a content determination unit 225, a correction unit 226, a notification unit 227, and an output control unit 228. The image loss determination unit 224 is an example of a determination unit.

Figure 6 is a flowchart showing an example of the operation of the image reading process of the image reading device 100.

Below, an example of the operation of the image reading process of the image reading device 100 will be described with reference to the flowchart shown in FIG. 6. Note that the flow of the operation described below is executed mainly by the first processing circuit 140 in cooperation with each element of the image reading device 100 based on a program previously stored in the first storage device 130.

First, the control unit 141 waits until a user inputs an instruction to read a medium using the first operating device 105 or the information processing device, and an operation signal instructing the user to read a medium is received from the first operating device 105 or the first communication device 122 (step S101).

Next, the control unit 141 acquires a first medium signal from the first medium sensor 111, and determines whether or not a medium is placed on the placement table 103 based on the acquired first medium signal (step S102). If no medium is placed on the placement table 103, the control unit 141 ends the series of steps.

On the other hand, if a medium is placed on the placement table 103, the control unit 141 drives the motor 121. As a result, the control unit 141 rotates the feed roller 112, the separation roller 113, the first conveyor roller 115, the second conveyor roller 116, the third conveyor roller 119, and/or the fourth conveyor roller 120 to feed and convey the medium (step S103).

Next, the control unit 141 sets the detection flag to OFF (step S104). The detection flag is a flag that indicates whether the second medium sensor 114 has detected a medium, i.e., whether the medium being transported has passed outside the imaging range of the imaging device 118.

Next, the control unit 141 determines whether the medium has passed outside the imaging range of the imaging device 118 (step S105). The control unit 141 periodically acquires a second medium signal from each second medium sensor 114. If the signal value of the second medium signal indicates that a medium is present, the control unit 141 determines that the medium has passed the position of the second medium sensor 114 that transmitted the second medium signal, and determines that the medium has passed outside the imaging range of the imaging device 118. If the medium has not passed outside the imaging range of the imaging device 118, the control unit 141 proceeds to step S107.

On the other hand, if the medium passes outside the imaging range of the imaging device 118, the control unit 141 sets the detection flag to ON and stores information indicating the second medium sensor 114 that detected the medium in the first storage device 130. The control unit 141 also stores the imaging start time of the medium and the detection time when the second medium sensor 114 detected the medium in the first storage device 130 (step S106).

Next, the control unit 141 determines whether the rear end of the medium has passed the imaging position of the imaging device 118 (step S107). The control unit 141 periodically acquires a third medium signal from the third medium sensor 117. The control unit 141 determines that the rear end of the medium has passed the position of the third medium sensor 117 when the signal value of the third medium signal changes from a value indicating the presence of the medium to a value indicating the absence of the medium. The control unit 141 determines that the rear end of the medium has passed the imaging position when a predetermined time has elapsed since the rear end of the medium passed the position of the third medium sensor 117. The predetermined time is set to the time required for the medium to move from the position of the third medium sensor 117 to the imaging position of the imaging device 118. Note that the control unit 141 may determine that the rear end of the medium has passed the imaging position when a predetermined time has elapsed since the start of feeding the medium.

Next, the transmission unit 142 acquires an input image from the imaging device 118, stores the acquired input image in the first storage device 130, and outputs it by transmitting it to the information processing device 200 via the first communication device 122 (step S108). Note that the transmission unit 142 may acquire image data from the imaging device 118 for each predetermined unit (e.g., 3 lines) and transmit it to the information processing device 200, rather than acquiring the entire input image from the imaging device 118 and transmitting it to the information processing device 200. In that case, the information processing device 200 combines each image data to generate an input image.

Next, the control unit 141 determines whether or not a medium remains on the placement table 103 based on the first medium signal obtained from the first medium sensor 111 (step S109). If a medium remains on the placement table 103, the control unit 141 returns the process to step S104 and repeats the processes of steps S104 to S109.

On the other hand, if no media remains on the mounting table 103, the control unit 141 stops the motor 121. As a result, the control unit 141 stops the feed roller 112, the separation roller 113, the first conveyor roller 115, the second conveyor roller 116, the third conveyor roller 119, and/or the fourth conveyor roller 120 (step S110), and the series of steps ends.

The processing of steps S104 to S106 may be omitted, and the control unit 141 may not need to determine whether the medium has passed outside the imaging range of the imaging device 118.

Figure 7 is a flowchart showing an example of the operation of the determination process of the information processing device 200.

Below, an example of the operation of the determination process of the information processing device 200 will be described with reference to the flowchart shown in FIG. 7. Note that the flow of the operation described below is executed mainly by the second processing circuit 220 in cooperation with each element of the information processing device 200 based on a program previously stored in the second storage device 210. The determination process is executed every time an input image is received from the image reading device 100.

First, the acquisition unit 221 acquires an input image by receiving it from the image reading device 100 via the second communication device 203, and stores it in the second storage device 210 (step S201). As described above, the determination process is executed each time an input image is received from the image reading device 100, and the acquisition unit 221 acquires multiple input images and stores them in the second storage device 210. An input image acquired before or after the input image to be processed, i.e., an input image in which a medium is captured before or after the input image to be processed, is an example of a second input image.

Next, the extraction unit 222 extracts edge pixels from the input image (step S202). The extraction unit 222 calculates, for each horizontal line extending in the horizontal direction (main scanning direction) in the input image, the absolute value of the difference in gradation values between the pixels on both sides of each pixel in each horizontal line, starting from the left end, as an adjacent difference value. The extraction unit 222 extracts pixels in each horizontal line whose adjacent difference value exceeds a gradation threshold as edge pixels. The gradation value is a luminance value or a color value (R value, G value, or B value), etc. The gradation threshold is set to a luminance value difference (e.g., 20) that allows a person to visually distinguish the difference in luminance on the image. The extraction unit 222 determines the edge pixel detected first in each horizontal line, i.e., the pixel located on the leftmost side, as the left edge pixel, and the edge pixel detected last in each horizontal line, i.e., the pixel located on the rightmost side, as the right edge pixel. The extraction unit 222 may detect edge pixels in order from the right end, and may determine that the first edge pixel detected in each horizontal line is the right end edge pixel.

Similarly, the extraction unit 222 calculates, for each vertical line extending in the vertical direction (sub-scanning direction) in the input image, the absolute value of the difference in gradation values between the pixels adjacent to each pixel in each vertical line, starting from the top end, as an adjacent difference value. The extraction unit 222 extracts pixels in each vertical line whose adjacent difference value exceeds the gradation threshold as edge pixels. The extraction unit 222 determines the edge pixel detected first in each vertical line, i.e., the pixel located at the top, as the upper edge pixel, and the edge pixel detected last in each vertical line, i.e., the pixel located at the bottom, as the lower edge pixel. The extraction unit 222 may detect edge pixels starting from the bottom end, and determine the edge pixel detected first in each vertical line as the lower edge pixel. Hereinafter, the left edge pixel, right edge pixel, upper edge pixel, and lower edge pixel may be collectively referred to as end edge pixels.

The extraction unit 222 may calculate, as the adjacent difference value, the absolute value of the difference in gradation values between two pixels that are a predetermined distance away from each pixel in the input image in the horizontal or vertical direction. The extraction unit 222 may also extract edge pixels by comparing the gradation value of each pixel in the input image with a threshold value. For example, when the gradation value of a specific pixel is less than the threshold value and the gradation value of a pixel adjacent to the specific pixel in the horizontal or vertical direction or a pixel that is a predetermined distance away from the specific pixel is equal to or greater than the threshold value, the extraction unit 222 extracts the specific pixel as an edge pixel.

In addition, the extraction unit 222 may extract edge pixels at regular intervals (e.g., 4 pixels) in the horizontal or vertical direction in the input image, instead of extracting edge pixels for all pixels in the input image. The extraction unit 222 extracts target lines for detecting edge pixels at regular intervals from among horizontal or vertical lines in the input image, and extracts edge pixels for the extracted target lines. The extraction unit 222 may also group the pixels in the input image into groups of multiple pixels arranged continuously in the horizontal or vertical direction, and extract edge pixels based on the statistical values (average, median, minimum, maximum, or variance, etc.) of the gradation values of each pixel in each group. In this case, the extraction unit 222 calculates, as the adjacent difference value, the absolute value of the difference between the statistical value for each group and the statistical value for the group adjacent to that group, in order from the left end side or the top end side, for each horizontal or vertical line in the input image. The extraction unit 222 detects the representative pixel of the group (e.g., the pixel arranged in the center of the group, or the pixel arranged in the position closest to the adjacent group) whose adjacent difference value exceeds the gradation threshold as an edge pixel. As a result, the extraction unit 222 can reduce the time required to detect the edge of the medium, and can reduce the processing time and processing load of the medium reading process. The extraction unit 222 may extract edge pixels using any known edge detection method other than the above-mentioned method.

Figure 8 is a schematic diagram showing an example of an input image N in which a medium is captured.

The input image N shown in FIG. 8 includes a medium M with rounded corners. In the medium M, the upper left corner, lower left corner, and lower right corner are rounded, but the upper right corner is cut diagonally (in a straight line). In the input image N, a left edge pixel E1 is detected from the left side of the medium M, a right edge pixel E2 is detected from the right side of the medium M, an upper edge pixel E3 is detected from the top side of the medium M, and a lower edge pixel E4 is detected from the bottom side of the medium M.

Next, the detection unit 223 detects four straight lines corresponding to the four sides of the medium from the end edge pixels extracted by the extraction unit 222 (step S203). The detection unit 223 detects a left end straight line corresponding to the left side of the medium from the left end edge pixels, a right end straight line corresponding to the right side of the medium from the right end edge pixels, an upper end straight line corresponding to the upper side of the medium from the upper end edge pixels, and a lower end straight line corresponding to the lower side of the medium from the lower end edge pixels. The detection unit 223 detects straight lines (line segments) from each end edge pixel using, for example, the least squares method. The detection unit 223 may detect straight lines using a Hough transform.

When multiple straight lines are detected from each of the left edge pixels, right edge pixels, top edge pixels, or bottom edge pixels, the detection unit 223 detects the straight line with the largest number of end edge pixels present within a predetermined distance from each straight line as the left edge straight line, right edge straight line, top edge straight line, or bottom edge straight line. The predetermined distance is set to a sufficiently small distance (for example, 3 pixels). Alternatively, the detection unit 223 extracts all combinations of four straight lines detected from each end edge pixel. The detection unit 223 detects the straight lines included in the combination with the smallest sum of the difference between the angle between each of the straight lines that are approximately perpendicular to each other and 90° among the extracted combinations as the left edge straight line, right edge straight line, top edge straight line, and bottom edge straight line. A combination of either the left edge straight line or the right edge straight line and either the top edge straight line or the bottom edge straight line is an example of two straight lines that are approximately perpendicular to each other detected from edge pixels.

In the input image N shown in FIG. 8, each corner of the medium M is rounded or cut diagonally (straight), so some of the end edge pixels do not lie on a straight line. Therefore, multiple straight lines are detected from each end edge pixel. In the input image N, a left edge straight line L1, a right edge straight line L2, an upper edge straight line L3, and a lower edge straight line L4 are detected from the left edge pixel E1, the right edge pixel E2, the upper edge pixel E3, and the lower edge pixel E4, respectively.

Next, the detection unit 223 detects multiple corner regions, particularly four corner regions, each of which is estimated to include a corner of the medium in the input image (step S204). The detection unit 223 detects the intersection of the left end line and the top end line as the top left intersection, the intersection of the right end line and the top end line as the top right intersection, the intersection of the left end line and the bottom end line as the bottom left intersection, and the intersection of the right end line and the bottom end line as the bottom right intersection. The top left intersection, top right intersection, bottom left intersection, and bottom right intersection are examples of intersections of two lines that are approximately perpendicular to each other and detected from edge pixels.

The detection unit 223 detects the area in the input image that includes the upper left intersection as an upper left corner area that is estimated to include the upper left corner of the medium. The detection unit 223 detects the area in the input image that includes the upper right intersection as an upper right corner area that is estimated to include the upper right corner of the medium. The detection unit 223 detects the area in the input image that includes the lower left intersection as a lower left corner area that is estimated to include the lower left corner of the medium. The detection unit 223 detects the area in the input image that includes the lower right intersection as a lower right corner area that is estimated to include the lower right corner of the medium. The detection unit 223 sets square areas of the same size as each corner area. The detection unit 223 sets each corner area so that the distances from the ends of the corner areas opposite the center of the medium to each intersection are the same in the horizontal and vertical directions, and the distances from the ends of the corner areas closer to the center of the medium to each intersection are the same in the horizontal and vertical directions. It is preferable that each corner region is set so that each intersection point is located near the end of the corner region opposite the center of the medium.

The detection unit 223 stores position information (coordinate information) within the input image that indicates the detected corner area in the second storage device 210. That is, the detection unit 223 detects corner areas that are estimated to include the corners of the medium within each of the multiple input images, and stores position information that indicates the detected corner areas in the second storage device 210.

In the example shown in FIG. 8, the upper left intersection V1, upper right intersection V2, lower left intersection V3, and lower right intersection V4 are detected from the left end line L1 and the upper end line L3, the right end line L2 and the upper end line L3, the left end line L1 and the lower end line L4, and the right end line L2 and the lower end line L4, respectively. In addition, square regions including the upper left intersection V1, the upper right intersection V2, the lower left intersection V3, and the lower right intersection V4, respectively, are detected as the upper left corner region R1, the upper right corner region R2, the lower left corner region R3, and the lower right corner region R4.

Next, the image loss determination unit 224 executes image loss determination processing (step S205). In the image loss determination processing, the image loss determination unit 224 determines whether or not image loss has occurred in the corner area detected by the detection unit 223, and identifies the corner area in which image loss has occurred. Image loss refers to a state in which part of the medium, particularly the corners of the medium, are not included in the image. The image loss determination processing will be described in detail later.

The following steps S206 to S214 are performed for each corner region detected by the detection unit 223. First, the content determination unit 225 determines whether or not the image loss determination unit 224 has determined that image loss has occurred in the target corner region in the image loss determination process (step S206). If it has been determined that no image loss has occurred in the target corner region, the content determination unit 225 does not perform any particular process, but instead performs steps S206 to S214 for the other corner regions.

On the other hand, if it is determined that image loss has occurred in the target corner region, the content determination unit 225 executes a content determination process (step S207). In the content determination process, the content determination unit 225 determines whether or not content is present around the edge of the medium in the corner region determined by the image loss determination unit 224 to have image loss. Furthermore, if the content determination unit 225 determines that content is present, it identifies the type of content (characters, lines, patterns, images, etc.). The content determination process will be described in detail later.

Next, the content determination unit 225 determines whether or not content is present around the edge of the medium in the corner area in which the image loss determination unit 224 has determined that image loss has occurred during the content determination process (step S208).

When the content determination unit 225 determines that no content exists around the edge of the medium in the corner area to be determined, the correction unit 226 forms (restores, corrects) the corner in the corner area determined to have image loss, and corrects the corner area (step S209). For example, the correction unit 226 corrects the corner area determined to have image loss by using the corner area determined to have no image loss by the content determination unit 225. The correction unit 226 generates a correction corner area whose orientation (arrangement direction) matches that of the corner area where image loss occurs by rotating the corner area where image loss does not occur by 90°, 180°, or 270°, or by inverting it vertically, horizontally, or diagonally. The correction unit 226 corrects the corner area where image loss occurs by arranging (copying) the correction corner area in the corner area where image loss occurs.

The correction unit 226 may correct only the missing portion in the corner region where the image loss occurs. In this case, the correction unit 226 sets the area outside each end edge pixel (opposite the center of the medium) in the corner region where the image loss occurs as the correction target area. In the upper left corner region, the area outside each end edge pixel is the area to the left and above each end edge pixel. In the upper right corner region, the area outside each end edge pixel is the area to the right and above each end edge pixel. In the lower left corner region, the area outside each end edge pixel is the area to the left and below each end edge pixel. In the lower right corner region, the area outside each end edge pixel is the area to the right and below each end edge pixel. The correction target area is set to include the end edge pixel itself. The correction target area may be set not to include the end edge pixel itself. The correction unit 226 corrects the corner region where the image loss occurs by arranging (copying) the corresponding area of the correction corner region in the correction target area of the corner region where the image loss occurs. This allows the correction unit 226 to restore the edge of the medium within the corner area while maintaining the content (non-missing part) contained in the corner area where the image loss occurs.

The correction unit 226 may also determine whether or not content exists in a corner area where no image loss occurs, in a manner similar to the content determination process, and correct the corner area where image loss occurs using a corner area where it has been determined that no content exists. This allows the correction unit 226 to prevent erroneous copying of content contained in another corner area to the corner area where image loss occurs.

In the example shown in FIG. 8, it is determined that no image loss occurs in the upper left corner region R1, the lower left corner region R3, and the lower right corner region R4, and that image loss occurs in the upper right corner region R2. The correction unit 226 corrects the upper right corner region R2 by rotating the upper left corner region R1 by 90° clockwise and arranging the upper left intersection point V1 in the upper right corner region R2 so that it coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by rotating the lower left corner region R3 by 180° clockwise and arranging the lower left intersection point V3 in the upper right corner region R2 so that it coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by rotating the lower right corner region R4 by 270° clockwise and arranging the lower right intersection point V4 in the upper right corner region R2 so that it coincides with the upper right intersection point V2.

Alternatively, the correction unit 226 corrects the upper right corner region R2 by flipping the upper left corner region R1 horizontally across a predetermined vertical line, and then arranging the upper left intersection point V1 in the upper right corner region R2 so that the upper left intersection point V1 coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by flipping the lower left corner region R3 across a diagonal line that extends from the upper left to the lower right and is inclined at 45 degrees with respect to the horizontal and vertical lines, and then arranging the lower left intersection point V3 in the upper right corner region R2 so that the lower left intersection point V3 coincides with the upper right intersection point V2. Alternatively, the correction unit 226 corrects the upper right corner region R2 by flipping the lower right corner region R4 vertically across a predetermined horizontal line, and then arranging the lower right intersection point V4 in the upper right corner region R2 so that the upper right intersection point V2 coincides with the upper right intersection point V2.

On the other hand, if the content determination unit 225 determines in step S208 that content exists around the edge of the medium in the target corner area, the correction unit 226 determines whether or not the content can be corrected (step S210). If the type of content identified by the content determination unit 225 is a character, a line, or a pattern, the correction unit 226 determines that the content can be corrected. On the other hand, if the type of content identified by the content determination unit 225 is something other than a character, a line, or a pattern (such as an image), the correction unit 226 determines that the content cannot be corrected.

If it is determined that the content can be corrected, the correction unit 226 corrects the corner area in which it is determined that image loss has occurred, in a manner similar to the processing of step S209 (step S211).

Next, the correction unit 226 executes a content correction process (step S212). In the content correction process, the correction unit 226 corrects a corner region in which the image loss determination unit 224 has determined that an image loss has occurred and the content determination unit 225 has determined that content exists around the edge of the medium, based on the surrounding pixels of the edge of the medium within the corner region. Details of the content correction process will be described later.

On the other hand, if it is determined in step S210 that the content cannot be corrected, the correction unit 226 does not correct the corner area (step S213). In this way, the correction unit 226 corrects a corner area in which the image loss determination unit 224 has determined that image loss has occurred, and in which the content determination unit 225 has determined that no content exists around the edge of the medium. On the other hand, the correction unit 226 does not correct a corner area in which the image loss determination unit 224 has determined that image loss has occurred, and in which the content determination unit 225 has determined that content exists around the edge of the medium. In this way, the correction unit 226 can appropriately restore the damaged corner area in the input image, while suppressing damage to the content in the input image by correcting the corner area.

Next, the notification unit 227 notifies the user that image loss has occurred by displaying the information on the second display device 202 or transmitting the information to the terminal device used by the user via the second communication device 203 (step S214). In this way, the notification unit 227 notifies the user that image loss has occurred when there is a corner area where the content determination unit 225 has determined that content exists around the edge of the medium, particularly when there is a corner area where irreparable content exists. This allows the user to recognize that image loss has occurred in the input image. As a result, the user can repair the state of the medium as necessary and then have the image reading device 100 reread the medium, and the image processing system 1 can improve user convenience.

When the processes of steps S206 to S214 have been performed for all corner areas detected by the detection unit 223, the output control unit 228 outputs the input image and/or image loss determination result (step S215) and ends the series of steps. The output control unit 228 outputs the input image and/or image loss determination result by displaying it on the second display device 202 or transmitting it to the terminal device used by the user via the second communication device 203. When the input image has not been corrected by the correction unit 226, the output control unit 228 outputs the input image received from the image reading device 100, and when the input image has been corrected by the correction unit 226, the output control unit 228 outputs the input image corrected by the correction unit 226. The image loss determination result includes whether or not image loss occurs in the input image, whether or not content exists around the edge of the medium in the corner area where it is determined that image loss occurs, and/or whether or not the corner area where image loss occurs has been corrected. The input image and/or image loss determination result is an example of information on the determination result by the image loss determination unit 224.

The processes of steps S206 to S214 may be omitted, and the second processing circuit 220 may not perform correction processing on the corner region where the image loss occurs. The processes of steps S210 to S212 may be omitted, and the correction unit 226 may not correct the corner region when content exists around the edge of the medium in the corner region, regardless of whether the content is correctable or not. The processes of steps S210, S213, and S214 may be omitted, and the correction unit 226 may correct the corner region when content exists around the edge of the medium in the corner region, regardless of whether the content is correctable or not. The processes of steps S212 and/or S214 may be omitted. In the process of step S204, the detection unit 223 may detect any number of corner regions (any one, two, or three) instead of detecting four corner regions. The processes of steps S206 to S214 may be performed only on any number of corner regions (any one, two, or three) instead of all corner regions.

Figure 9 is a flowchart showing an example of the operation of the image loss determination process. The image loss determination process is executed in step S205 of the determination process shown in Figure 7.

First, the image missing determination unit 224 calculates the similarity between the multiple corner regions detected by the detection unit 223 (step S301). The image missing determination unit 224 extracts all combinations (pairs) consisting of two corner regions from all corner regions detected by the detection unit 223. The image missing determination unit 224 calculates the similarity between the two corner regions included in each combination for all the extracted combinations. The image missing determination unit 224 aligns the orientation of the two corner regions by rotating at least one of the two corner regions by 90°, 180°, or 270°, or by inverting it up and down, left and right, or diagonally. The image missing determination unit 224 calculates, for example, the normalized cross-correlation (NCC) of the two corner regions as the similarity. The image missing determination unit 224 may calculate the inverse of the Sum of Squared Difference (SSD) or the inverse of the Sum of Absolute Difference (SAD) of the two corner regions as the similarity.

Next, the image loss determination unit 224 classifies the multiple corner regions into groups based on the calculated similarity between the multiple corner regions (step S302). The image loss determination unit 224 classifies two corner regions whose similarity is equal to or greater than the similarity threshold into the same group. The image loss determination unit 224 also classifies into a specific group a corner region whose similarity to a corner region classified into a specific group is equal to or greater than the similarity threshold. The similarity threshold is set to a value between the similarity calculated for corner regions without image loss and the similarity calculated for corner regions with image loss, based on a prior experiment.

Next, the image loss determination unit 224 determines whether the multiple corner areas detected by the detection unit 223 are classified into two or more groups (step S303).

When all corner regions detected by the detection unit 223 are classified into one group, the image loss determination unit 224 determines that no image loss has occurred in any corner region (step S304), and ends the series of steps.

On the other hand, if the corner areas detected by the detection unit 223 are classified into two or more groups, the image loss determination unit 224 determines that image loss has occurred in one of the corner areas (step S305). The media to be transported include media with right-angled corners such as PPC (Plain Paper Copier) paper or business cards, and media with rounded corners such as ID (Identification) cards or passports specified by ISO (International Organization for Standardization)/IEC (International Electrotechnical Commission) 7810. For both media with right-angled corners and media with rounded corners, the shape of the four corners is likely to be the same. On the other hand, it is unlikely that all corners of the medium are transported outside the imaging range of the imaging device 118 in the same way, and it is unlikely that all corners of the medium are folded in the same way, and it is unlikely that the same type of image loss will occur in all corner areas. Therefore, if the shapes of all corner regions are similar, the image loss determination unit 224 determines that no image loss has occurred in any corner region, and if the shape of any corner region differs from the shape of the other corner regions, it determines that image loss has occurred in one of the corner regions.

In this way, the image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner areas by comparing the multiple corner areas detected by the detection unit 223. This allows the image loss determination unit 224 to correctly determine whether or not image loss has occurred even if the shape of the corner of the medium being transported is not recognized in advance. Therefore, the image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes, such as right-angled corners or rounded corners.

In particular, the image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner regions based on the number of groups classified based on the degree of similarity between the multiple corner regions. This allows the image loss determination unit 224 to more accurately determine whether or not image loss has occurred for media that have corners of various shapes, such as right angles or rounded corners.

Next, the image loss determination unit 224 identifies corner areas where image loss occurs based on the number of corner areas belonging to each classified group (step S306), and ends the series of steps. The image loss determination unit 224 determines that no image loss occurs in corner areas belonging to the group with the largest number of corner areas among the classified groups, and determines that image loss occurs in corner areas belonging to other groups. As described above, since the same type of image loss is unlikely to occur in each corner area, it is highly likely that no image loss occurs in corner areas that have similar shapes, and that image loss occurs in corner areas that have shapes that are dissimilar to the shapes of the other corner areas. The image loss determination unit 224 can identify corner areas where image loss occurs with high accuracy by using the number of corner areas belonging to each classified group.

In addition, in the process of step S204, if the detection unit 223 detects only one corner region, the image loss determination unit 224 may determine whether or not image loss has occurred in that corner region based only on that corner region. For example, if the distance between the intersection of the two straight lines corresponding to the edge of the medium in that corner region and the edge edge pixel located closest to the intersection is equal to or less than a predetermined threshold, the image loss determination unit 224 determines that the corner is a right angle and that image loss has not occurred. The threshold is set to a sufficiently small value (e.g., 3 pixels). On the other hand, if the distance between the intersection of the two straight lines corresponding to the edge of the medium in that corner region and the edge edge pixel located closest to the intersection is greater than the threshold, the image loss determination unit 224 determines that the corner is not a right angle and that image loss has occurred.

Figure 10 is a flowchart showing an example of the operation of the content determination process. The content determination process is executed in step S207 of the determination process shown in Figure 7.

First, the content determination unit 225 detects the medium edge in the corner region where the image loss determination unit 224 has determined that image loss has occurred (step S401). The content determination unit 225 detects the end edge pixel included in the corner region where the image loss has been determined to have occurred as the medium edge.

The content determination unit 225 may detect only the edge pixels corresponding to the missing portion among the edge pixels included in the corner region where the image loss occurs as the medium edge. In this case, the content determination unit 225 detects only the edge pixels not corresponding to the edge pixels included in the corner region where the image loss occurs as the medium edge among the edge pixels included in the corner region where the image loss occurs as the medium edge. The content determination unit 225 selects one of the corner regions where the image loss does not occur. The content determination unit 225 generates a correction corner region for the selected corner region in the same manner as the process of step S209, with the orientation of the selected corner region being aligned with the corner region where the image loss occurs. The content determination unit 225 detects, as the medium edge, the edge pixels that do not have a pixel corresponding to the edge pixel in the correction corner region within the first distance among the edge pixels included in the corner region where the image loss occurs. The first distance is set to the number of pixels (for example, 10 pixels) that can be considered to have the same shape as the corner.

Next, the content determination unit 225 calculates characteristic values of a plurality of pixels corresponding to the medium edge in the corner region where it is determined that image loss has occurred (step S402). The content determination unit 225 extracts pixels located in the corner region inside (toward the center of the medium) each end edge pixel detected in step S401 and within a second distance from each end edge pixel as a plurality of pixels corresponding to the medium edge. In the upper left corner region, the region inside each end edge pixel is the region to the right and below each end edge pixel. In the upper right corner region, the region inside each end edge pixel is the region to the left and below each end edge pixel. In the lower left corner region, the region inside each end edge pixel is the region to the right and above each end edge pixel. In the lower right corner region, the region inside each end edge pixel is the region to the left and above each end edge pixel. The plurality of pixels corresponding to the medium edge are set so as not to include the end edge pixel itself. The plurality of pixels corresponding to the medium edge may be set so as to include the end edge pixel itself. In the following, the pixels that correspond to the edge of the medium may be referred to as edge surrounding pixels.

For example, the content determination unit 225 calculates the difference between the maximum and minimum gradation values of each edge peripheral pixel as the characteristic value of the edge peripheral pixel. The content determination unit 225 may also calculate the difference between the maximum or minimum gradation value and the average or median of each edge peripheral pixel as the characteristic value of the edge peripheral pixel.

Figure 11 (A) is a graph G1 showing an example of the gradation values of pixels on the edge of a medium that contains characters or lines as content.

The horizontal axis of graph G1 indicates the (horizontal) position of the pixel on the edge of the medium in the image, and the vertical axis indicates the gradation value of each pixel. In the example shown in graph G1, a white background (gradation value U1) exists in region S1, a black ruled line (gradation value U2) exists in region S2, and black characters (gradation value U3) exists in region S3. In graph G1, the difference between the maximum and minimum gradation values of the pixels around each edge is the difference between the gradation value U1 of the background and the gradation value U3 of the characters. If this difference is large, it is highly likely that the pixels around the edge contain content such as characters or rules, and if this difference is small, it is highly likely that the pixels around the edge do not contain content such as characters or rules. The content determination unit 225 can determine with high accuracy whether or not content such as characters or rules is included in the corner region determined to have image loss by using the difference in gradation values of the pixels around the edge as the characteristic value of the pixels around the edge.

The content determination unit 225 may also calculate the variance (or standard deviation) of the gradation values of each edge-periphery pixel as the characteristic value of the edge-periphery pixel.

Figure 11 (B) is a graph G2 showing an example of the gradation values of pixels on the edge of a medium that contains a photograph as content.

The horizontal axis of graph G2 indicates the (horizontal) position of the pixel on the edge of the medium in the image, and the vertical axis indicates the gradation value of each pixel. In the example shown in graph G2, the background is present in area S4, and the photograph is present in area S5. In the area containing the photograph, pixels with various gradation values are present, and the gradation value of each pixel varies greatly. Therefore, if the variance of the gradation values is large, it is highly likely that the edge surrounding pixels contain content such as a photograph, and if the variance of the gradation values is small, it is highly likely that the edge surrounding pixels do not contain content such as a photograph. By using the variance of the gradation values of each edge surrounding pixel as the characteristic value of the edge surrounding pixel, the content determination unit 225 can determine with high accuracy whether or not a corner region determined to have image loss contains content such as a photograph.

The content determination unit 225 may also calculate the periodicity of the gradation values of each edge peripheral pixel as the characteristic value of the edge peripheral pixel. The content determination unit 225 calculates the periodicity so that the more pixels having the same gradation value exist periodically, the higher the value, and the more pixels having the same gradation value exist randomly, the lower the value. For example, the content determination unit 225 calculates the inverse of the variance (or standard deviation) of the number of consecutive pixels having gradation values within a predetermined range in the horizontal or vertical direction as the periodicity of the gradation values of each edge peripheral pixel. The predetermined range is a range of gradation values that is considered to indicate the same color (for example, 200 or more and 255 or less for white, 0 or more and 55 or less for black, etc.). The content determination unit 225 may set multiple predetermined ranges and calculate the average value of the inverse of the variance (or standard deviation) of the number of consecutive pixels having gradation values within each predetermined range in the horizontal or vertical direction as the periodicity of the gradation values of each edge peripheral pixel. The multiple predetermined ranges are ranges of gradation values corresponding to each of multiple colors such as white, gray, and black.

Figure 11 (C) is a graph G3 showing an example of the gradation values of pixels on the edge of a medium that contains a pattern (background pattern) as content.

The horizontal axis of graph G3 indicates the (horizontal) position of the pixel on the edge of the medium in the image, and the vertical axis indicates the gradation value of each pixel. In the example shown in graph G3, a hatched pattern is present throughout the image, with white pixels (gradation value U6) in region S6 and gray pixels (gradation value U7) in region S7. As shown in graph G3, pixels having specific gradation values are periodically present in regions containing patterns such as hatched, dotted, and lined patterns. Therefore, when the gradation value periodicity is high, it is highly likely that the edge surrounding pixels contain content such as a pattern, and when the gradation value periodicity is low, it is highly likely that the edge surrounding pixels do not contain content such as a pattern. The content determination unit 225 can determine with high accuracy whether or not a corner region determined to have image loss contains content such as a pattern by using the periodicity of the gradation value of each edge surrounding pixel as the characteristic value of the edge surrounding pixel.

The content determination unit 225 may also calculate the distribution of gradation values of each edge peripheral pixel as the characteristic value of the edge peripheral pixel. The content determination unit 225 calculates the frequency of each gradation value as the distribution of gradation values of each edge peripheral pixel. The content determination unit 225 may calculate the number of gradation values whose frequency is equal to or greater than a frequency threshold, or the number of groups of consecutive gradation values whose frequency is equal to or greater than a frequency threshold, as the distribution of gradation values of each edge peripheral pixel. The frequency threshold is set to a value (e.g., 3) that is sufficiently larger than the number of gradation values that may occur due to noise, etc.

Figure 11 (D) is graph G4 showing a histogram of graph G1 shown in Figure 11 (A).

The horizontal axis of graph G4 indicates the gradation value, and the vertical axis indicates the frequency of each gradation value. In the example shown in graph G4, the frequency is high around gradation value U1 corresponding to the white background, around gradation value U2 corresponding to the black lines, and around gradation value U3 corresponding to the black characters, and is equal to or greater than the frequency threshold T1. That is, in the example shown in graph G4, there are three groups of consecutive gradation values whose frequency is equal to or greater than the frequency threshold: a group of gradation values corresponding to the white background, a group of gradation values corresponding to the black lines, and a group of gradation values corresponding to the black characters.

When the number of groups of consecutive gradation values whose frequency is equal to or greater than the frequency threshold is one, it is highly likely that the edge surrounding pixels contain only background and no content. On the other hand, when the number of groups of consecutive gradation values whose frequency is equal to or greater than the frequency threshold is two or more and is small, it is highly likely that the edge surrounding pixels contain content such as characters, lines, or patterns. On the other hand, when the number of groups of consecutive gradation values whose frequency is equal to or greater than the frequency threshold is large, it is highly likely that the edge surrounding pixels contain content such as photographs. The content determination unit 225 can determine with high accuracy whether or not content is contained in a corner region determined to have image loss by using the distribution of gradation values of each edge surrounding pixel as a characteristic value of the edge surrounding pixel.

The characteristic value may include at least one of the above characteristic values, and may include multiple characteristic values.

Next, the content determination unit 225 determines whether each characteristic value of the edge peripheral pixels is within a specified range indicating the absence of content (step S403). The specified range is set, for example, to a range equal to or less than a characteristic threshold value. Each characteristic threshold value is set, based on prior experiments, to a value between each characteristic value when the edge peripheral pixels contain content and each characteristic value when the edge peripheral pixels do not contain content.

If all characteristic values of the edge surrounding pixels are within the specified range, the content determination unit 225 determines that no content is present around the edge of the medium in the target corner region (step S404), and ends the series of steps.

On the other hand, if any of the characteristic values of the edge surrounding pixels is outside the specified range, the content determination unit 225 determines that content is present around the edge of the medium within the corner region being determined (step S405).

In this way, the content determination unit 225 determines whether or not content exists around the medium edge in the corner region where the image loss determination unit 224 has determined that image loss has occurred. In particular, the content determination unit 225 determines whether or not content exists around the medium edge based on the difference, variance, periodicity, or distribution of the gradation values of multiple pixels corresponding to the medium edge in the corner region where the image loss determination unit 224 has determined that image loss has occurred. This allows the content determination unit 225 to determine with high accuracy whether or not content exists around the medium edge.

Next, the content determination unit 225 identifies the type of content present around the edge of the medium (step S406), and ends the series of steps. If the variance of the gradation values of the pixels around each edge is outside the specified range, the content determination unit 225 determines that the content is an image that cannot be corrected, and if the variance of the gradation values of the pixels around each edge is within the specified range, the content determination unit 225 determines that the content is a correctable character, ruled line, or pattern. If the variance of the gradation values of the pixels around each edge is outside the specified range, the content determination unit 225 may determine that the content is a correctable pattern. If the number of groups of consecutive gradation values whose frequency is equal to or greater than the frequency threshold is equal to or greater than the group number threshold, the content determination unit 225 may determine that the content is an image that cannot be corrected. On the other hand, if the number of groups of consecutive gradation values whose frequency is equal to or greater than the frequency threshold is less than the group number threshold, the content determination unit 225 may determine that the content is a correctable character, ruled line, or pattern. The group number threshold is set to a value sufficiently greater than 2 (for example, 4, etc.).

Figure 12 is a flowchart showing an example of the operation of the content correction process. The content correction process is executed in step S212 of the determination process shown in Figure 7.

First, the correction unit 226 sets the area inside (towards the center of the medium) of the original (before correction in step S211) end edge pixels in the corner area to be corrected where image loss occurs as the medium area. The correction unit 226 also sets the area outside (opposite the center of the medium) of the original end edge pixels and inside the end edge pixels corrected in step S211 in the corner area to be corrected where image loss occurs as the correction target area (step S501). The media area is set so as not to include the end edge pixels themselves, and the correction target area is set so as to include the end edge pixels themselves. The media area may be set so as to include the end edge pixels themselves, and the correction target area may be set so as not to include the end edge pixels themselves.

Next, the correction unit 226 transmits a search request to the search server via the second communication device 203 to request a search for similar images that include pixels similar to the medium area in the corner area to be corrected (step S502). The search request includes an image that includes the medium area in the corner area to be corrected. The search server has a database that includes various images, and when it receives a search request, it detects similar images that are similar to the image included in the search request from among the images included in the database. The search server calculates the similarity between the image included in the search request and each image included in the database, and detects the image with the highest similarity as a similar image. The search server calculates the similarity of the NCC, reciprocal of SSD, or reciprocal of SAD, etc., of the two images. The search server transmits the detected similar images to the information processing device 200.

Next, the correction unit 226 receives similar images from the search server via the second communication device 203 (step S503).

Next, the correction unit 226 forms (restores, corrects) the content in the correction target area in the corner area of the correction target based on the received similar image, corrects the correction target area (step S504), and ends the series of steps. The correction unit 226 cuts out an area having the same size as the correction target area from the received similar image while shifting its position to generate a cut-out image. Note that the correction unit 226 may generate a cut-out image by cutting out an area having a different size from the correction target area and enlarging or reducing it to the same size as the correction target area. The correction unit 226 calculates the similarity between each cut-out image and the correction target area, and corrects the correction target area by replacing the correction target area with the cut-out image with the highest similarity. The correction unit 226 calculates the similarity between the NCC, the reciprocal of SSD, the reciprocal of SAD, etc. of the two images.

In this way, the correction unit 226 corrects the corner region based on a similar image that includes pixels similar to the surrounding pixels of the medium edge in the corner region to be corrected. That is, the correction unit 226 corrects a corner region that has been determined by the image loss determination unit 224 to have image loss and that has been determined by the content determination unit 225 to have content present around the medium edge, based on the surrounding pixels of the medium edge within that corner region. This allows the correction unit 226 to appropriately correct a corner region where image loss has occurred and content exists around the medium edge.

In addition, the information processing device 200 may store various images in advance in the second storage device 210, and in steps S501 and S502, the correction unit 226 may detect similar images from among the images stored in the second storage device 210.

As described above in detail, the image processing system 1 compares multiple corner regions of a medium in an input image to determine whether or not image loss has occurred in each of the multiple corner regions. This makes it possible for the image processing system 1 to determine whether or not image loss has occurred in each corner region for media having corners of various shapes. Therefore, the image processing system 1 can easily and accurately determine whether or not image loss has occurred in a corner region of a medium in an image.

Furthermore, the image processing system 1 corrects corner areas where image loss occurs and where no content exists around the edge of the medium, but does not correct corner areas where content exists around the edge of the medium. This makes it possible for the image processing system 1 to prevent content from being lost by correcting corner areas where image loss occurs. Therefore, the image processing system 1 is able to appropriately correct corner areas of the medium in the image.

In addition, the image processing system 1 corrects corner areas where image loss occurs and content exists around the edge of the medium based on the surrounding pixels of the edge of the medium within the corner area. This makes it possible for the image processing system 1 to satisfactorily restore content in corner areas where image loss occurs.

In particular, the image processing system 1 is able to deal appropriately with cases in which skew of the medium occurs in the image reading device 100, causing image loss in the input image, and is able to tolerate the occurrence of skew of the medium in the image reading device 100 to a certain extent. Therefore, the image reading device 100 is able to miniaturize the imaging device 118, and is able to reduce the device size and device costs.

FIG. 13 is a flowchart showing an example of the operation of image loss determination processing according to another embodiment. The image loss determination processing shown in FIG. 13 is executed instead of the image loss determination processing shown in FIG. 9.

First, the image loss determination unit 224 calculates the area of the medium area in each corner region detected by the detection unit 223 (step S601). The image loss determination unit 224 identifies the area in each corner region that is inside (toward the center) the edge pixels as the medium area, and calculates the number of pixels in the medium area as the area of the medium area in each corner region.

Next, the image loss determination unit 224 determines whether or not there is a combination of corner regions in which the difference in the area of the medium area is greater than the area threshold (step S602). The image loss determination unit 224 extracts all combinations (pairs) of two corner regions from all corner regions detected by the detection unit 223. For all extracted combinations, the image loss determination unit 224 calculates the difference in the area of the medium area of the two corner regions included in each combination. The area threshold is set to a value between the difference in the area of the medium area calculated for two corner regions where no image loss occurs and the difference in the area of the medium area calculated for a corner region where no image loss occurs and a corner region where image loss occurs, based on a prior experiment.

If there is no combination of corner regions in which the difference in the area of the media regions is greater than the area threshold, the image loss determination unit 224 determines that no image loss has occurred in any corner region (step S603), and ends the series of steps.

On the other hand, if there is a combination of corner regions in which the difference in the area of the medium regions is greater than the area threshold, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions (step S604). As described above, the shapes of the four corners of the medium are likely to be the same, and if no image loss has occurred, the areas of the medium regions in all corner regions are likely to be similar. On the other hand, the possibility of image loss occurring in the same manner in all corner regions is low, and the possibility of image loss occurring such that the areas of the medium regions in each corner region are similar is low. Therefore, if the areas of the medium regions in all corner regions are similar, the image loss determination unit 224 determines that no image loss has occurred in any corner region. On the other hand, if the area of the medium region in any corner region is significantly different from the area of the medium region in the other corner regions, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions.

In this way, the image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner regions by comparing the areas of the medium areas in the multiple corner regions detected by the detection unit 223. This allows the image loss determination unit 224 to determine whether or not image loss has occurred even if it does not recognize in advance the shape of the corners of the medium being transported. Therefore, the image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes.

Next, the image loss determination unit 224 identifies a corner region where image loss occurs based on the area of the medium area in each corner region (step S605), and ends the series of steps. The image loss determination unit 224 determines that image loss does not occur in the corner region where the area of the medium area is the largest, and in the corner region where the difference between the area of the medium area and the maximum area is equal to or less than the area threshold. On the other hand, the image loss determination unit 224 determines that image loss occurs in the corner region where the difference between the area of the medium area and the maximum area is greater than the area threshold. As described above, it is unlikely that the same type of image loss occurs in all corner regions, and it is unlikely that image loss occurs so that the areas of the medium areas are similar in each corner region. Therefore, it is highly likely that image loss does not occur in a corner region where the area of the medium area is similar to that of the other corner regions, and it is highly likely that image loss occurs in a corner region where the area of the medium area is significantly different from that of the other corner regions. The image loss determination unit 224 can identify the corner region where image loss occurs with high accuracy by using the area of the medium area in each corner region.

As described above in detail, the information processing device 200 can easily and accurately determine whether or not image loss has occurred in a corner area of a medium in an image, even when comparing the areas of the medium areas in multiple corner areas.

Figure 14 is a flowchart showing an example of the operation of image loss determination processing according to yet another embodiment. The image loss determination processing shown in Figure 14 is executed instead of the image loss determination processing shown in Figure 9.

First, the image loss determination unit 224 calculates, for each of a plurality of corner regions, the distance between the intersection of two straight lines corresponding to two sides of the medium that form the corner corresponding to each corner region and a predetermined edge pixel as the corner distance (step S701). For each of the upper left corner region, upper right corner region, lower left corner region, and lower right corner region, the image loss determination unit 224 calculates the corner distance between each of the upper left intersection, upper right intersection, lower left intersection, and lower right intersection detected in step S203 of the determination process and a predetermined edge pixel. The end edge pixel closest to each intersection in each corner region is set as the predetermined edge pixel. The end edge pixel closest to the straight line passing through each intersection and a point located diagonally opposite each intersection in each corner region may also be set as the predetermined edge pixel.

The image loss determination unit 224 calculates the Euclidean distance between each intersection and a specified edge pixel as the corner distance. Note that the image loss determination unit 224 may calculate the Manhattan distance between each intersection and a specified edge pixel, that is, the sum of the distance in the horizontal direction and the distance in the vertical direction, as the corner distance. This allows the image loss determination unit 224 to calculate the corner distance efficiently and accurately. The image loss determination unit 224 may also calculate the Chebyshev distance between each intersection and a specified edge pixel, that is, the greater of the distance in the horizontal direction and the distance in the vertical direction, as the corner distance. This allows the image loss determination unit 224 to calculate the corner distance efficiently.

Figure 15 is a schematic diagram to explain corner distance.

Figure 15 shows the input image N shown in Figure 8. In this example, the end edge pixel E5 closest to the upper left intersection V1 is extracted as the specified edge pixel, and the corner distance D1 from the upper left intersection V1 to the end edge pixel E5 is calculated. The end edge pixel E6 closest to the upper right intersection V2 is extracted as the specified edge pixel, and the corner distance D2 from the upper right intersection V2 to the end edge pixel E6 is calculated. The end edge pixel E7 closest to the lower left intersection V3 is extracted as the specified edge pixel, and the corner distance D3 from the lower left intersection V3 to the end edge pixel E7 is calculated. The end edge pixel E8 closest to the lower right intersection V4 is extracted as the specified edge pixel, and the corner distance D4 from the lower right intersection V4 to the end edge pixel E8 is calculated.

As shown in FIG. 15, the corner distances D1, D3, and D4 calculated for the upper left intersection V1, the lower left intersection V3, and the lower right intersection V4, which have no bent corners, are approximately the same as each other. On the other hand, the corner distance D2 calculated for the upper right intersection V2, which has a bent corner, is significantly different from the corner distances D1, D3, and D4. Therefore, the image loss determination unit 224 can accurately determine whether or not image loss has occurred in each corner area based on the corner distances.

Next, the image loss determination unit 224 determines whether or not there is a combination of corner regions in which the difference in corner distance is greater than the distance threshold (step S702). The image loss determination unit 224 extracts all combinations (pairs) consisting of two corner regions from all corner regions detected by the detection unit 223. For all extracted combinations, the image loss determination unit 224 calculates the difference in corner distances calculated for the two corner regions included in each combination. The distance threshold is set to a value between the difference in corner distances calculated for two corner regions in which no image loss occurs and the difference in corner distances calculated for a corner region in which no image loss occurs and a corner region in which image loss occurs, based on a prior experiment. Note that the image loss determination unit 224 may set a predetermined number of end edge pixels in each corner region in order of proximity to each intersection as the predetermined edge pixels, and calculate the statistical value (average value, median value, mode, maximum value, minimum value, etc.) of the distance between each intersection and each predetermined edge pixel as the corner distance.

If there is no combination of corner regions in which the difference in corner distance is greater than the distance threshold, the image loss determination unit 224 determines that no image loss has occurred in any corner region (step S703), and ends the series of steps.

On the other hand, if there is a combination of corner regions in which the difference in corner distance is greater than the distance threshold, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions (step S704). As described above, the shapes of the four corners of the medium are likely to be the same, and if no image loss has occurred, the corner distances are likely to be similar in all corner regions. On the other hand, the same type of image loss is unlikely to occur in all corner regions, and image loss is unlikely to occur such that the corner distances are similar in each corner region. Therefore, if the corner distances in all corner regions are similar, the image loss determination unit 224 determines that no image loss has occurred in any corner region. On the other hand, if the corner distance in any corner region is significantly different from the corner distance in the other corner regions, the image loss determination unit 224 determines that image loss has occurred in one of the corner regions.

In this way, the image loss determination unit 224 determines whether or not image loss has occurred in each of the multiple corner areas detected by the detection unit 223 by comparing the distances calculated for each of the multiple corner areas detected by the detection unit 223. This allows the image loss determination unit 224 to determine whether or not image loss has occurred even if it does not recognize in advance the shape of the corners of the medium being transported. Therefore, the image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes.

Next, the image loss determination unit 224 identifies the corner area where image loss occurs based on the corner distance calculated for each of the multiple corner areas (step S705), and ends the series of steps. The image loss determination unit 224 determines that image loss does not occur in the corner area where the corner distance is the smallest and in the corner area where the difference between the corner distance and the smallest corner distance is equal to or less than the distance threshold. On the other hand, the image loss determination unit 224 determines that image loss occurs in the corner area where the difference between the corner distance and the smallest corner distance is greater than the distance threshold. As described above, it is unlikely that the same type of image loss occurs in all corner areas, and it is unlikely that image loss occurs so that the corner distances are similar in each corner area. Therefore, it is highly likely that image loss does not occur in a corner area where the corner distance is similar to the corner distance in other corner areas, and it is highly likely that image loss occurs in a corner area where the corner distance is significantly different from the corner distance in other corner areas. By using the corner distance in each corner area, the image loss determination unit 224 can identify the corner area where image loss is occurring with high accuracy.

As described above in detail, the information processing device 200 is now able to easily and accurately determine whether or not image loss has occurred in a corner area of a medium in an image, even when comparing corner distances in multiple corner areas.

Figure 16 is a flowchart showing an example of the operation of image loss determination processing according to yet another embodiment. The image loss determination processing shown in Figure 16 is executed instead of the image loss determination processing shown in Figure 9.

First, the image loss determination unit 224 reads out, from the second storage device 210, an input image captured before the input image to be determined as the second input image (step S801). For example, the image loss determination unit 224 reads out, as the second input image, an input image captured immediately before the input image to be determined.

If an input image captured before the input image to be determined has not been acquired, the image loss determination unit 224 waits until an input image captured immediately after the input image to be determined is acquired, and uses the input image captured immediately after the input image to be determined as the second input image. Also, the image loss determination unit 224 may use the input image captured immediately after the input image to be determined as the second input image even if image loss occurs in a corner area detected from the input image captured immediately before the input image to be determined. The image loss determination unit 224 may use an input image captured a predetermined number of times before or after the input image to be determined as the second input image, instead of an input image captured immediately before or after the input image to be determined.

In addition, in a booklet or the like consisting of multiple pages, the shape of the cover may be different from the shapes of the other pages. The image missing judgment unit 224 may not use the input image generated first in the medium reading process as the input image in which the cover is captured as the second input image. In addition, in a booklet or the like consisting of multiple pages, the shape of the inner cover may be different from the shapes of the other pages. For example, the image missing judgment unit 224 may execute character recognition processing in the input image using a known OCR (Optical Character Recognition) technology. When the image missing judgment unit 224 recognizes characters of a predetermined font size (e.g., 20 points) or more in the center of the input image, it considers the input image to be the input image in which the inner cover is captured, and does not use it as the second input image. In addition, in a booklet or the like consisting of multiple pages, the shape of the odd-numbered pages and the shape of the even-numbered pages may be different from each other. The image missing judgment unit 224 may use the input image captured two pages before or two pages after the input image to be judged as the second input image. With these, the image missing judgment unit 224 can judge whether or not an image missing has occurred with a higher degree of accuracy.

The image loss determination unit 224 may use as the second input image only an input image in which a medium of approximately the same size as the medium included in the input image to be determined is captured. In this case, the image loss determination unit 224 detects edge pixels in each input image in the same manner as the process of step S202 in FIG. 7, and detects the number of pixels included in the area surrounded by the detected edge pixels as the size of the medium. The image loss determination unit 224 determines whether the sizes of the media included in each input image are approximately the same depending on whether the difference in the detected medium sizes is equal to or less than a predetermined threshold value.

The image-missing determination unit 224 may use as the second input image only an input image in which a medium having a document layout (or ruled lines) substantially identical to the document layout (or ruled lines) in the medium included in the input image to be determined is captured. In this case, the image-missing determination unit 224 further detects edge pixels in the area surrounded by the edge pixels of each input image in the same manner as the process of step S202 in FIG. 7. The image-missing determination unit 224 detects straight lines (line segments) from each edge pixel using the least squares method or Hough transform, etc. The image-missing determination unit 224 generates a binary image in which only pixels corresponding to the detected straight lines are valid pixels and other pixels are invalid pixels. The image-missing determination unit 224 calculates the NCC, reciprocal of SSD, reciprocal of SAD, etc. of two binary images among the generated binary images as the degree of similarity of the two input images corresponding to each binary image. The image loss determination unit 224 determines whether the document layouts of the media contained in each input image are substantially the same based on whether the calculated similarity is equal to or greater than a predetermined threshold.

In addition, at the edge of a document on which characters are printed, there may be marks such as logos, fixed characters such as headers/footers, or page numbers, and it is highly likely that such marks, fixed characters, or page numbers are written in the same position on all pages. The image missing determination unit 224 may use as the second input image only an input image in which a medium having the same type of mark, fixed characters, or page number as the mark, fixed characters, or page number written on the medium included in the input image to be determined is captured. In this case, the image missing determination unit 224 calculates the NCC, reciprocal of SSD, reciprocal of SAD, etc. of the upper end area or lower end area (margin area) of the medium of the two input images as the degree of similarity. The image missing determination unit 224 determines whether the marks or fixed characters written on the medium included in each input image are the same or not depending on whether the calculated degree of similarity is equal to or greater than a predetermined threshold value. Alternatively, the image missing determination unit 224 performs character recognition processing within the upper end area or lower end area of the medium of the two input images using a known OCR technique. The image loss determination unit 224 detects numbers at the same position in each area and determines whether a page number is written on each medium based on whether the difference between the detected numbers matches the difference in the capture order of each input image.

Next, the image loss determination unit 224 calculates the degree of similarity between the multiple corner regions (step S802). The image loss determination unit 224 calculates the degree of similarity between each corner region detected from the input image to be determined and a corner region at a corresponding position detected from the second input image. The image loss determination unit 224 calculates the reciprocal of the NCC, SSD, or the reciprocal of the SAD, etc., of the two corner regions as the degree of similarity.

Next, the image loss determination unit 224 determines whether or not there is a corner region whose similarity is greater than a similarity threshold (step S803). The similarity threshold is set to a value between the similarity calculated for two corner regions where no image loss occurs and the similarity calculated for a corner region where no image loss occurs and a corner region where image loss occurs, based on a prior experiment.

If there is no corner region whose similarity is less than the similarity threshold, the image loss determination unit 224 determines that no image loss has occurred in any corner region (step S804), and ends the series of steps.

On the other hand, if there is a corner region whose similarity is smaller than the similarity threshold, the image loss determination unit 224 determines that image loss has occurred in that corner region (step S805) and ends the series of steps. Media placed together on the placement table 103 and transported, especially media transported continuously, are likely to be the same type of media, and the shapes of corresponding corners are likely to be the same. On the other hand, in media placed together on the placement table 103 and transported, especially media transported continuously, the possibility that corresponding corners are transported outside the imaging range of the imaging device 118 in the same way, and the possibility that corresponding corners are bent in the same way is low. Therefore, if corresponding corner regions in multiple input images are similar to each other, the image loss determination unit 224 determines that image loss has not occurred in that corner region. On the other hand, if corresponding corner regions in multiple input images are not similar to each other, the image loss determination unit 224 determines that image loss has occurred in that corner region.

In this way, the image loss determination unit 224 determines whether or not image loss has occurred in a corner area by comparing corresponding corner areas detected by the detection unit 223 in multiple input images. This allows the image loss determination unit 224 to determine whether or not image loss has occurred even if it does not recognize in advance the shape of the corners of the medium being transported. Therefore, the image loss determination unit 224 can easily and accurately determine whether or not image loss has occurred for media having corners of various shapes.

The image loss determination unit 224 may determine whether or not image loss has occurred using parameters other than the degree of similarity between multiple corner regions. For example, the image loss determination unit 224 may calculate the area of the medium area in each corner region, similar to the processing of step S601 in FIG. 13, and determine whether or not image loss has occurred based on whether the difference in the area of the medium area in corresponding corner regions is greater than an area threshold. The image loss determination unit 224 may also calculate the corner distance in each corner region, similar to the processing of step S601 in FIG. 13, and determine whether or not image loss has occurred based on whether the difference in the corner distance in corresponding corner regions is greater than a distance threshold.

The image loss determination unit 224 may also determine whether image loss has occurred in only one, two, or three of the corner regions, rather than in all four corner regions.

As described above in detail, the information processing device 200 can easily and accurately determine whether or not image loss has occurred in a corner area of a medium in an image, even when comparing corresponding corner areas in multiple input images.

In addition, the image loss determination unit 224 may determine whether or not image loss occurs in a corner area in the input image by using another method. For example, the image loss determination unit 224 determines whether or not image loss occurs in a corner area in the input image based on whether or not the medium transported in the media reading device 100 passes outside the imaging range of the imaging device 118. If the medium passes outside the imaging range of the imaging device 118, in step S106 of FIG. 6, the control unit 141 transmits image loss occurrence information indicating that image loss has occurred to the information processing device 200 via the first communication device 122. The image loss occurrence information includes information indicating the second media sensor 114 that detected the medium, the imaging start time of the medium, the detection time when the second media sensor 114 detected the medium, and the transport speed of the medium. When the image loss determination unit 224 receives image loss occurrence information from the media reading device 100 via the second communication device 203, it determines that image loss has occurred in a corner area in the input image. The image loss determination unit 224 identifies the end in the main scanning direction where the image loss occurs in the input image based on information indicating the second medium sensor 114 that detected the medium. The image loss determination unit 224 also identifies the end in the sub-scanning direction where the image loss occurs in the input image based on the image capture start time of the medium, the detection time when the second medium sensor 114 detected the medium, and the medium transport speed.

The image loss determination unit 224 may also determine that no image loss occurs in the corresponding corner area when the corner of the medium in the input image is an approximately right angle, and determine that image loss occurs in the corresponding corner area when the corner of the medium in the input image is not an approximately right angle. In this case, the image loss determination unit 224 detects the left end straight line, the right end straight line, the top end straight line, and the bottom end straight line from the input image in the same manner as the processing of step S203 in FIG. 7. The image loss determination unit 224 generates a corrected image in which the detected left end straight line, the right end straight line, the top end straight line, and the bottom end straight line are extended to a position where they intersect with each other. The image loss determination unit 224 calculates the reciprocal of the NCC, SSD, or SAD between each corner area in the input image and the corresponding corner area in the corrected image as the degree of similarity. The image loss determination unit 224 determines whether the corner of the medium in each corner area is an approximately right angle depending on whether the calculated degree of similarity is equal to or greater than a predetermined threshold.

Figure 17 is a flowchart showing an example of the operation of a content determination process according to another embodiment. The content determination process shown in Figure 17 is executed instead of or in addition to the content determination process shown in Figure 10. If the content determination process shown in Figure 10 is not executed, the process of step S210 in Figure 7 may be omitted, and either the process of steps S211 to S212 or the process of steps S213 to S214 may be fixed and executed regardless of the type of content.

First, the content determination unit 225 detects the medium edge in the corner area where it is determined that image loss has occurred (step S901), in a manner similar to the processing of step S401 in FIG. 10.

Next, the content determination unit 225 executes character recognition processing in the input image (step S902). That is, the content determination unit 225 recognizes characters in an area including a corner area determined by the image loss determination unit 224 to have image loss. The content determination unit 225 detects characters from the input image using a known OCR technology and detects a plurality of characters having a predetermined positional relationship as a character string. The plurality of characters having a predetermined positional relationship are, for example, characters that overlap each other in the vertical direction and are adjacent to each other within a certain range in the horizontal direction and are connected together. The certain range is set, for example, to the number of pixels corresponding to the margin. The plurality of characters having a predetermined positional relationship includes characters that are connected with a line feed in between. That is, the plurality of characters having a predetermined positional relationship includes a character located at the right end in the horizontal direction and a character located adjacent to the character below the character within a certain range in the vertical direction and located at the left end in the horizontal direction. The plurality of characters having a predetermined positional relationship also includes a character located at the bottom end in the vertical direction and a character located adjacent to the character within a certain range to the left of the character in the horizontal direction and located at the top end in the vertical direction. The content determination unit 225 stores the detected characters (character strings) and the position information (coordinate information) of the characters in the input image in the second storage device 210.

Next, the content determination unit 225 determines whether or not characters have been detected around the edge of the medium in each corner region (step S903). For example, the content determination unit 225 determines that characters have been detected around the edge of the medium when characters are detected within a range of a predetermined number of pixels from the edge of the medium. The predetermined number of pixels is set to, for example, the number of pixels equivalent to the vertical or horizontal length of one or two characters.

If no characters are detected around the edge of the medium in any of the corner regions, the content determination unit 225 determines that no content is present around the edge of the medium in each corner region (step S904), and ends the series of steps.

On the other hand, if a character is detected around the edge of the medium in any of the corner regions, the content determination unit 225 determines whether or not the character string including the character detected around the edge of the medium is a natural language (step S905). The content determination unit 225 uses known natural language analysis techniques (morphological analysis, syntactic analysis, semantic analysis, contextual analysis) to determine whether or not the character string is a natural language. The content determination unit 225 may determine whether or not the character string is a natural language using a learning model that has been trained to output whether or not the input character string is a natural language when the character string is input. The learning model is pre-trained by deep learning or the like using various natural language character strings and/or non-natural language character strings as teacher data. The content determination unit 225 inputs the detected character string into the learning model and determines whether or not the character string is a natural language based on the output value output from the learning model.

If the character string including the characters detected around the edge of the medium in each corner region is a natural language, the content determination unit 225 determines that no content exists around the edge of the medium in each corner region (step S904), and ends the series of steps. In other words, if the character string including the characters detected around the edge of the medium in each corner region is a natural language, the content determination unit 225 determines that the characters around the edge of the medium in each corner region are not lost due to image loss.

On the other hand, if the character string including the characters detected around the edge of the medium in any of the corner regions is not a natural language, the content determination unit 225 determines that content exists around the edge of the medium in that corner region (step S906), and ends the series of steps. In other words, if the character string including the characters detected around the edge of the medium in each corner region is not a natural language, the content determination unit 225 determines that the characters around the edge of the medium in each corner region have been lost due to image loss. In this case, the content determination unit 225 determines that the content is characters that can be corrected.

The processing of step S905 may be omitted, and if text is detected near the edge of the medium in any corner area, the content determination unit 225 may determine that content is present near the edge of the medium.

In this way, when the content determination unit 225 recognizes characters around the edge of the medium in a corner area where the image loss determination unit 224 has determined that image loss has occurred, the content determination unit 225 determines that content exists around the edge of the medium. In general, there is a high probability that margins are set around the edges of a document on which characters are printed, and there is a high probability that no characters exist around the edge of the medium. When characters exist around the edge of the medium, it is highly likely that at least the margin portion has been lost due to image loss, and that the portion lost due to image loss also contains characters. When the content determination unit 225 recognizes characters around the edge of the medium, it determines that content exists around the edge of the medium, thereby making it possible to determine with high accuracy that characters have been lost due to image loss.

Furthermore, if the characters recognized in an area including a corner area determined to have image loss are not a natural language, the content determination unit 225 determines that content exists around the edge of the medium in that corner area. In general, character strings written in a document are likely to be natural language that humans use on a daily basis to communicate, and are unlikely to be meaningless random character strings. If the recognized character string is not a natural language, the content determination unit 225 determines that part of the character string has been damaged by image loss, thereby making it possible to determine with high accuracy that characters have been lost due to image loss.

As described above in detail, the image processing system 1 is now capable of appropriately correcting corner areas of a medium within an image, even when character recognition technology is used.

Figure 18 is a flowchart showing an example of the operation of a content determination process according to yet another embodiment. The content determination process shown in Figure 18 is executed instead of the content determination process shown in Figure 10.

First, the content determination unit 225 reads the second input image from the second storage device 210 (step S1001), similar to the processing of step S801 in FIG. 16.

Next, the content determination unit 225 determines whether or not content exists in a corner area in the second input image that corresponds to the corner area determined to have image loss in the input image to be determined (step S1002). The content determination unit 225 calculates characteristic values of multiple pixels included in each corner area in the second input image in a manner similar to the process of step S402 in FIG. 10. If the characteristic value is within a specified range indicating that no content exists, the content determination unit 225 determines that no content exists in the corresponding corner area in the second input image. On the other hand, if the characteristic value is not within the specified range, the content determination unit 225 determines that content exists in the corresponding corner area in the second input image.

If no content is present in the corner region of the second input image, the content determination unit 225 determines that no content is present around the edge of the medium in the corresponding corner region of the input image to be determined (step S1003), and ends the series of steps.

On the other hand, if content is present in a corner region in the second input image, the content determination unit 225 determines that content is present around the edge of the medium in the corresponding corner region of the input image to be determined (step S1004).

In this way, when content is present in a corner area in the second input image corresponding to the corner area determined by the image loss determination unit 224 as having image loss, the content determination unit 225 determines that content is present around the edge of the medium in the corner area determined by the image loss determination unit 224 as having image loss. As described above, marks, static characters, page numbers, etc. may be written on the edge of a document on which characters are printed, and such marks, static characters, or page numbers are likely to be written in the same position on all pages. The content determination unit 225 can determine with high accuracy whether or not marks, static characters, or page numbers are present around the edge of the medium by using the second input image in which the medium is imaged before or after the input image to be determined.

Next, the content determination unit 225 identifies the type of content present around the medium edge (step S1005), and ends the series of steps. The content determination unit 225 identifies the type of content present around the medium edge in the same manner as the processing of step S406 in FIG. 10. However, the content determination unit 225 identifies the type of content present around the medium edge of each corner region based on the characteristic values of multiple pixels included in each corner region in the second input image. The content determination unit 225 determines that the type of content present around the medium edge of each corner region in the input image to be determined is the same as the identified type of content.

Note that, if the content determination process of this embodiment determines that image content exists near the edge of the medium, the content is likely to be a logo or the like that is common to all pages. Therefore, in step S210 of FIG. 7, the correction unit 226 may determine that the content can be corrected regardless of the type of content.

As described above in detail, the image processing system 1 is now capable of appropriately correcting the corner areas of the medium in the image even when using the second input image.

FIG. 19 is a flowchart showing an example of the operation of content correction processing according to another embodiment. The content correction processing shown in FIG. 19 is executed instead of the content correction processing shown in FIG. 12.

First, the correction unit 226 sets the medium area and the correction target area in the corner area to be corrected where image loss occurs, in the same manner as in the processing of step S501 in FIG. 12 (step S1101).

Next, the correction unit 226 forms (restores, corrects) a background in the correction target area and corrects the correction target area (step S1102). When the content is determined to be a pattern (background pattern) in the content determination process, the correction unit 226 corrects the pattern (background pattern) in the correction target area. In addition, when the difference between the color value (RGB value) of each pixel in the medium area in the corner area of the correction target and the color value of each pixel in the correction target area is equal to or greater than a predetermined threshold, the correction unit 226 may correct the background color in the correction target area. The correction unit 226 corrects the background in the correction target area by cutting out an image of the same size as the correction target area from the medium area and arranging (copying) it in the correction target area. This allows the correction unit 226 to properly restore the background that has been damaged by image loss.

Next, the correction unit 226 forms (restores, corrects) characters in the correction target area to correct the correction target area (step S1103). When the content is determined to be characters in the content determination process, the correction unit 226 corrects the characters in the correction target area. The correction unit 226 detects a plurality of characters having a predetermined positional relationship as a character string from the input image in the same manner as the process of step S902 in FIG. 10. The correction unit 226 converts a character string including the characters detected in the medium area into a character string in natural language using a known natural language analysis technique (morphological analysis, syntactic analysis, semantic analysis, context analysis). The correction unit 226 corrects the characters in the correction target area by arranging (writing) in the correction target area a character that is not included in the pre-conversion character string and is connected to a character that exists around the edge of the medium, among the characters included in the converted character string. The correction unit 226 may convert a character string including the characters detected in the medium area into a character string in natural language using a learning model that has been trained to output a character string obtained by converting an input character string into a natural language when a character string is input. The learning model is pre-trained by deep learning or the like using a set of various non-natural language character strings and their corresponding natural language character strings as training data. The correction unit 226 inputs a character string including characters detected within the medium area to the learning model, and obtains a natural language character string based on an output value output from the learning model.

In this way, the correction unit 226 recognizes characters within an area including a corner area in which the image loss determination unit 224 has determined that an image loss has occurred, and corrects the corner area based on a natural language similar to the recognized characters. This allows the correction unit 226 to properly restore characters that have been damaged due to image loss.

Next, the correction unit 226 forms (restores, corrects) a ruled line in the correction target area, corrects the correction target area (step S1104), and ends the series of steps. If the content is determined to be a ruled line in the content determination process, the correction unit 226 corrects the ruled line in the correction target area. The correction unit 226 extracts edge pixels from the medium area in the corner area to be corrected in the same manner as the process of step S202 in FIG. 7, and detects straight lines from the extracted edge pixels in the same manner as the process of step S203. The correction unit 226 detects, among the detected straight lines, straight lines that extend to the original end edge pixels (before being corrected in step S211) as ruled lines cut at the end of the medium in the corner area. The correction unit 226 corrects the ruled line in the correction target area by further extending the straight lines detected as ruled lines to a predetermined position in the correction target area. The predetermined position is set, for example, to a position that is inward (toward the center of the medium) by the number of pixels equivalent to the margin from the end edge pixels corrected in step S211.

When the correction unit 226 detects a line extending horizontally and a line extending vertically as a cut ruled line, the correction unit 226 may correct the ruled line by extending each line to a position where the lines intersect with each other. This makes it possible to properly restore table-format ruled lines that have been damaged due to image loss.

In this way, the correction unit 226 corrects the corner area where the image loss determination unit 224 has determined that image loss has occurred by extending the ruled line that was cut at the edge of the medium in the corner area. This allows the correction unit 226 to properly restore the ruled line that has been damaged by the image loss.

As described above in detail, the image processing system 1 is now able to satisfactorily restore content in corner areas where image loss occurs, even when correcting background patterns, characters and/or lines.

Figure 20 is a flowchart showing an example of the operation of content correction processing according to yet another embodiment. The content correction processing shown in Figure 20 is executed instead of the content correction processing shown in Figure 12.

First, the correction unit 226 reads out the second input image from the second storage device 210 (step S1201), similar to the processing of step S801 in FIG. 16. The correction unit 226 reads out multiple second input images.

Next, the correction unit 226 determines whether or not content exists in the corner region in each corresponding second input image (step S1202), in the same manner as the process of step S1002 in FIG. 18. If it is determined that no content exists in the corner region in each corresponding second input image, the correction unit 226 ends the series of steps without performing correction.

On the other hand, if it is determined that content exists in the corner regions in each corresponding second input image, the correction unit 226 determines whether the content is a mark, a static character, or a background pattern (step S1203). The correction unit 226 calculates the similarity of the corner regions in each corresponding second input image in the same manner as the processing of step S802 in FIG. 16. If the similarity of each corner region is equal to or greater than a threshold, the correction unit 226 determines that each corner region contains the same mark, static character, or background pattern, and if the similarity of each corner region is less than the threshold, the correction unit 226 determines that each corner region does not contain the same mark, static character, or background pattern.

If it is determined that the content is a mark, a static character, or a background pattern, the correction unit 226 forms (restores, corrects) a mark, a static character, or a background pattern in a corner area in the input image to be corrected, corrects the corner area (step S1204), and ends the series of steps. The correction unit 226 corrects the mark, static character, or background pattern by arranging (copying) the corresponding corner area in the second input image into the corner area in the input image to be corrected. In this way, the correction unit 226 can properly restore the mark, static character, or background pattern that has been damaged due to image loss.

Note that even if no content exists in a corner area of the input image to be corrected, the correction unit 226 may correct the background color by placing (copying) a corresponding corner area of the second input image in the corner area of the input image to be corrected.

On the other hand, if it is determined that the content is not a mark, the correction unit 226 determines whether the content is a page number (step S1205). The correction unit 226 performs character recognition processing in the corner regions of each corresponding second input image in the same manner as the processing of step S902 in FIG. 17. The correction unit 226 determines that the content is a page number if a number is detected in the corner region of each second input image and the difference between the numbers detected in the corner region of each second input image matches the difference between the imaging orders of each second input image. On the other hand, the correction unit 226 determines that the content is not a page number if a number is not detected in the corner region of any second input image, or if the difference between the numbers detected in the corner region of each second input image does not match the difference between the imaging orders of each second input image. If it is determined that the content is not a page number, the correction unit 226 does not perform correction and ends the series of steps.

On the other hand, if it is determined that the content is a page number, the correction unit 226 forms (restores, corrects) a page number in a corner region of the input image to be corrected, corrects the corner region (step S1206), and ends the series of steps. The correction unit 226 identifies the page number of the input image to be corrected from the relationship between the numbers detected in the corner regions of each second input image, and corrects the page number by placing (writing) the identified page number in the corner region of the input image to be corrected. This allows the correction unit 226 to properly restore the page number that was damaged due to image loss.

In this way, the correction unit 226 corrects a corner area in a specific input image among the multiple input images, where the image loss determination unit 224 has determined that an image loss has occurred and the content determination unit 225 has determined that content exists around the edge of the medium, based on the corresponding corner area in the other input image. This allows the correction unit 226 to properly restore the content that has been lost due to the image loss.

As described above in detail, the image processing system 1 is now able to satisfactorily restore content in corner areas where image loss occurs, even when using the second input image.

Figure 21 (A) is a diagram showing a schematic configuration of a first processing circuit 340 in an image reading device according to another embodiment. The first processing circuit 340 is used in place of the first processing circuit 140 in the image reading device 100, and executes image reading processing and the like in place of the first processing circuit 140. The first processing circuit 340 has a control circuit 341, a transmission circuit 342, and the like. Each of these components may be configured as an independent integrated circuit, microprocessor, firmware, and the like.

The control circuit 341 is an example of a control unit, and has the same functions as the control unit 141. The control circuit 341 receives an operation signal from the first operation device 105 or the first communication device 122, a first medium signal from the first medium sensor 111, a second medium signal from the second medium sensor 114, and a third medium signal from the third medium sensor 117, and controls the motor 121 and the imaging device 118 based on each of the received signals.

The transmission circuit 342 is an example of a transmission unit, and has the same function as the transmission unit 142. The transmission circuit 342 acquires an input image from the imaging device 118, and outputs it to the first communication device 122.

As described above in detail, even when the image reading device uses the first processing circuit 340, the image processing system 1 can easily and accurately determine whether or not image loss has occurred in a corner area of the medium in the image. Furthermore, the image processing system 1 can appropriately correct the corner area of the medium in the image. Furthermore, the image processing system 1 can satisfactorily restore content in the corner area where image loss has occurred.

Figure 21 (B) is a diagram showing a schematic configuration of the second processing circuit 420 in an information processing device according to another embodiment. The second processing circuit 420 is used in place of the second processing circuit 420 of the information processing device 200, and executes the determination process and the like in place of the second processing circuit 220. The second processing circuit 420 has an acquisition circuit 421, an extraction circuit 422, a detection circuit 423, an image loss determination circuit 424, a content determination circuit 425, a correction circuit 426, a notification circuit 427, an output control circuit 428, and the like. Each of these components may be composed of an independent integrated circuit, microprocessor, firmware, and the like.

The acquisition circuit 421 is an example of an acquisition unit, and has the same functions as the acquisition unit 221. The acquisition circuit 421 receives an input image from the second communication device 203 and stores it in the second storage device 210.

The extraction circuit 422 is an example of an extraction unit, and has the same functions as the extraction unit 222. The extraction circuit 422 reads the input image from the second storage device 210, extracts edge pixels from the input image, and stores the extraction result in the second storage device 210.

The detection circuit 423 is an example of a detection unit, and has the same functions as the detection unit 223. The detection circuit 423 reads the extraction results of the input image and edge image from the second storage device 210, detects corner areas based on the read information, and stores the detection results in the second storage device 210.

The image loss determination circuit 424 is an example of an image loss determination unit, and has the same function as the image loss determination unit 224. The image loss determination circuit 424 reads the input image and the corner region detection results from the second storage device 210, determines whether or not image loss has occurred in the corner region based on each piece of information read, and stores the determination result in the second storage device 210.

The content determination circuit 425 is an example of a content determination unit, and has the same functions as the content determination unit 225. The content determination circuit 425 reads the input image and the detection results of the corner region from the second storage device 210, determines whether or not content is present around the edge of the medium in the corner region based on each piece of information read, and stores the determination result in the second storage device 210.

The correction circuit 426 is an example of a correction unit, and has the same functions as the correction unit 226. The correction circuit 426 reads out the input image and the detection results of the corner area from the second storage device 210, corrects the corner area based on each piece of information read out, and stores the corrected input image in the second storage device 210.

The notification circuit 427 is an example of a notification unit, and has the same function as the notification unit 227. The notification circuit 427 reads out the image loss and content determination results from the second storage device 210, and outputs to the second display device 202 or the second communication device 203 that an image loss has occurred.

The output control circuit 428 is an example of an output control unit, and has the same functions as the output control unit 228. The output control circuit 428 reads the input image or the image loss and content judgment results from the second storage device 210, and outputs information related to the judgment results to the second display device 202 or the second communication device 203.

As described above in detail, even when the information processing device uses the second processing circuit 420, the image processing system 1 is able to easily and accurately determine whether or not image loss has occurred in a corner area of the medium in the image. The image processing system 1 is also able to appropriately correct the corner area of the medium in the image. The image processing system 1 is also able to satisfactorily restore content in the corner area where image loss has occurred.

Although preferred embodiments have been described above, the embodiments are not limited to these. For example, the image reading device may have a so-called U-turn path, and feed and transport the media placed on the loading table from the top, and discharge the media onto the discharge table. In this case, the feed roller is disposed above the separation roller, facing the separation roller.

The determination process in FIG. 7 may be executed by the image reading device 100, not by the information processing device 200. In that case, the first processing circuit 140 of the image reading device 100 has each unit of the second processing circuit 220 of the information processing device 200, and executes the determination process. The acquisition unit of the first processing circuit 140 acquires the input image and various information by reading them from the first storage device 130. The image processing system 1 may also have multiple information processing devices 200, rather than one, and each information processing device 200 may work together to share each process in the overall process and the second process. In that case, multiple information processing devices 200 may be distributed and arranged on a network so that image processing services can be provided in the form of cloud computing.

In addition, in steps S209 and/or S211 in Fig. 7, the correction unit 226 may correct the corner region determined to have image loss by using the corner region in the second input image corresponding to the corner region. In this case, the correction unit 226 reads out the second input image in the same manner as in the process of step S801 in Fig. 16, and corrects the corner region in which image loss occurs by arranging (copying) the corner region in the second input image corresponding to the corner region in which image loss occurs. In this case, the correction unit 226 may correct only the missing portion in the corner region in which image loss occurs.
(Appendix 1)
an acquisition unit that acquires an input image obtained by capturing an image of a medium;
a detection unit that detects a corner region that is estimated to include a corner of a medium within the input image;
an image loss determination unit that determines whether or not an image loss occurs in the corner area detected by the detection unit;
a content determination unit that determines whether or not content is present around a medium edge in a corner area in which the image loss determination unit has determined that an image loss has occurred;
a correction unit that corrects corner areas in which it has been determined by the image loss determination unit that image loss has occurred, where the content determination unit has determined that no content exists around an edge of a medium, and does not correct corner areas in which it has been determined by the content determination unit that content exists around an edge of the medium;
an output unit that outputs the input image corrected by the correction unit;
13. An image processing device comprising:
(Appendix 2)
2. The image processing device according to claim 1, further comprising a notification unit that notifies a user that an image is missing when there is a corner area in which the content determination unit determines that content exists around an edge of the medium.
(Appendix 3)
The image processing device according to claim 1 or 2, wherein the content determination unit determines whether or not content is present near the edge of the medium based on the difference, variance, periodicity, or distribution of gradation values of multiple pixels corresponding to the edge of the medium within a corner region where the image loss determination unit has determined that an image loss has occurred.
(Appendix 4)
The image processing device according to claim 1 or 2, wherein the content determination unit determines that content is present around the edge of the medium when characters are recognized around the edge of the medium in a corner area in which the image loss determination unit has determined that an image loss has occurred.
(Appendix 5)
The image processing device described in Appendix 1 or 2, wherein the content determination unit recognizes characters within an area including a corner area in which the image loss determination unit has determined that an image loss has occurred, and if the recognized characters are not a natural language, determines that content is present around the edge of the medium within the corner area.
(Appendix 6)
The acquisition unit further acquires a second input image in which a medium is captured before or after the input image,
The image processing device described in Appendix 1 or 2, wherein the content determination unit determines that content exists around the edge of a medium in a corner area determined by the image loss determination unit as having an image loss when content exists in a corner area in the second input image corresponding to the corner area determined by the image loss determination unit as having an image loss.
(Appendix 7)
The image processing device
An input image of a medium is obtained;
Detecting a corner region within the input image that is estimated to include a corner of a medium;
determining whether or not an image defect occurs in the detected corner area;
determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred;
Among the corner areas determined to have image loss, the corner areas determined to have no content around the edge of the medium are corrected, and the corner areas determined to have content around the edge of the medium are not corrected;
outputting the corrected input image;
13. An image processing method comprising:
(Appendix 8)
A control program for an image processing device,
An input image of a medium is obtained;
Detecting a corner region within the input image that is estimated to include a corner of a medium;
determining whether or not an image defect occurs in the detected corner area;
determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred;
Among the corner areas determined to have image loss, the corner areas determined to have no content around the edge of the medium are corrected, and the corner areas determined to have content around the edge of the medium are not corrected;
outputting the corrected input image;
A control program for causing the image processing device to execute the above steps.
(Appendix 9)
an acquisition unit that acquires an input image obtained by capturing an image of a medium;
a detection unit that detects four corner regions in the input image, each of which is estimated to include a corner of a medium;
an image loss determination unit that determines whether or not an image loss occurs in the corner area detected by the detection unit;
a content determination unit that determines whether or not content is present around a medium edge in a corner area in which the image loss determination unit has determined that an image loss has occurred;
a correction unit that corrects a corner region in which the image loss determination unit has determined that an image loss has occurred and in which the content determination unit has determined that content exists around the edge of the medium, based on surrounding pixels of the edge of the medium within the corner region;
an output unit that outputs the input image corrected by the correction unit;
13. An image processing device comprising:
(Appendix 10)
Further comprising a communication unit,
The image processing device according to claim 9, wherein the correction unit receives an image including pixels similar to the surrounding pixels via the communication unit, and corrects the corner region based on the received image.
(Appendix 11)
The image processing device according to claim 9 or 10, wherein the correction unit recognizes characters within an area including a corner area in which the image loss determination unit has determined that an image loss has occurred, and corrects the corner area based on a natural language similar to the recognized character.
(Appendix 12)
11. The image processing device according to claim 9, wherein the correction unit corrects the corner area by extending a ruled line cut at an edge of a medium in the corner area where the image loss determination unit has determined that an image loss has occurred.
(Appendix 13)
an acquisition unit that acquires a plurality of input images obtained by capturing an image of a medium;
a detection unit that detects a corner region that is estimated to include a corner of a medium in each of the plurality of input images;
an image loss determination unit that determines whether or not an image loss occurs in the corner area detected by the detection unit;
a content determination unit that determines whether or not content is present around a medium edge in a corner area in which the image loss determination unit has determined that an image loss has occurred;
a correction unit that corrects a corner area in a specific input image among the plurality of input images, the corner area being determined by the image loss determination unit as having an image loss and in which the content determination unit has determined that content exists around an edge of a medium, based on a corresponding corner area in another input image;
an output unit that outputs the input image corrected by the correction unit;
13. An image processing device comprising:
(Appendix 14)
The image processing device
An input image of a medium is obtained;
Detecting four corner regions in the input image, each of which is estimated to include a corner of a medium;
determining whether or not an image defect occurs in the detected corner area;
determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred;
A corner region where it is determined that an image is missing and that content exists around the edge of the medium is corrected based on the surrounding pixels of the edge of the medium within the corner region;
outputting the corrected input image;
13. An image processing method comprising:
(Appendix 15)
The image processing device
Acquire a plurality of input images of the medium;
Detecting corner regions that are estimated to include corners of a medium in each of the plurality of input images;
determining whether or not an image defect occurs in the detected corner area;
determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred;
correcting a corner area in which it is determined that an image loss has occurred in a specific input image among the plurality of input images and that content exists around a medium edge based on a corresponding corner area in another input image;
outputting the corrected input image;
13. An image processing method comprising:
(Appendix 16)
A control program for an image processing device,
An input image of a medium is obtained;
Detecting four corner regions in the input image, each of which is estimated to include a corner of a medium;
determining whether or not an image defect occurs in the detected corner area;
determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred;
A corner region where it is determined that an image is missing and that content exists around the edge of the medium is corrected based on the surrounding pixels of the edge of the medium within the corner region;
outputting the corrected input image;
A control program for causing the image processing device to execute the above steps.
(Appendix 17)
A control program for an image processing device,
Acquire a plurality of input images of the medium;
Detecting a corner region that is estimated to include a corner of a medium in each of the plurality of input images;
determining whether or not an image defect occurs in the detected corner area;
determining whether or not content exists around the edge of the medium in the corner region where it is determined that image loss has occurred;
correcting a corner area in which it is determined that an image loss has occurred in a specific input image among the plurality of input images and that content exists around a medium edge based on a corresponding corner area in another input image;
outputting the corrected input image;
A control program for causing the image processing device to execute the above steps.

1 Image processing system, 100 Image reading device, 200 Information processing device, 221 Acquisition unit, 222 Extraction unit, 223 Detection unit, 224 Image missing determination unit, 225 Content determination unit, 226 Correction unit, 227 Notification unit, 228 Output control unit

Claims (11)

an acquisition unit that acquires an input image obtained by capturing an image of a medium;
a detection unit that detects a plurality of corner regions within the input image, each of which is estimated to include a corner of a medium;
a determination unit that determines whether or not an image defect occurs in each of the plurality of corner regions by comparing the plurality of corner regions detected by the detection unit;
an output unit that outputs information regarding a determination result by the determination unit;
13. An image processing device comprising: The image processing device according to claim 1, further comprising a correction unit that corrects a corner area determined to have image loss using a corner area determined by the determination unit to have no image loss. The image processing device according to claim 1 or 2, wherein the determination unit classifies the plurality of corner regions into groups based on the degree of similarity between the plurality of corner regions, and determines whether or not image loss occurs in the plurality of corner regions based on the number of groups into which the plurality of corner regions are classified. The image processing device according to claim 3, wherein the determination unit identifies corner areas where image loss occurs based on the number of corner areas belonging to each of the classified groups. The image processing device according to claim 1 or 2, wherein the determination unit determines whether or not image loss occurs in each of the plurality of corner regions by comparing the areas of the medium regions in the plurality of corner regions. An extraction unit that extracts edge pixels from the input image,
The determination unit is
calculating a distance between a predetermined edge pixel and an intersection point of two straight lines that are substantially perpendicular to each other and are detected from the edge pixels for each of the plurality of corner regions;
The image processing device according to claim 1 , further comprising: a step of comparing the distances calculated for each of the plurality of corner regions to determine whether or not image loss occurs in each of the plurality of corner regions. an acquisition unit that acquires a plurality of input images obtained by capturing an image of a medium;
a detection unit that detects a corner region that is estimated to include a corner of a medium in each of the plurality of input images;
a determination unit that determines whether or not an image loss occurs in the corner region by comparing the corner regions detected by the detection unit;
an output unit that outputs information regarding a result of the determination by the determination unit;
13. An image processing device comprising: The image processing device
An input image of a medium is obtained;
Detecting a plurality of corner regions within the input image, each of which is estimated to include a corner of a medium;
By comparing the detected corner regions with each other, it is determined whether or not image loss occurs in each of the corner regions;
outputting information regarding the result of the determination;
13. An image processing method comprising: The image processing device
Acquire a plurality of input images of the medium;
Detecting a corner region that is estimated to include a corner of a medium in each of the plurality of input images;
By comparing the detected corner areas with each other, it is determined whether or not an image loss occurs in the corner area;
outputting information regarding the result of the determination;
13. An image processing method comprising: A control program for an image processing device,
An input image of a medium is obtained;
Detecting a plurality of corner regions within the input image, each of which is estimated to include a corner of a medium;
By comparing the detected corner regions with each other, it is determined whether or not image loss occurs in each of the corner regions;
outputting information regarding the result of the determination;
A control program for causing the image processing device to execute the above steps. A control program for an image processing device,
Acquire a plurality of input images of the medium;
Detecting a corner region that is estimated to include a corner of a medium in each of the plurality of input images;
By comparing the detected corner areas with each other, it is determined whether or not an image loss occurs in the corner area;
outputting information regarding the result of the determination;
A control program for causing the image processing device to execute the above steps.

Images