Nothing Special   »   [go: up one dir, main page]

US20220129211A1 - Converting voice command into text code blocks that support printing services - Google Patents

Converting voice command into text code blocks that support printing services Download PDF

Info

Publication number
US20220129211A1
US20220129211A1 US17/571,983 US202217571983A US2022129211A1 US 20220129211 A1 US20220129211 A1 US 20220129211A1 US 202217571983 A US202217571983 A US 202217571983A US 2022129211 A1 US2022129211 A1 US 2022129211A1
Authority
US
United States
Prior art keywords
text code
image forming
code blocks
forming apparatus
voice command
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US17/571,983
Inventor
Sung Jin Park
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hewlett Packard Development Co LP
Original Assignee
Hewlett Packard Development Co LP
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hewlett Packard Development Co LP filed Critical Hewlett Packard Development Co LP
Priority to US17/571,983 priority Critical patent/US20220129211A1/en
Assigned to HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. reassignment HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. CONFIRMATORY ASSIGNMENT Assignors: HP PRINTING KOREA CO., LTD.
Assigned to HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. reassignment HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: HP PRINTING KOREA CO., LTD.
Assigned to HP PRINTING KOREA CO., LTD. reassignment HP PRINTING KOREA CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: PARK, SUNG JIN
Publication of US20220129211A1 publication Critical patent/US20220129211A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31User authentication
    • G06F21/32User authentication using biometric data, e.g. fingerprints, iris scans or voiceprints
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60Protecting data
    • G06F21/606Protecting data by securing the transmission between two devices or processes
    • G06F21/608Secure printing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/12Digital output to print unit, e.g. line printer, chain printer
    • G06F3/1201Dedicated interfaces to print systems
    • G06F3/1202Dedicated interfaces to print systems specifically adapted to achieve a particular effect
    • G06F3/1203Improving or facilitating administration, e.g. print management
    • G06F3/1205Improving or facilitating administration, e.g. print management resulting in increased flexibility in print job configuration, e.g. job settings, print requirements, job tickets
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/12Digital output to print unit, e.g. line printer, chain printer
    • G06F3/1201Dedicated interfaces to print systems
    • G06F3/1202Dedicated interfaces to print systems specifically adapted to achieve a particular effect
    • G06F3/1222Increasing security of the print job
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/12Digital output to print unit, e.g. line printer, chain printer
    • G06F3/1201Dedicated interfaces to print systems
    • G06F3/1223Dedicated interfaces to print systems specifically adapted to use a particular technique
    • G06F3/1236Connection management
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/12Digital output to print unit, e.g. line printer, chain printer
    • G06F3/1201Dedicated interfaces to print systems
    • G06F3/1223Dedicated interfaces to print systems specifically adapted to use a particular technique
    • G06F3/1275Print workflow management, e.g. defining or changing a workflow, cross publishing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/12Digital output to print unit, e.g. line printer, chain printer
    • G06F3/1201Dedicated interfaces to print systems
    • G06F3/1278Dedicated interfaces to print systems specifically adapted to adopt a particular infrastructure
    • G06F3/1285Remote printer device, e.g. being remote from client or server
    • G06F3/1289Remote printer device, e.g. being remote from client or server in server-client-printer device configuration, e.g. the server does not see the printer
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/12Digital output to print unit, e.g. line printer, chain printer
    • G06F3/1201Dedicated interfaces to print systems
    • G06F3/1278Dedicated interfaces to print systems specifically adapted to adopt a particular infrastructure
    • G06F3/1292Mobile client, e.g. wireless printing
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16Sound input; Sound output
    • G06F3/167Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/22Procedures used during a speech recognition process, e.g. man-machine dialogue
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00Speaker identification or verification techniques
    • G10L17/22Interactive procedures; Man-machine interfaces
    • G10L17/24Interactive procedures; Man-machine interfaces the user being prompted to utter a password or a predefined phrase
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/12Digital output to print unit, e.g. line printer, chain printer
    • G06F3/1201Dedicated interfaces to print systems
    • G06F3/1202Dedicated interfaces to print systems specifically adapted to achieve a particular effect
    • G06F3/1218Reducing or saving of used resources, e.g. avoiding waste of consumables or improving usage of hardware resources
    • G06F3/122Reducing or saving of used resources, e.g. avoiding waste of consumables or improving usage of hardware resources with regard to computing resources, e.g. memory, CPU

Definitions

  • a user may remotely control an image forming apparatus, such as a printer, a copier, a facsimile machine, a multifunction apparatus, and the like, through a user terminal without having to directly manipulate the image forming apparatus.
  • an image forming apparatus such as a printer, a copier, a facsimile machine, a multifunction apparatus, and the like.
  • Such remote control increases the convenience for the user of the image forming apparatus.
  • various devices may be connected to each other using wired/wireless communication technology, and complex operations may be processed through cloud computing.
  • FIG. 1 is a diagram for explaining a printing service environment including an image forming apparatus, a user terminal, and a cloud server, according to an example;
  • FIG. 2 is a diagram for explaining a configuration and an operation of each of an image forming apparatus, a user terminal, and a cloud server, according to an example;
  • FIG. 3 is a diagram for explaining a process of executing a printing job using a voice command, according to an example
  • FIG. 4 is a diagram for explaining an image forming apparatus and a user terminal performing user authentication using voice passwords, according to an example
  • FIG. 5 is a diagram for explaining an instructions mapping table specialized for a printing service used when a cloud server converts a voice command into text code blocks, according to an example
  • FIG. 6 is a diagram for explaining an example in which a cloud server converts a voice command into text code blocks based on a language habit of a user, according to an example
  • FIG. 7 is a diagram for explaining text code blocks converted from voice commands and supporting a printing service, according to an example
  • FIG. 8 is a diagram illustrating a user terminal providing a menu corresponding to a selected text code block in response to a user manipulation selecting the text code block, according to an example
  • FIG. 9 is a diagram illustrating a user terminal changing an order of text code blocks in response to a user manipulation moving a text code block, according to an example
  • FIG. 10 is a diagram illustrating a user terminal merging sets of text code blocks, according to an example.
  • FIG. 11 is a diagram for explaining an example in which a user interface device of a user terminal and a user interface device of an image forming apparatus interact with each other and the user interface devices simultaneously process multiple manipulations of a user, according to an example.
  • an element when an element is “connected” to another element, the elements may not only be “directly connected,” but may also be “electrically connected” or otherwise “indirectly connected” via another element there between. Also, when a region “includes” an element, the region may further include another element instead of excluding the other element, unless otherwise differently stated.
  • FIG. 1 is a diagram for explaining a printing service environment including an image forming apparatus, a user terminal, and a cloud server, according to an example.
  • an image forming apparatus 100 may perform image forming operations such as printing, scanning, copying, or faxing.
  • the image forming apparatus 100 may form an image on a recording medium such as a printing paper by using various printing methods such as an electrophotographic method, an inkjet method, a thermal transfer method, a direct thermal method, or the like.
  • a user terminal 200 may be a computing device such as a smart phone, a notebook, or the like.
  • a cloud server 300 may support a printing service and provide a response to a request when requested by the image forming apparatus 100 or the user terminal 200 .
  • FIG. 1 a schematic sequence in which the user terminal 200 controls the image forming apparatus 100 to connect the image forming apparatus 100 to the user terminal 200 to perform a printing job is shown in sequences ⁇ circle around ( 1 ) ⁇ , ⁇ circle around ( 2 ) ⁇ , ⁇ circle around ( 3 ) ⁇ , and ⁇ circle around ( 4 ) ⁇ .
  • the user terminal 200 may access the image forming apparatus 100 supporting the printing service and execute a printing application supporting a voice command installed in the user terminal 200 . Since the printing application supporting the voice command is executed, the user terminal 200 may scan an advertising signal repeatedly transmitted from the neighboring image forming apparatus 100 and obtain device information of the image forming apparatus 100 .
  • the advertising signal may be a Bluetooth low energy (BLE) beacon signal, a wireless fidelity (Wi-Fi) aware signal, or the like.
  • the user terminal 200 may request a connection of the image forming apparatus 100 using the device information of the image forming apparatus 100 through communication between the user terminal 200 and the cloud server 300 and confirm that the printing service using the voice command is supported.
  • the cloud server 300 may inform the image forming apparatus 100 of a connection request of the user terminal 200 and establish a connection between voice command sessions of the image forming apparatus 100 and the cloud server 300 .
  • the cloud server 300 may receive a voice command from the user terminal 200 and transmit a request for executing a printing job related to the received voice command to the image forming apparatus 100 through voice recognition and natural language processing.
  • FIG. 2 is a diagram for explaining a configuration and an operation of each of an image forming apparatus, a user terminal, and a cloud server, according to an example.
  • the image forming apparatus 100 may include a user interface device 110 , a controller 120 , a communicator 130 , a memory 140 , and an image forming operator 150 . Further, although not shown, the image forming apparatus 100 may include a power supply for supplying power to the respective components.
  • the user interface device 110 may include an input unit for receiving an input for performing an image forming operation from a user and an output unit for displaying information about a result of the image forming operation or a state of the image forming apparatus 100 .
  • the user interface device 110 may include an operation panel for receiving a user input and a display panel for displaying a screen.
  • the input may include devices capable of receiving various types of user input such as a keyboard, a physical button, a touch screen, a camera, a microphone, or the like.
  • the output unit may include, for example, a display panel, a speaker, or the like.
  • the present disclosure is not limited to this, and the user interface device 110 may include various input/output supporting devices.
  • the controller 120 may control an operation of the image forming apparatus 100 and may include at least one processor such as a central processing unit (CPU).
  • the controller 120 may control other components included in the image forming apparatus 100 to perform an operation corresponding to the user input received through the user interface device 110 .
  • the controller 120 may include at least one specialized processor corresponding to each function, or may be a single integrated processor.
  • the controller 120 may execute a program stored in the memory 140 , read data or a file stored in the memory 140 , or store a new file in the memory 140 .
  • the communicator 130 may perform wired/wireless communication with another device or a network.
  • the communicator 130 may include a communication module, such as a transceiver, supporting at least one of various wired/wireless communication methods.
  • the communication module may be in the form of a chipset, or may be a sticker/barcode (e.g., a sticker including a near-field communication (NFC) tag) including information necessary for communication.
  • NFC near-field communication
  • Wireless communication may include, for example, at least one of Wi-Fi, Wi-Fi Direct, Bluetooth, BLE, ultra-wide band (UWB), NFC, or the like.
  • Wired communication may include, for example, at least one of universal serial bus (USB), High-Definition Multimedia Interface (HDMI), or the like.
  • USB universal serial bus
  • HDMI High-Definition Multimedia Interface
  • the communicator 130 may be connected to an external device located outside the image forming apparatus 100 and may transmit and receive signals or data.
  • the communicator 130 may transmit signals or data received from the user terminal 200 or the cloud server 300 to the controller 120 or transmit signals or data generated by the controller 120 to the user terminal 200 or the cloud server 300 .
  • the communicator 130 may be directly connected to the cloud server 300 and may transmit and receive signals or data.
  • the communicator 130 may be connected to the user terminal 200 through the cloud server 300 . That is, the communicator 130 of the image forming apparatus 100 may transmit/receive signals or data to/from a communicator 230 of the user terminal 200 through the cloud server 300 .
  • the memory 140 may install and store programs such as applications and various types of data such as files.
  • the controller 120 may access the data stored in the memory 140 and use the data or store new data in the memory 140 . Further, the controller 120 may execute the programs installed in the memory 140 , and may install an application received from outside through the communicator 130 in the memory 140 .
  • the image forming operator 150 may perform an image forming operation such as printing, scanning, copying, or faxing.
  • the image forming operator 150 is shown as including a printing unit 151 , a scanning unit 152 , and a fax unit 153 .
  • the image forming operator 150 may include only some of these components or may further include components for performing other kinds of image forming operations.
  • the printing unit 151 may form an image on a recording medium by using various printing methods such as an electrophotographic method, an inkjet method, a thermal transfer method, a direct thermal method, or the like.
  • the scanning unit 152 may irradiate light onto a document, receive the reflected light, and read an image recorded in the document.
  • a charge-coupled device (CCD) or a contact image sensor (CIS) may be employed as an image sensor for reading the image from the document.
  • the fax unit 153 may share a configuration for scanning an image with the scanning unit 152 , may share a configuration for printing a received file with the printing unit 151 , may transmit a scan file to a destination, or may receive a file from outside.
  • the user terminal 200 may include a processor 210 , a user interface device 220 , the communicator 230 , and a memory 240 .
  • the processor 210 may control an operation of the user terminal 200 and may include at least one processing unit such as a CPU.
  • the processor 210 may control other components included in the user terminal 200 to perform operations corresponding to user inputs received via the user interface device 220 .
  • the processor 210 may include at least one specialized processing unit corresponding to each function.
  • the processor 210 may execute a program stored in the memory 240 , read data or files stored in the memory 240 , or store a new file in the memory 240 .
  • the user interface device 220 may include an input unit for obtaining a user input and an output unit for providing information to the user.
  • the input unit may be an operation panel capable of receiving a user manipulation or a microphone capable of acquiring a voice or noise of the user.
  • the output unit may be a display panel for displaying a screen or a speaker.
  • the user terminal 200 may be connected to an external device such as the image forming apparatus 100 or the cloud server 300 through the communicator 230 .
  • the communicator 230 may include a transceiver and support at least one of Wi-Fi, Wi-Fi Direct, Bluetooth, BLE, UWB, 3G, 4G, and 5G.
  • the memory 240 may store instructions executable by the processor 210 .
  • the memory 240 may install and store programs such as applications and various types of data such as files.
  • the memory 240 may store an operating system.
  • the cloud server 300 may include a memory 310 , a processor 320 , and a communicator 330 .
  • the memory 310 may store instructions executable by the processor 320 .
  • the processor 320 may control an operation of the cloud server 300 and may include at least one processing unit.
  • the communicator 330 may include a communication module, such as a transceiver, that may communicate with the communicator 130 of the image forming apparatus 100 or the communicator 230 of the user terminal 200 .
  • the image forming apparatus 100 , the user terminal 200 , and the cloud server 300 described above may be used to perform a printing operation through a printing service supporting voice commands.
  • the processor 210 of the user terminal 200 may execute commands stored in the memory 240 to transmit voice commands obtained from the user to the cloud server 300 providing the printing service.
  • the processor 320 of the cloud server 300 may control the communicator 330 to execute the commands stored in the memory 310 , convert the voice command received from the user terminal 200 through the communicator 330 into text code blocks supporting the printing service, and transmit the text code blocks to the user terminal 200 .
  • the processor 320 of the cloud server 300 may control the communicator 330 to search for text code blocks corresponding to a command that is similar to a command of the text code blocks in terms of pronunciation or function and transmit the retrieved text code blocks to the user terminal 200 .
  • the processor 210 of the user terminal 200 may control the communicator 230 to receive the text code blocks supporting the printing service from the cloud server 300 and may control the user interface device 220 to display the text code blocks on the user interface device 220 .
  • the printing job execution request may be transmitted to the image forming apparatus 100 through the cloud server 300 , and the printing operation may be performed.
  • FIG. 3 a process of performing a printing job using a voice command will be described with reference to FIG. 3 .
  • FIG. 3 is a diagram for explaining a process of executing a printing job using a voice command, according to an example.
  • the cloud server 300 may receive device information about the image forming apparatus 100 from the image forming apparatus 100 in operation S 305 .
  • the cloud server 300 may register the device information of the image forming apparatus 100 by storing the device information about the image forming apparatus 100 in operation S 310 .
  • a user who wishes to execute the printing job may access the image forming apparatus 100 with the user terminal 200 in operation S 315 .
  • the image forming apparatus 100 may repeatedly transmit an advertising signal including the device information on a space corresponding to a predetermined range in operation S 320 .
  • the advertising signal may be a BLE beacon signal or a Wi-Fi aware signal.
  • the user terminal 200 may receive the advertising signal repeatedly transmitted from the image forming apparatus 100 and use the device information about the image forming apparatus 100 to use a printing service.
  • each of the user terminal 200 and the image forming apparatus 100 may obtain sound information generated in the image forming apparatus 100 or a keyword phrase spoken by the user as a voice password in operations S 325 and S 330 .
  • the predetermined space may be a region within a range that is reached by the advertising signal repeatedly transmitted by the image forming apparatus 100 and may be a space in which each of the image forming apparatus 100 and the user terminal 200 collect the voice password together. Since the user terminal 200 and the image forming apparatus 100 collect sound generated in the predetermined space together, each of the user terminal 200 and the image forming apparatus 100 may collect similar sound as the voice password.
  • the user terminal 200 and the image forming apparatus 100 may transmit the device information and the voice password to the cloud server 300 for user authentication in operations S 335 and S 340 .
  • the cloud server 300 may confirm whether the voice passwords respectively received from the image forming apparatus 100 and the user terminal 200 match with each other, perform user authentication, and establish interconnection through the cloud server 300 in operation S 345 .
  • the image forming apparatus 100 and the user terminal 200 may obtain information necessary for establishing a connection between the image forming apparatus 100 and the user terminal 200 from the cloud server 300 .
  • a security method using both a network for receiving the advertising signal repeatedly transmitted by the image forming apparatus 100 and the voice password based on sound generated in the predetermined space for collecting the voice password together is applied, logical security and physical security may be performed together.
  • FIG. 4 is a diagram for explaining an image forming apparatus and a user terminal performing user authentication using voice passwords, according to an example.
  • each of the user terminal 200 and the image forming apparatus 100 may obtain sound information (in the case of ⁇ circle around ( 1 ) ⁇ ) generated in the image forming apparatus 100 or a keyword phrase (in the case of ⁇ circle around ( 2 ) ⁇ ) spoken by a user as the voice password and transmit the voice password to the cloud server 300 .
  • the cloud server 300 may confirm whether the voice passwords respectively received from the image forming apparatus 100 and the user terminal 200 match with each other, perform user authentication, and establish interconnection through the cloud server 300 .
  • the cloud server 300 may determine whether the voice passwords match based on unique features identified in the voice passwords. As an example, the cloud server 300 may confirm whether there is a match between the two voice passwords through a convolution operation on the received voice passwords to confirm whether the voice passwords match. When the voice passwords match, it may be proved that the user terminal 200 and the image forming apparatus 100 are in the same space in the physical sense.
  • the user terminal 200 may obtain the voice command from the user who wishes to perform the printing job in operation S 350 .
  • the user terminal 200 may transmit the voice command obtained from the user to the cloud server 300 providing the printing service in operation S 355 .
  • the cloud server 300 may convert the voice command received from the user terminal 200 into text code blocks supporting the printing service in operation S 360 .
  • the cloud server 300 may analyze a relationship between the received voice command and a previous voice command to determine a context of the printing job according to a request of the user terminal 200 and convert the received voice command into the text code blocks based on the context.
  • the cloud server 300 may accurately determine a type of the printing job or a current situation that the user wishes to execute through the continuous voice commands.
  • the cloud server 300 may generate the text code blocks corresponding to the voice command based on an instructions mapping table specialized for the printing service for one voice command or a language habit of the user.
  • FIG. 5 is a diagram for explaining an instructions mapping table specialized for a printing service used when a cloud server converts a voice command into text code blocks, according to an example.
  • the cloud server 300 may convert the received voice command into the text code blocks based on the instructions mapping table specialized for the printing service.
  • the instructions mapping table specialized for a printing service is shown, but the instructions mapping table is not limited thereto.
  • the cloud server 300 interprets the received voice command and what functions the cloud server 300 generates the text code blocks to perform. For example, when the voice command is “upload,” the cloud server 300 may interpret the voice command as a “scan” function and generate the text code blocks to scan a print document and then upload the scanned print document to a predetermined space. Also, the cloud server 300 may accurately identify features of the document for voice commands of similar format such as “latest document,” “recent document,” and “final document” and determine the document on which the printing job is to be performed.
  • the cloud server 300 may limitedly interpret the expressions as being related to the printing job and reflect the expressions to generate the text code blocks.
  • the expression “small” may be interpreted to mean lowering of a voice, but in the instructions mapping table specialized for the printing service, the cloud server 300 may interpret the expression as meaning to reduce output of the printing job and generate the text code blocks.
  • FIG. 6 is a diagram for explaining an example in which a cloud server converts a voice command into text code blocks based on a language habit of a user, according to an example.
  • the cloud server 300 may convert the received voice command into the text code blocks based on a learned language habit of a user.
  • the cloud server 300 may learn the language habit of the user through machine learning based on a language learning model and accurately determine the intended or desired meaning of the voice command intended by the user based on the learned language habit.
  • a value of a reduction option may be determined based on the language habit of the existing user with respect to the value of the reduction option.
  • there may be derived commands such as “slightly small (10-20%),” “moderately small (30%),” and “very small (50%)” with respect to the expression “small.”
  • the cloud server 300 may determine “small” spoken by a user A as a command to reduce “moderately small (30%)” based on the language habit of the user and a history of past printing jobs and accordingly generate the text code blocks.
  • the cloud server 300 may transmit the text code blocks to the user terminal 200 in operation S 365 .
  • the cloud server 300 may search for not only the text code blocks corresponding to the voice command but also text code blocks corresponding to a command that is similar to a command of the text code blocks in terms of pronunciation or function and transmit the found text code blocks to the user terminal 200 .
  • the user terminal 200 may display the text code blocks supporting the printing service received from the cloud server 300 on the user interface device 220 and receive a user manipulation for the text code blocks and a request for execution of the printing job in operation S 370 . Examples of the text code blocks and the user manipulation with respect to the text code blocks will be described with reference to FIGS. 7 to 10 .
  • FIG. 7 is a diagram for explaining text code blocks converted from voice commands and supporting a printing service, according to an example.
  • the text code blocks converted from each voice command by the cloud server 300 are displayed on the user interface device 220 .
  • a set of the text code blocks corresponding to each voice command may be displayed separately by speech bubbles as shown on the user interface device 220 in FIG. 7 .
  • Each unit voice corresponding to a function of an image forming apparatus constituting the voice command and a printing job option may be separately displayed as a unit code block. For example, as shown in FIG.
  • the voice command “Scan and print 2 copy” may be divided into functions of the image forming apparatus such as “Scan” and “and print” and the printing job option such as “ 2 copy” and may be displayed as a total of three unit code blocks.
  • the unit code blocks may be displayed in different colors.
  • the text code blocks may be stored in the user terminal 200 according to categories such as the user, time, etc., such that the user may confirm history information of the text code blocks executed in the user terminal 200 and reuse the text code blocks.
  • the user may edit the text code blocks through the user interface device 220 of the user terminal 200 to compensate for incompleteness of the voice command or lack of conversion into the text code blocks.
  • FIG. 8 is a diagram illustrating a user terminal providing a menu corresponding to a selected text code block in response to a user manipulation selecting the text code block, according to an example.
  • the user interface device 220 of the user terminal 200 may display a menu including at least one text code block corresponding to a command that is similar to a command of a selected text code block in terms of pronunciation or function below the selected text code block in response to the user manipulation selecting one of text code blocks.
  • a menu including text code blocks “Fax” and “Download” corresponding to a functionally similar command that may be performed in place of “Scan” may be displayed below the selected text code block.
  • FIG. 9 is a diagram illustrating a user terminal changing an order of text code blocks in response to a user manipulation moving a text code block, according to an example.
  • the user interface device 220 of the user terminal 200 may change and display the order of the text code blocks according to a position of the moved text code block in response to the user manipulation of moving one of the text code blocks.
  • FIG. 9 it may be seen that when a user selects a text code block “and Print” from among the text code blocks and drags and drops the text code block “and Print” behind a text code block “and Email,” the order of the text code blocks may be changed such that text code blocks “Scan and Print and Email” may be changed to text code blocks “Scan and Email and Print.”
  • the order of the text code blocks may be changed as shown in FIG. 9 .
  • FIG. 10 is a diagram illustrating a user terminal merging sets of text code blocks, according to an example.
  • the user interface device 220 of the user terminal 200 may display overlapped sets as one set of text code blocks in response to a user manipulation of overlapping any one of the sets of text code blocks corresponding to each of a plurality of voice commands with the other set.
  • FIG. 10 it may be seen that when a user performs a user manipulation selecting a set of text code blocks “Scan and Print 2 Copy” and merging the set of text code blocks “Scan and Print 2 Copy” with a set of text code blocks “Send to Marketer By email,” one set of text code blocks “Scan and Print 2 Copy Send to Marketer By email” may be generated and displayed.
  • FIG. 10 it may be seen that when a user performs a user manipulation selecting a set of text code blocks “Scan and Print 2 Copy” and merging the set of text code blocks “Scan and Print 2 Copy” with a set of text code blocks “Send to Marketer By email,” one set of text code blocks “Scan and Print 2 Copy Send to Marketer By email” may be generated and displayed.
  • the two sets of text code blocks may be merged into one set through the user manipulation touching a part of a speech bubble icon corresponding to the set of text code blocks “Scan and Print 2 Copy” and linking parts of a speech bubble icon corresponding to the set of text code blocks “Send to Marketer By email.”
  • the user manipulation merging the sets of text code blocks may use various methods, and thus the present disclosure is not limited thereto.
  • the user terminal 200 may transmit the request for execution of the printing job to the cloud server 300 in operation S 375 .
  • the replaceable information may be replaced with the corresponding information stored in the user terminal 200 at the request of execution of the printing job and transmitted to the image forming apparatus 100 through the cloud server 300 .
  • the request for execution of the printing job may be transmitted after replacing “Marketer” with the corresponding email address information.
  • the cloud server 300 may transmit the request for execution of the printing job to the image forming apparatus 100 corresponding to the printing job according to the request for execution of the printing job received from the user terminal 200 in operation S 380 .
  • the image forming apparatus 100 may execute the printing job in response to the request for execution of the printing job in operation S 385 .
  • the user when the user attempts to start the printing service supported by the cloud server 300 using the user terminal 200 or requests execution of the printing job from the image forming apparatus 100 , there may be cases in which user multi-manipulation is required. For example, there may be a case in which the user wishes to view content displayed on the user interface device 220 and perform a user manipulation on the displayed text code blocks while instructing a microphone of the user terminal 200 with a voice command. In such a case, it may be difficult for the user to simultaneously process the user multi-manipulation with one user terminal 200 .
  • FIG. 11 is a diagram for explaining an example in which a user interface device of a user terminal and a user interface device of an image forming apparatus interact with each other and the user interface devices simultaneously process multiple manipulations of a user, according to an example.
  • the user interface device 220 of the user terminal 200 may interact with the user interface device 110 of the image forming apparatus 100 performing a printing job according to a request for execution by the user terminal 200 in real time and simultaneously process the user multi-manipulation for progress of the printing job with the user interface device 110 of the image forming apparatus 100 .
  • the user interface device 220 of the user terminal 200 may interact with the user interface device 110 of the image forming apparatus 100 and provide the same screen in real time. In this case, the user may make a voice command using the user terminal 200 and simultaneously edit text code blocks displayed on the user interface device 110 of the image forming apparatus 100 .
  • the real time interaction between the user interface device 220 of the user terminal 200 and the user interface device 110 of the image forming apparatus 100 may be performed in a wireless local area network between the user terminal 200 and the image forming apparatus 100 and printing data for the printing job may be transmitted to the image forming apparatus 100 through the cloud server 300 .
  • immediate feedback may be transmitted to the user terminal 200 and the image forming apparatuses 100 through a communication using the wireless local area network between the user terminal 200 and the image forming apparatuses 100 without passing through the cloud server 300 .
  • the above description may be implemented in the form of a non-transitory computer-readable storage medium storing instructions or data executable by a computer or a processor.
  • the above description may be implemented in a general-purpose digital computer, which may be embodied as a program executable on a computer and executes such a program using the non-transitory computer-readable storage medium.
  • Such non-transitory computer-readable storage mediums may include read only memory (ROM), random access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-DVD-Rs, DVD-Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tape, floppy disk, an optical magnetic data storage device, an optical data storage device, hard disk, solid-state disk (SSD), and any device capable of storing instructions or software, associated data, data files, and data structures and providing instructions or software, associated data, data files, and data structures to the processor or the computer such that the processor or the computer may execute instructions.
  • ROM read only memory
  • RAM random access memory
  • flash memory CD-ROMs, CD-Rs, CD+Rs, CD-DVD-Rs, DVD-Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Security & Cryptography (AREA)
  • Multimedia (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • General Health & Medical Sciences (AREA)
  • Acoustics & Sound (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • Bioethics (AREA)
  • Facsimiles In General (AREA)
  • Accessory Devices And Overall Control Thereof (AREA)

Abstract

A cloud server for converting a voice command into text code blocks that support a printing service is presented. The cloud server includes a communicator, a processor, and a memory storing instructions executable by the processor. The processor executes the instructions to convert a voice command received from a user terminal through the communicator into text code blocks supporting a printing service and to control the communicator to transmit the text code blocks to the user terminal.

Description

    BACKGROUND
  • A user may remotely control an image forming apparatus, such as a printer, a copier, a facsimile machine, a multifunction apparatus, and the like, through a user terminal without having to directly manipulate the image forming apparatus. Such remote control increases the convenience for the user of the image forming apparatus.
  • To provide a printing service, various devices may be connected to each other using wired/wireless communication technology, and complex operations may be processed through cloud computing.
  • BRIEF DESCRIPTION OF DRAWINGS
  • The above and other aspects, features, and advantages of certain examples of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:
  • FIG. 1 is a diagram for explaining a printing service environment including an image forming apparatus, a user terminal, and a cloud server, according to an example;
  • FIG. 2 is a diagram for explaining a configuration and an operation of each of an image forming apparatus, a user terminal, and a cloud server, according to an example;
  • FIG. 3 is a diagram for explaining a process of executing a printing job using a voice command, according to an example;
  • FIG. 4 is a diagram for explaining an image forming apparatus and a user terminal performing user authentication using voice passwords, according to an example;
  • FIG. 5 is a diagram for explaining an instructions mapping table specialized for a printing service used when a cloud server converts a voice command into text code blocks, according to an example;
  • FIG. 6 is a diagram for explaining an example in which a cloud server converts a voice command into text code blocks based on a language habit of a user, according to an example;
  • FIG. 7 is a diagram for explaining text code blocks converted from voice commands and supporting a printing service, according to an example;
  • FIG. 8 is a diagram illustrating a user terminal providing a menu corresponding to a selected text code block in response to a user manipulation selecting the text code block, according to an example;
  • FIG. 9 is a diagram illustrating a user terminal changing an order of text code blocks in response to a user manipulation moving a text code block, according to an example;
  • FIG. 10 is a diagram illustrating a user terminal merging sets of text code blocks, according to an example; and
  • FIG. 11 is a diagram for explaining an example in which a user interface device of a user terminal and a user interface device of an image forming apparatus interact with each other and the user interface devices simultaneously process multiple manipulations of a user, according to an example.
  • Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, parts, components, and structures.
  • DETAILED DESCRIPTION OF EXAMPLES
  • Hereinafter, various examples will be described with reference to the accompanying drawings. The examples described may be modified in many different forms. To more clearly describe features of examples, matters well known to one of ordinary skill in the art to which the below examples pertain will not be described in detail. In the following description and drawings, components having substantially the same function and configuration are denoted by the same reference numerals, and redundant description thereof will be omitted.
  • In the specification, when an element is “connected” to another element, the elements may not only be “directly connected,” but may also be “electrically connected” or otherwise “indirectly connected” via another element there between. Also, when a region “includes” an element, the region may further include another element instead of excluding the other element, unless otherwise differently stated.
  • FIG. 1 is a diagram for explaining a printing service environment including an image forming apparatus, a user terminal, and a cloud server, according to an example.
  • Referring to FIG. 1, an image forming apparatus 100 may perform image forming operations such as printing, scanning, copying, or faxing. The image forming apparatus 100 may form an image on a recording medium such as a printing paper by using various printing methods such as an electrophotographic method, an inkjet method, a thermal transfer method, a direct thermal method, or the like. A user terminal 200 may be a computing device such as a smart phone, a notebook, or the like. A cloud server 300 may support a printing service and provide a response to a request when requested by the image forming apparatus 100 or the user terminal 200.
  • In FIG. 1, a schematic sequence in which the user terminal 200 controls the image forming apparatus 100 to connect the image forming apparatus 100 to the user terminal 200 to perform a printing job is shown in sequences {circle around (1)}, {circle around (2)}, {circle around (3)}, and {circle around (4)}.
  • In the first sequence {circle around (1)}, the user terminal 200 may access the image forming apparatus 100 supporting the printing service and execute a printing application supporting a voice command installed in the user terminal 200. Since the printing application supporting the voice command is executed, the user terminal 200 may scan an advertising signal repeatedly transmitted from the neighboring image forming apparatus 100 and obtain device information of the image forming apparatus 100. For example, the advertising signal may be a Bluetooth low energy (BLE) beacon signal, a wireless fidelity (Wi-Fi) aware signal, or the like.
  • In the second sequence {circle around (2)}, the user terminal 200 may request a connection of the image forming apparatus 100 using the device information of the image forming apparatus 100 through communication between the user terminal 200 and the cloud server 300 and confirm that the printing service using the voice command is supported.
  • In the third sequence {circle around (3)}, the cloud server 300 may inform the image forming apparatus 100 of a connection request of the user terminal 200 and establish a connection between voice command sessions of the image forming apparatus 100 and the cloud server 300.
  • In the fourth sequence {circle around (4)}, the cloud server 300 may receive a voice command from the user terminal 200 and transmit a request for executing a printing job related to the received voice command to the image forming apparatus 100 through voice recognition and natural language processing.
  • Examples of operations of the image forming apparatus 100, the user terminal 200, and the cloud server 300 in a printing service environment supporting the voice command will now be described in more detail with reference to FIGS. 2 to 11.
  • FIG. 2 is a diagram for explaining a configuration and an operation of each of an image forming apparatus, a user terminal, and a cloud server, according to an example.
  • Referring to FIG. 2, the image forming apparatus 100 may include a user interface device 110, a controller 120, a communicator 130, a memory 140, and an image forming operator 150. Further, although not shown, the image forming apparatus 100 may include a power supply for supplying power to the respective components.
  • The user interface device 110 may include an input unit for receiving an input for performing an image forming operation from a user and an output unit for displaying information about a result of the image forming operation or a state of the image forming apparatus 100. For example, the user interface device 110 may include an operation panel for receiving a user input and a display panel for displaying a screen.
  • As an example, the input may include devices capable of receiving various types of user input such as a keyboard, a physical button, a touch screen, a camera, a microphone, or the like. Further, the output unit may include, for example, a display panel, a speaker, or the like. However, the present disclosure is not limited to this, and the user interface device 110 may include various input/output supporting devices.
  • The controller 120 may control an operation of the image forming apparatus 100 and may include at least one processor such as a central processing unit (CPU). The controller 120 may control other components included in the image forming apparatus 100 to perform an operation corresponding to the user input received through the user interface device 110. The controller 120 may include at least one specialized processor corresponding to each function, or may be a single integrated processor.
  • For example, the controller 120 may execute a program stored in the memory 140, read data or a file stored in the memory 140, or store a new file in the memory 140.
  • The communicator 130 may perform wired/wireless communication with another device or a network. To this end, the communicator 130 may include a communication module, such as a transceiver, supporting at least one of various wired/wireless communication methods. For example, the communication module may be in the form of a chipset, or may be a sticker/barcode (e.g., a sticker including a near-field communication (NFC) tag) including information necessary for communication.
  • Wireless communication may include, for example, at least one of Wi-Fi, Wi-Fi Direct, Bluetooth, BLE, ultra-wide band (UWB), NFC, or the like. Wired communication may include, for example, at least one of universal serial bus (USB), High-Definition Multimedia Interface (HDMI), or the like.
  • The communicator 130 may be connected to an external device located outside the image forming apparatus 100 and may transmit and receive signals or data. The communicator 130 may transmit signals or data received from the user terminal 200 or the cloud server 300 to the controller 120 or transmit signals or data generated by the controller 120 to the user terminal 200 or the cloud server 300. The communicator 130 may be directly connected to the cloud server 300 and may transmit and receive signals or data. Also, the communicator 130 may be connected to the user terminal 200 through the cloud server 300. That is, the communicator 130 of the image forming apparatus 100 may transmit/receive signals or data to/from a communicator 230 of the user terminal 200 through the cloud server 300.
  • The memory 140 may install and store programs such as applications and various types of data such as files. The controller 120 may access the data stored in the memory 140 and use the data or store new data in the memory 140. Further, the controller 120 may execute the programs installed in the memory 140, and may install an application received from outside through the communicator 130 in the memory 140.
  • The image forming operator 150 may perform an image forming operation such as printing, scanning, copying, or faxing. In FIG. 2, the image forming operator 150 is shown as including a printing unit 151, a scanning unit 152, and a fax unit 153. However, the image forming operator 150 may include only some of these components or may further include components for performing other kinds of image forming operations.
  • The printing unit 151 may form an image on a recording medium by using various printing methods such as an electrophotographic method, an inkjet method, a thermal transfer method, a direct thermal method, or the like.
  • The scanning unit 152 may irradiate light onto a document, receive the reflected light, and read an image recorded in the document. For example, a charge-coupled device (CCD) or a contact image sensor (CIS) may be employed as an image sensor for reading the image from the document.
  • The fax unit 153 may share a configuration for scanning an image with the scanning unit 152, may share a configuration for printing a received file with the printing unit 151, may transmit a scan file to a destination, or may receive a file from outside.
  • The user terminal 200 may include a processor 210, a user interface device 220, the communicator 230, and a memory 240.
  • The processor 210 may control an operation of the user terminal 200 and may include at least one processing unit such as a CPU. The processor 210 may control other components included in the user terminal 200 to perform operations corresponding to user inputs received via the user interface device 220. The processor 210 may include at least one specialized processing unit corresponding to each function. The processor 210 may execute a program stored in the memory 240, read data or files stored in the memory 240, or store a new file in the memory 240.
  • The user interface device 220 may include an input unit for obtaining a user input and an output unit for providing information to the user. For example, the input unit may be an operation panel capable of receiving a user manipulation or a microphone capable of acquiring a voice or noise of the user. The output unit may be a display panel for displaying a screen or a speaker.
  • The user terminal 200 may be connected to an external device such as the image forming apparatus 100 or the cloud server 300 through the communicator 230. To this end, the communicator 230 may include a transceiver and support at least one of Wi-Fi, Wi-Fi Direct, Bluetooth, BLE, UWB, 3G, 4G, and 5G.
  • The memory 240 may store instructions executable by the processor 210. The memory 240 may install and store programs such as applications and various types of data such as files. The memory 240 may store an operating system.
  • The cloud server 300 may include a memory 310, a processor 320, and a communicator 330. The memory 310 may store instructions executable by the processor 320. The processor 320 may control an operation of the cloud server 300 and may include at least one processing unit. The communicator 330 may include a communication module, such as a transceiver, that may communicate with the communicator 130 of the image forming apparatus 100 or the communicator 230 of the user terminal 200.
  • The image forming apparatus 100, the user terminal 200, and the cloud server 300 described above may be used to perform a printing operation through a printing service supporting voice commands. For example, the processor 210 of the user terminal 200 may execute commands stored in the memory 240 to transmit voice commands obtained from the user to the cloud server 300 providing the printing service. The processor 320 of the cloud server 300 may control the communicator 330 to execute the commands stored in the memory 310, convert the voice command received from the user terminal 200 through the communicator 330 into text code blocks supporting the printing service, and transmit the text code blocks to the user terminal 200. The processor 320 of the cloud server 300 may control the communicator 330 to search for text code blocks corresponding to a command that is similar to a command of the text code blocks in terms of pronunciation or function and transmit the retrieved text code blocks to the user terminal 200. The processor 210 of the user terminal 200 may control the communicator 230 to receive the text code blocks supporting the printing service from the cloud server 300 and may control the user interface device 220 to display the text code blocks on the user interface device 220. In response to a printing job execution request of the user terminal 200 based on the voice command, the printing job execution request may be transmitted to the image forming apparatus 100 through the cloud server 300, and the printing operation may be performed. Hereinafter, a process of performing a printing job using a voice command will be described with reference to FIG. 3.
  • FIG. 3 is a diagram for explaining a process of executing a printing job using a voice command, according to an example.
  • Referring to FIG. 3, the cloud server 300 may receive device information about the image forming apparatus 100 from the image forming apparatus 100 in operation S305.
  • The cloud server 300 may register the device information of the image forming apparatus 100 by storing the device information about the image forming apparatus 100 in operation S310.
  • A user who wishes to execute the printing job may access the image forming apparatus 100 with the user terminal 200 in operation S315.
  • The image forming apparatus 100 may repeatedly transmit an advertising signal including the device information on a space corresponding to a predetermined range in operation S320. At this time, the advertising signal may be a BLE beacon signal or a Wi-Fi aware signal. The user terminal 200 may receive the advertising signal repeatedly transmitted from the image forming apparatus 100 and use the device information about the image forming apparatus 100 to use a printing service.
  • When the user terminal 200 and the image forming apparatus 100 are located in a predetermined space, each of the user terminal 200 and the image forming apparatus 100 may obtain sound information generated in the image forming apparatus 100 or a keyword phrase spoken by the user as a voice password in operations S325 and S330. As an example, the predetermined space may be a region within a range that is reached by the advertising signal repeatedly transmitted by the image forming apparatus 100 and may be a space in which each of the image forming apparatus 100 and the user terminal 200 collect the voice password together. Since the user terminal 200 and the image forming apparatus 100 collect sound generated in the predetermined space together, each of the user terminal 200 and the image forming apparatus 100 may collect similar sound as the voice password.
  • The user terminal 200 and the image forming apparatus 100 may transmit the device information and the voice password to the cloud server 300 for user authentication in operations S335 and S340.
  • The cloud server 300 may confirm whether the voice passwords respectively received from the image forming apparatus 100 and the user terminal 200 match with each other, perform user authentication, and establish interconnection through the cloud server 300 in operation S345. The image forming apparatus 100 and the user terminal 200 may obtain information necessary for establishing a connection between the image forming apparatus 100 and the user terminal 200 from the cloud server 300. When a security method using both a network for receiving the advertising signal repeatedly transmitted by the image forming apparatus 100 and the voice password based on sound generated in the predetermined space for collecting the voice password together is applied, logical security and physical security may be performed together.
  • FIG. 4 is a diagram for explaining an image forming apparatus and a user terminal performing user authentication using voice passwords, according to an example.
  • Referring to FIG. 4, a security method using voice passwords according to operations S325, S330, S335, S340, and S345 described in FIG. 3 is shown. When the user terminal 200 and the image forming apparatus 100 are located in a predetermined space, each of the user terminal 200 and the image forming apparatus 100 may obtain sound information (in the case of {circle around (1)}) generated in the image forming apparatus 100 or a keyword phrase (in the case of {circle around (2)}) spoken by a user as the voice password and transmit the voice password to the cloud server 300. The cloud server 300 may confirm whether the voice passwords respectively received from the image forming apparatus 100 and the user terminal 200 match with each other, perform user authentication, and establish interconnection through the cloud server 300. The cloud server 300 may determine whether the voice passwords match based on unique features identified in the voice passwords. As an example, the cloud server 300 may confirm whether there is a match between the two voice passwords through a convolution operation on the received voice passwords to confirm whether the voice passwords match. When the voice passwords match, it may be proved that the user terminal 200 and the image forming apparatus 100 are in the same space in the physical sense.
  • Referring again to FIG. 3, the user terminal 200 may obtain the voice command from the user who wishes to perform the printing job in operation S350.
  • The user terminal 200 may transmit the voice command obtained from the user to the cloud server 300 providing the printing service in operation S355.
  • The cloud server 300 may convert the voice command received from the user terminal 200 into text code blocks supporting the printing service in operation S360. The cloud server 300 may analyze a relationship between the received voice command and a previous voice command to determine a context of the printing job according to a request of the user terminal 200 and convert the received voice command into the text code blocks based on the context. The cloud server 300 may accurately determine a type of the printing job or a current situation that the user wishes to execute through the continuous voice commands.
  • The cloud server 300 may generate the text code blocks corresponding to the voice command based on an instructions mapping table specialized for the printing service for one voice command or a language habit of the user.
  • FIG. 5 is a diagram for explaining an instructions mapping table specialized for a printing service used when a cloud server converts a voice command into text code blocks, according to an example.
  • Referring to FIG. 5, the cloud server 300 may convert the received voice command into the text code blocks based on the instructions mapping table specialized for the printing service. In FIG. 5, an example of an instructions mapping table specialized for a printing service is shown, but the instructions mapping table is not limited thereto.
  • In the example of FIG. 5, it may be known in what sense the cloud server 300 interprets the received voice command and what functions the cloud server 300 generates the text code blocks to perform. For example, when the voice command is “upload,” the cloud server 300 may interpret the voice command as a “scan” function and generate the text code blocks to scan a print document and then upload the scanned print document to a predetermined space. Also, the cloud server 300 may accurately identify features of the document for voice commands of similar format such as “latest document,” “recent document,” and “final document” and determine the document on which the printing job is to be performed. Also, when the voice command has expressions such as “small,” “large,” “clear,” and “easy to read,” the cloud server 300 may limitedly interpret the expressions as being related to the printing job and reflect the expressions to generate the text code blocks. For example, the expression “small” may be interpreted to mean lowering of a voice, but in the instructions mapping table specialized for the printing service, the cloud server 300 may interpret the expression as meaning to reduce output of the printing job and generate the text code blocks.
  • FIG. 6 is a diagram for explaining an example in which a cloud server converts a voice command into text code blocks based on a language habit of a user, according to an example.
  • Referring to FIG. 6, the cloud server 300 may convert the received voice command into the text code blocks based on a learned language habit of a user. The cloud server 300 may learn the language habit of the user through machine learning based on a language learning model and accurately determine the intended or desired meaning of the voice command intended by the user based on the learned language habit.
  • In the example of FIG. 6, when the voice command includes an expression “small,” a value of a reduction option may be determined based on the language habit of the existing user with respect to the value of the reduction option. In other words, there may be derived commands such as “slightly small (10-20%),” “moderately small (30%),” and “very small (50%)” with respect to the expression “small.” The cloud server 300 may determine “small” spoken by a user A as a command to reduce “moderately small (30%)” based on the language habit of the user and a history of past printing jobs and accordingly generate the text code blocks.
  • Referring again to FIG. 3, the cloud server 300 may transmit the text code blocks to the user terminal 200 in operation S365. The cloud server 300 may search for not only the text code blocks corresponding to the voice command but also text code blocks corresponding to a command that is similar to a command of the text code blocks in terms of pronunciation or function and transmit the found text code blocks to the user terminal 200.
  • The user terminal 200 may display the text code blocks supporting the printing service received from the cloud server 300 on the user interface device 220 and receive a user manipulation for the text code blocks and a request for execution of the printing job in operation S370. Examples of the text code blocks and the user manipulation with respect to the text code blocks will be described with reference to FIGS. 7 to 10.
  • FIG. 7 is a diagram for explaining text code blocks converted from voice commands and supporting a printing service, according to an example.
  • Referring to FIG. 7, with respect to voice commands “Scan and print 2 copy” and “Send to marketer by email” of a user, the text code blocks converted from each voice command by the cloud server 300 are displayed on the user interface device 220. A set of the text code blocks corresponding to each voice command may be displayed separately by speech bubbles as shown on the user interface device 220 in FIG. 7. Each unit voice corresponding to a function of an image forming apparatus constituting the voice command and a printing job option may be separately displayed as a unit code block. For example, as shown in FIG. 7, the voice command “Scan and print 2 copy” may be divided into functions of the image forming apparatus such as “Scan” and “and print” and the printing job option such as “2 copy” and may be displayed as a total of three unit code blocks. The unit code blocks may be displayed in different colors.
  • The text code blocks may be stored in the user terminal 200 according to categories such as the user, time, etc., such that the user may confirm history information of the text code blocks executed in the user terminal 200 and reuse the text code blocks.
  • The user may edit the text code blocks through the user interface device 220 of the user terminal 200 to compensate for incompleteness of the voice command or lack of conversion into the text code blocks.
  • FIG. 8 is a diagram illustrating a user terminal providing a menu corresponding to a selected text code block in response to a user manipulation selecting the text code block, according to an example.
  • Referring to FIG. 8, the user interface device 220 of the user terminal 200 may display a menu including at least one text code block corresponding to a command that is similar to a command of a selected text code block in terms of pronunciation or function below the selected text code block in response to the user manipulation selecting one of text code blocks. In FIG. 8, it may be seen that when a user selects a text code block “Scan” from among the text code blocks, a menu including text code blocks “Fax” and “Download” corresponding to a functionally similar command that may be performed in place of “Scan” may be displayed below the selected text code block.
  • FIG. 9 is a diagram illustrating a user terminal changing an order of text code blocks in response to a user manipulation moving a text code block, according to an example.
  • Referring to FIG. 9, the user interface device 220 of the user terminal 200 may change and display the order of the text code blocks according to a position of the moved text code block in response to the user manipulation of moving one of the text code blocks. In FIG. 9, it may be seen that when a user selects a text code block “and Print” from among the text code blocks and drags and drops the text code block “and Print” behind a text code block “and Email,” the order of the text code blocks may be changed such that text code blocks “Scan and Print and Email” may be changed to text code blocks “Scan and Email and Print.” When the user makes a voice command and determines that it is necessary to change the order that a function requiring a short time is firstly processed in consideration of time required for processing each function, the order of the text code blocks may be changed as shown in FIG. 9.
  • FIG. 10 is a diagram illustrating a user terminal merging sets of text code blocks, according to an example.
  • Referring to FIG. 10, the user interface device 220 of the user terminal 200 may display overlapped sets as one set of text code blocks in response to a user manipulation of overlapping any one of the sets of text code blocks corresponding to each of a plurality of voice commands with the other set. In FIG. 10, it may be seen that when a user performs a user manipulation selecting a set of text code blocks “Scan and Print 2 Copy” and merging the set of text code blocks “Scan and Print 2 Copy” with a set of text code blocks “Send to Marketer By email,” one set of text code blocks “Scan and Print 2 Copy Send to Marketer By email” may be generated and displayed. In FIG. 10, it may be seen that the two sets of text code blocks may be merged into one set through the user manipulation touching a part of a speech bubble icon corresponding to the set of text code blocks “Scan and Print 2 Copy” and linking parts of a speech bubble icon corresponding to the set of text code blocks “Send to Marketer By email.” However, the user manipulation merging the sets of text code blocks may use various methods, and thus the present disclosure is not limited thereto.
  • Referring again to FIG. 3, the user terminal 200 may transmit the request for execution of the printing job to the cloud server 300 in operation S375. At this time, when replaceable information is included in the text code blocks and corresponds to information stored in the user terminal 200, the replaceable information may be replaced with the corresponding information stored in the user terminal 200 at the request of execution of the printing job and transmitted to the image forming apparatus 100 through the cloud server 300. For example, when there are text code blocks “Send to Marketer By email,” and corresponding email address information of “Marketer” is stored in the user terminal 200, the request for execution of the printing job may be transmitted after replacing “Marketer” with the corresponding email address information.
  • The cloud server 300 may transmit the request for execution of the printing job to the image forming apparatus 100 corresponding to the printing job according to the request for execution of the printing job received from the user terminal 200 in operation S380.
  • The image forming apparatus 100 may execute the printing job in response to the request for execution of the printing job in operation S385.
  • In an example, when the user attempts to start the printing service supported by the cloud server 300 using the user terminal 200 or requests execution of the printing job from the image forming apparatus 100, there may be cases in which user multi-manipulation is required. For example, there may be a case in which the user wishes to view content displayed on the user interface device 220 and perform a user manipulation on the displayed text code blocks while instructing a microphone of the user terminal 200 with a voice command. In such a case, it may be difficult for the user to simultaneously process the user multi-manipulation with one user terminal 200.
  • FIG. 11 is a diagram for explaining an example in which a user interface device of a user terminal and a user interface device of an image forming apparatus interact with each other and the user interface devices simultaneously process multiple manipulations of a user, according to an example.
  • Referring to FIG. 11, the user interface device 220 of the user terminal 200 may interact with the user interface device 110 of the image forming apparatus 100 performing a printing job according to a request for execution by the user terminal 200 in real time and simultaneously process the user multi-manipulation for progress of the printing job with the user interface device 110 of the image forming apparatus 100. As shown in FIG. 11, the user interface device 220 of the user terminal 200 may interact with the user interface device 110 of the image forming apparatus 100 and provide the same screen in real time. In this case, the user may make a voice command using the user terminal 200 and simultaneously edit text code blocks displayed on the user interface device 110 of the image forming apparatus 100. To this end, the real time interaction between the user interface device 220 of the user terminal 200 and the user interface device 110 of the image forming apparatus 100 may be performed in a wireless local area network between the user terminal 200 and the image forming apparatus 100 and printing data for the printing job may be transmitted to the image forming apparatus 100 through the cloud server 300. Thus, immediate feedback may be transmitted to the user terminal 200 and the image forming apparatuses 100 through a communication using the wireless local area network between the user terminal 200 and the image forming apparatuses 100 without passing through the cloud server 300.
  • The above description may be implemented in the form of a non-transitory computer-readable storage medium storing instructions or data executable by a computer or a processor. The above description may be implemented in a general-purpose digital computer, which may be embodied as a program executable on a computer and executes such a program using the non-transitory computer-readable storage medium. Such non-transitory computer-readable storage mediums may include read only memory (ROM), random access memory (RAM), flash memory, CD-ROMs, CD-Rs, CD+Rs, CD-DVD-Rs, DVD-Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, magnetic tape, floppy disk, an optical magnetic data storage device, an optical data storage device, hard disk, solid-state disk (SSD), and any device capable of storing instructions or software, associated data, data files, and data structures and providing instructions or software, associated data, data files, and data structures to the processor or the computer such that the processor or the computer may execute instructions.
  • While one or more examples have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.

Claims (20)

What is claimed is:
1. A method comprising:
receiving a voice command for a printing job;
obtaining converted text code blocks supporting a printing service from the received voice command;
transmitting the text code blocks to an image forming apparatus;
when replaceable information is included in the text code blocks and corresponds to stored information, obtaining the corresponding stored information as part of a request for execution of the printing job; and
transmitting the request for execution of the printing job using the corresponding information to the image forming apparatus.
2. The method of claim 1, further comprising:
analyzing a relationship between the received voice command and a previous voice command; and
determining a context of the printing job,
wherein the text code blocks are converted from the received voice command based on the context.
3. The method of claim 1, wherein the text code blocks are converted from the received voice command based on an instructions mapping table specialized for the printing service.
4. The method of claim 1, wherein the text code blocks are converted from the received voice command based on a learned language habit of a user.
5. The method of claim 1, further comprising:
searching for at least one text code block corresponding to a command that is similar to a command of the text code blocks in terms of pronunciation or function; and
transmitting a found at least one text code block to the image forming apparatus.
6. The method of claim 1, further comprising performing user authentication based on a signal transmitted by the image forming apparatus.
7. The method of claim 6, wherein the signal transmitted by the image forming apparatus comprises a near field communication signal.
8. The method of claim 6, wherein the signal transmitted by the image forming apparatus comprises an advertising signal including at least one of a Bluetooth low energy (BLE) beacon signal or a wireless fidelity (Wi-Fi) aware signal.
9. A method comprising:
receiving a text code block for a printing service, the text code block obtained by converting a voice command for a printing job;
when replaceable information is included in the text code block and corresponds to information stored in a memory, transmitting the corresponding information as part of a request for execution of the printing job; and
receiving the request for execution of the printing job using the corresponding information.
10. The method of claim 9, further comprising transmitting a signal to perform user authentication.
11. The method of claim 10, wherein the transmitted signal comprises a near field communication signal.
12. The method of claim 10, wherein the transmitted signal comprises an advertising signal including at least one of a Bluetooth low energy (BLE) beacon signal or a wireless fidelity (Wi-Fi) aware signal.
13. A device comprising:
a user interface device;
a communicator;
a processor; and
a memory storing instructions executable by the processor,
wherein the processor executes the instructions to:
receive a voice command for a printing job,
obtain converted text code blocks supporting a printing service from the received voice command,
transmit the text code blocks to an image forming apparatus,
when replaceable information is included in the text code blocks and corresponds to stored information, obtain the corresponding stored information as part of a request for execution of the printing job, and
transmit the request for execution of the printing job using the corresponding information to the image forming apparatus.
14. The device of claim 13, wherein the processor further executes the instructions to:
analyze a relationship between the received voice command and a previous voice command, and
determine a context of the printing job,
wherein the text code blocks are converted from the received voice command based on the context.
15. The device of claim 13, wherein the text code blocks are converted from the received voice command based on an instructions mapping table specialized for the printing service.
16. The device of claim 13, wherein the text code blocks are converted from the received voice command based on a learned language habit of a user.
17. The device of claim 13, wherein the processor further executes the instructions to:
search for at least one text code block corresponding to a command that is similar to a command of the text code blocks in terms of pronunciation or function, and
transmit a found at least one text code block to the image forming apparatus.
18. The device of claim 13, wherein the processor further executes the instructions to perform user authentication based on a signal transmitted by the image forming apparatus.
19. The device of claim 18, wherein the signal transmitted by the image forming apparatus comprises a near field communication signal.
20. The device of claim 18, wherein the signal transmitted by the image forming apparatus comprises an advertising signal including at least one of a Bluetooth low energy (BLE) beacon signal or a wireless fidelity (Wi-Fi) aware signal.
US17/571,983 2018-05-31 2022-01-10 Converting voice command into text code blocks that support printing services Abandoned US20220129211A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/571,983 US20220129211A1 (en) 2018-05-31 2022-01-10 Converting voice command into text code blocks that support printing services

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
KR1020180063014A KR20190136832A (en) 2018-05-31 2018-05-31 Converting voice command into text code blcoks that support printing services
KR10-2018-0063014 2018-05-31
PCT/KR2018/011914 WO2019231055A1 (en) 2018-05-31 2018-10-11 Converting voice command into text code blocks that support printing services
US202017049718A 2020-10-22 2020-10-22
US17/571,983 US20220129211A1 (en) 2018-05-31 2022-01-10 Converting voice command into text code blocks that support printing services

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
US17/049,718 Continuation US11249696B2 (en) 2018-05-31 2018-10-11 Converting voice command into text code blocks that support printing services
PCT/KR2018/011914 Continuation WO2019231055A1 (en) 2018-05-31 2018-10-11 Converting voice command into text code blocks that support printing services

Publications (1)

Publication Number Publication Date
US20220129211A1 true US20220129211A1 (en) 2022-04-28

Family

ID=68697645

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/049,718 Active US11249696B2 (en) 2018-05-31 2018-10-11 Converting voice command into text code blocks that support printing services
US17/571,983 Abandoned US20220129211A1 (en) 2018-05-31 2022-01-10 Converting voice command into text code blocks that support printing services

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US17/049,718 Active US11249696B2 (en) 2018-05-31 2018-10-11 Converting voice command into text code blocks that support printing services

Country Status (4)

Country Link
US (2) US11249696B2 (en)
EP (1) EP3718104A4 (en)
KR (1) KR20190136832A (en)
WO (1) WO2019231055A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3910466A1 (en) * 2020-04-27 2021-11-17 Ricoh Company, Ltd. Information processing apparatus, information processing system, method for processing information, and carrier means
CN112286745B (en) * 2020-10-27 2022-11-15 上海商米科技集团股份有限公司 Automatic testing method for cloud printer

Family Cites Families (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4466572B2 (en) 2006-01-16 2010-05-26 コニカミノルタビジネステクノロジーズ株式会社 Image forming apparatus, voice command execution program, and voice command execution method
KR101631496B1 (en) * 2008-06-03 2016-06-17 삼성전자주식회사 Robot apparatus and method for registrating contracted commander thereof
WO2012116110A1 (en) 2011-02-22 2012-08-30 Speak With Me, Inc. Hybridized client-server speech recognition
JP5234160B2 (en) * 2011-03-23 2013-07-10 株式会社デンソー Vehicle apparatus and information display system
US9329808B2 (en) * 2011-03-24 2016-05-03 Ricoh Company, Ltd. User interfaces for rule-based workflow generation in a print shop environment
JP5831251B2 (en) 2012-01-25 2015-12-09 コニカミノルタ株式会社 Image forming system, image forming apparatus, terminal device, control program for image forming apparatus, and control program for terminal device
KR101383552B1 (en) 2013-02-25 2014-04-10 미디어젠(주) Speech recognition method of sentence having multiple instruction
JP6171511B2 (en) 2013-04-09 2017-08-02 コニカミノルタ株式会社 Control device, image forming apparatus, portable terminal device, control method, and control program
EP2797303A1 (en) * 2013-04-26 2014-10-29 Océ-Technologies B.V. Apparatus having a number of options for operating and/or configuring the apparatus and method and computer program product therefor
WO2015184186A1 (en) * 2014-05-30 2015-12-03 Apple Inc. Multi-command single utterance input method
WO2016017978A1 (en) * 2014-07-31 2016-02-04 Samsung Electronics Co., Ltd. Device and method for performing functions
US20160150124A1 (en) 2014-11-24 2016-05-26 Kyocera Document Solutions Inc. Image Forming Apparatus with User Identification Capabilities
US9959129B2 (en) * 2015-01-09 2018-05-01 Microsoft Technology Licensing, Llc Headless task completion within digital personal assistants
JP2016168707A (en) 2015-03-12 2016-09-23 コニカミノルタ株式会社 Image formation device and program
WO2017039648A1 (en) * 2015-09-02 2017-03-09 Hewlett-Packard Development Company, L.P. Voice control of a printer unit
KR20170070649A (en) 2015-12-14 2017-06-22 에스프린팅솔루션 주식회사 Image forming apparuatus, cloud server, image forming system, and method for setting of connection with image forming apparatus
JP6696246B2 (en) * 2016-03-16 2020-05-20 富士ゼロックス株式会社 Image processing device and program
US10388282B2 (en) * 2017-01-25 2019-08-20 CliniCloud Inc. Medical voice command device
CN108656558A (en) * 2017-03-29 2018-10-16 三纬国际立体列印科技股份有限公司 Apply to the speech control system and sound control method of 3D printer
JP7003720B2 (en) * 2018-02-19 2022-01-21 コニカミノルタ株式会社 Image processing equipment, image processing equipment control methods, and programs

Also Published As

Publication number Publication date
US20210240403A1 (en) 2021-08-05
EP3718104A4 (en) 2021-11-10
WO2019231055A1 (en) 2019-12-05
EP3718104A1 (en) 2020-10-07
US11249696B2 (en) 2022-02-15
KR20190136832A (en) 2019-12-10

Similar Documents

Publication Publication Date Title
JP7487807B2 (en) Information processing system, server, information processing method, and information processing program
JP7230506B2 (en) Information processing system, information processing device, information processing method, and information processing program
JP7234612B2 (en) Information processing system and information processing method
JP7471379B2 (en) Voice operation system and voice operation method
JP6942995B2 (en) Information processing program, information processing device, and control method of information processing device
US20220129211A1 (en) Converting voice command into text code blocks that support printing services
US20170054866A1 (en) Mobile apparatus, image scan apparatus and method for processing a job
US11625200B2 (en) Information processing device, mobile terminal, and non-transitory computer readable medium for parameter presentation
US10911611B2 (en) Transmitting fax document by using cloud server
US11190652B2 (en) Electronic apparatus for providing user interface of image forming apparatus
KR20220050640A (en) Providing Shortened URL and Information related Contents corresponding to Original URL
US10440206B2 (en) Fax function diagnosis method, and apparatus for performing the method
CN110312050A (en) Print system, Method of printing, image forming apparatus and its control method and medium
JP2016096393A (en) Information processing apparatus, system, information processing method, and program
US11523024B2 (en) Embedding user specific information into user specific information input area of document
JP2021113899A (en) Information processing system, information processing method, and program
JP2016177387A (en) Information processing apparatus, system, information processing method, and program
JP2021144645A (en) Instruction transmitter, instruction transfer system, instruction transmission method, and program
US11755257B2 (en) Automatic output of document
US11563864B2 (en) Information processing apparatus and non-transitory computer-readable storage medium
JP7388079B2 (en) Information transmitting device, information transmitting method and program
US20220210291A1 (en) Processing of image forming job based on metadata tag input
JP2022167157A (en) Image processing device, image processing method, and program

Legal Events

Date Code Title Description
AS Assignment

Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:HP PRINTING KOREA CO., LTD.;REEL/FRAME:058605/0063

Effective date: 20190620

Owner name: HEWLETT-PACKARD DEVELOPMENT COMPANY, L.P., TEXAS

Free format text: CONFIRMATORY ASSIGNMENT;ASSIGNOR:HP PRINTING KOREA CO., LTD.;REEL/FRAME:058710/0507

Effective date: 20210218

Owner name: HP PRINTING KOREA CO., LTD., KOREA, REPUBLIC OF

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:PARK, SUNG JIN;REEL/FRAME:058604/0904

Effective date: 20190319

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION