Nothing Special   »   [go: up one dir, main page]

US9375841B1 - Robotic devices with ambient indications of joint status - Google Patents

Robotic devices with ambient indications of joint status Download PDF

Info

Publication number
US9375841B1
US9375841B1 US14/288,438 US201414288438A US9375841B1 US 9375841 B1 US9375841 B1 US 9375841B1 US 201414288438 A US201414288438 A US 201414288438A US 9375841 B1 US9375841 B1 US 9375841B1
Authority
US
United States
Prior art keywords
joints
visual feedback
robotic device
robotic
actuators
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US14/288,438
Inventor
Kevin Kemper
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Google LLC
Original Assignee
Google LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Google LLC filed Critical Google LLC
Assigned to GOOGLE INC. reassignment GOOGLE INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KEMPER, Kevin
Priority to US14/288,438 priority Critical patent/US9375841B1/en
Priority to EP15799198.5A priority patent/EP3148754A4/en
Priority to CN202210059416.5A priority patent/CN114367987A/en
Priority to KR1020167032952A priority patent/KR20160146979A/en
Priority to PCT/US2015/025506 priority patent/WO2015183419A1/en
Priority to CN201580026444.5A priority patent/CN107073709A/en
Priority to KR1020187023859A priority patent/KR20180096818A/en
Publication of US9375841B1 publication Critical patent/US9375841B1/en
Application granted granted Critical
Assigned to X DEVELOPMENT LLC reassignment X DEVELOPMENT LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: GOOGLE INC.
Assigned to GOOGLE LLC reassignment GOOGLE LLC CHANGE OF NAME (SEE DOCUMENT FOR DETAILS). Assignors: GOOGLE INC.
Assigned to GOOGLE LLC reassignment GOOGLE LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: X DEVELOPMENT LLC
Assigned to GOOGLE LLC reassignment GOOGLE LLC CORRECTIVE ASSIGNMENT TO CORRECT THE THE REMOVAL OF THE INCORRECTLY RECORDED APPLICATION NUMBERS 14/149802 AND 15/419313 PREVIOUSLY RECORDED AT REEL: 44144 FRAME: 1. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME. Assignors: GOOGLE INC.
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J9/00Programme-controlled manipulators
    • B25J9/16Programme controls
    • B25J9/1674Programme controls characterised by safety, monitoring, diagnostic
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00Controls for manipulators
    • B25J13/08Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B25J13/085Force or torque sensors
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J17/00Joints
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B25HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25JMANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J19/00Accessories fitted to manipulators, e.g. for monitoring, for viewing; Safety devices combined with or specially adapted for use in connection with manipulators

Definitions

  • a robotic device includes a mechanical agent, usually an electro-mechanical machine that is guided by a computer program or electronic circuitry.
  • Robots can be autonomous or semi-autonomous and range from humanoid designs, to large industrial designs with jointed arms and end effectors to perform specialized tasks.
  • interfaces have been designed to enable control of robotic devices, and to receive feedback information from the robotic devices. Such interfaces are often provided remotely from the robotic devices, such as in a control center for an operator, for example. The interfaces may provide numerous amounts of information of the robotic devices, including factors related to health of the robotic device.
  • Robotic devices may also include various indicators to provide notifications to a user.
  • Example indicators include lights to inform that power is on, a battery is charging, a connection is in place with a remote network, or a strength of signal in place with the remote network.
  • robotic devices with ambient indications of joint status are provided.
  • An example robotic device comprises one or more actuators coupled through one or more joints to form a robotic manipulator, and an indicator co-located in proximity to the one or more joints to provide visual feedback on the robotic manipulator at the one or more joints and about the one or more joints.
  • the visual feedback includes information indicative of an operating status of the one or more joints.
  • the robotic device also includes a controller programmable to control the indicator co-located proximal to the one or more joints to provide the visual feedback indicative of the operating status of the one or more joints.
  • a robotic device in another example, comprises one or more actuators coupled through one or more joints to form a robotic manipulator, an indicator co-located in proximity to the one or more joints to provide visual feedback on the robotic manipulator at the one or more joints and about the one or more joints, and a strain gauge for determining a tension experienced by the one or more actuators at the one or more joints.
  • the robotic device also comprises a controller coupled to the strain gauge for receiving information indicating the tension and for controlling the indicator co-located proximal to the one or more joints to provide the visual feedback indicative of the tension experienced.
  • a robotic device comprises a plurality of actuators coupled through a plurality of joints to form a robotic manipulator, a plurality of indicators co-located in proximity to the plurality of joints to provide visual feedback on the robotic manipulator at the plurality of joints and about the plurality of joints, and a controller coupled to the plurality of indicators for receiving information indicating a status of respective joints of the plurality of joints and for controlling the plurality of indicators co-located proximal to the plurality of joints to provide the visual feedback indicative of the status of the respective joints.
  • a system that includes one or more actuators coupled through one or more joints to form a robotic manipulator that includes a means for receiving information indicative of an operating status of the one or more joints including an indication of an amount of torque that the one or more joints is experiencing, and a means for controlling an indicator co-located proximal to the one or more joints to provide visual feedback indicative of the operating status of the one or more joints on the robotic manipulator at the one or more joints and about the one or more joints.
  • a method comprises providing one or more actuators coupled through one or more joints to form a robotic manipulator, receiving information indicative of an operating status of the one or more joints including an indication of an amount of torque that the one or more joints is experiencing, and controlling an indicator co-located proximal to the one or more joints to provide visual feedback indicative of the operating status of the one or more joints on the robotic manipulator at the one or more joints and about the one or more joints.
  • further methods and computer program products including instructions executable by a device or by one or more processors to perform functions of the methods are provided.
  • the methods may be executable for operating a robotic device, for example.
  • the methods may also be executable to control indicators of the robotic device so as to provide visual feedback information about operating characteristics of portions of the robotic device.
  • FIG. 1 illustrates an example robotic device in a form of a robotic manipulator that includes actuators coupled through joints.
  • FIG. 2A illustrates an example faceplate to couple to a joint including a light emitting diode (LED).
  • LED light emitting diode
  • FIG. 2B illustrates another example faceplate to couple to a joint that includes multiple lights.
  • FIG. 2C illustrates another example faceplate to couple to a joint that includes an indicator in a form of a slider bar with markings to show an operating range of the joint.
  • FIG. 2D illustrates another example faceplate to couple to a joint that includes an indicator in a form of another slider bar.
  • FIG. 2E illustrates another example faceplate to couple to a joint that includes a digital readout display.
  • FIG. 2F illustrates another example faceplate to couple to a joint that includes a display.
  • FIG. 3 is a block diagram illustrating an example system for control of a robotic device.
  • FIG. 4 illustrates a schematic drawing of an example computing device.
  • FIG. 5 is a flowchart illustrating an example method for operating a robotic device.
  • a robotic device may include multiple actuators coupled through joints, and the joints may be configured as multiple degree of freedom (DOF) joints.
  • the robotic device may be able to communicate to an interface data regarding operation of the device, such as a load or tension being applied to or experienced by a joint, for example.
  • the interface may not be co-located with the robotic device.
  • indicators may be provided at joints of a robotic device to provide information of the robotic device at the joint.
  • a tri-color LED can be located at each joint to indicate may types of joint-specific information, such as to light up a certain color to show a joint “effort” and operators can easily determine how close the robotic device is to an operating limits.
  • the LED color could be diffused or piped through a cover or through a specific feature to blend the color for overall aesthetics.
  • a tri-color LED may be used to indicate status of the arm.
  • the light may be red to indicate a maximum torque is being applied due to a heavy load, or the light may be green when the arm is operating within the operating limits.
  • Other types of indicating can be used such as blinking lights, light patterns, color patterns, etc., all or any of which may be indicative of a certain status of the joint.
  • the LEDs are co-located at joints to provide feedback on the robotic at the joint and about the joint itself.
  • an operator in view of the robotic device can infer a status of the joint based on the indicators at the joint.
  • FIG. 1 illustrates an example robotic device 100 in a form of a robotic manipulator that includes actuators 102 , 104 , and 106 coupled through joints 108 and 110 .
  • the robotic device 100 further includes an end-of-arm tool (EOAT) 112 coupled to the actuator 102 through a joint 114 .
  • EOAT end-of-arm tool
  • the actuators 102 , 104 , and 106 may include a torque controlled actuator that has an electric motor attached to a transmission, and the transmission may include an N-stage belt transmission arranged in a serial fashion that connects rotation of the motor to rotation of an output.
  • Each stage of the transmission may include a timing belt, a belt tensioning mechanism, and a tension measuring system, for example.
  • Other types of actuators may also be used as well, and any number of actuators may be coupled together to form the robotic device 100 .
  • the robotic device 100 further includes indicators 116 , 118 , and 120 co-located in proximity to the joints 110 , 108 , and 114 , respectively to provide visual feedback on the robotic device 100 at the joints 110 , 108 , and 114 and about the joints 110 , 108 , and 114 .
  • the indicators 116 , 118 , and 120 are provided on an exterior surface of the joints 110 , 108 , and 114 .
  • the indicators 116 , 118 , and 120 may be on an interior surface of the joints 110 , 108 , and 114 , and diffused or piped through a cover or through a specific feature to blend colors for overall aesthetics.
  • the joints 110 , 108 , and 114 may include respective interface elements that connect the actuators 102 , 104 , 106 , and the EOAT 112 , and the indicators 116 , 118 , and 120 may be provided on the interface elements so as to be located at or near the joints 110 , 108 , and 114 .
  • the indicators 116 , 118 , and 120 may be or include any number or type of indicators, such as lights, light emitting diodes (LEDs), displays, digital read-outs, or other types of electrical indicators.
  • the indicators 116 , 118 , and 120 may further includes multiple indicators, such as multiple LEDs as shown in the indicator 116 , or single indicators as shown by the indicators 118 and 120 .
  • the robotic device 100 may further include a controller (not shown) programmable to control the indicators 116 , 118 , and 120 , respectively that are co-located proximal to the joints 110 , 108 , and 114 to provide the visual feedback indicative of the operating status of the joints 110 , 108 , and 114 .
  • the controller may be electronically connected or in communication with the indicators 116 , 118 , and 120 to control operation of the indicators 116 , 118 , and 120 .
  • the visual feedback includes information indicative of an operating status of the joints 110 , 108 , and 114 including an indication of an amount of torque that the respective joints 110 , 108 , and 114 are experiencing.
  • the visual feedback may include or indicate other information as well, such as an operating status of the joints 110 , 108 , and 114 or an operating status of the actuators 102 , 104 , 106 , and the EOAT 112 .
  • An operating status may include normal, medium, or overload, or may provide more detailed information such as related to wear and tear of the joint (based on an age of the joint, an amount of movement of the joint over time, etc.), information related to metal fatigue (e.g., derived from performance measurements of the joint due to speed of rotation), or other information related to any type of operating characteristic.
  • the visual feedback information may also indicate whether any of the joints 110 , 108 , and 114 is experiencing a given amount of torque outside or within operating limits, or above or below a given threshold amount.
  • the robotic device 100 in FIG. 1 is in a form of a robotic manipulator, such as an arm of a robot.
  • the robotic device 100 may take other forms, and any combination of actuators coupled together through joints may be provided that include one or more indicators at the joints to provide the visual feedback information.
  • FIGS. 2A-2F illustrate example indicators and indicator displays.
  • FIG. 2A illustrates a faceplate 202 to couple to a joint including a light emitting diode (LED) 204 .
  • the LED 204 may be a multi-color, and may be configured to light at a given color to indicate whether a joint is experiencing a given amount of torque outside or within operating limits.
  • the LED 204 may light red to indicate a torque outside of an operating limit, green to indicate a torque within operating limits, and yellow to indicate torques at about the limit.
  • the LED 204 may be configured to light at an intensity level based on the amount of torque that the one or more joints is experiencing, such that the LED 204 is brighter for higher levels of torque.
  • the LED 204 may provide the visual feedback using blinking patterns, light patterns, or color patterns that map to a level of torque being applied or experienced by the joint.
  • FIG. 2B illustrates another example faceplate 206 to couple to a joint that includes multiple lights 208 , 210 , and 212 .
  • the lights 208 , 210 , and 212 may be LEDs or other lights, and the faceplate 206 may include textual indications of “LOW”, “MEDIUM”, and “HIGH” as well for indication of an amount of torque applied by or experienced by the joint due to which of the lights 208 , 210 , and 212 is lit. Any of the lights 208 , 210 , and 212 may also be multi-color lights as well.
  • FIG. 2C illustrates another example faceplate 214 to couple to a joint that includes an indicator in a form of a slider bar 216 with markings to show an operating range of the joint.
  • the operating range may refer to any number of operating characteristics including torque or tension, for example.
  • the faceplate 214 also includes textual indications of “LOW” and “HIGH”, and the slider bar 216 may be digital and light up along the slider bar 216 to illustrate an operating status of the joint.
  • FIG. 2D illustrates another example faceplate 218 to couple to a joint that includes an indicator in a form of another slider bar 220 , and the faceplate 218 includes textual indications of “LOW”, “MEDIUM”, and “HIGH”.
  • the slider bar 220 may be digital and light up along the slide bar 220 to illustrate an operating status of the joint.
  • FIG. 2E illustrates another example faceplate 222 to couple to a joint that includes a digital readout display 224 .
  • the digital readout display 224 may be programmed to indicate any number of information related to operating characteristics of the joint, such as an amount of torque being applied by or experienced by the joint. Additional or alternative data may be displayed as well for operating status, such as an amount of time that the joint experiences torque outside of operating limits, a health status indication of the joint, an identification of the joint, an age of the joint, information indicating mileage of the joint (or amount of usage of the joint), or still other information.
  • FIG. 2F illustrates another example faceplate 226 to couple to a joint that includes a display 228 .
  • the display 228 may be programmed to display any number or type of information related to operating characteristics of the joint, such a display of an arm of the robotic device, and an illustration showing any joint that may be experiencing an amount of torque outside of an operating limit (e.g., an overload on a joint). Additional or alternative data may be displayed as well to visually indicate an operating status of the joint, and at the joint. Any types of information regarding characteristics of the joint, how the joint is currently operating, or how the joint has operated may be provided as well.
  • any of the example indicators and indicator displays in FIGS. 2A-2E may be provided co-located at a joint of a robotic device to illustrate operating status of the joint, and at the joint.
  • any of the example indicators and indicator displays in FIGS. 2A-2E may be provided in proximity to joints, such as on or around a joint, or on a surface of an actuator that couples to the joint and is adjacent to the joint, for example.
  • the indicators are providing information of the joint, and the indicators are located at or near the joint so as to provide the information at an area of the robotic device that the information pertains to.
  • FIG. 3 is a block diagram illustrating an example system for control of a robotic device.
  • the system includes a base link comprising a single actuator 302 , coupled to a trunk link comprising side-by-side actuators 304 and 306 coupled to a joint 308 .
  • the joint 308 includes a differential 310 , a strain gauge 312 , and an indicator 314 .
  • the strain gauge 312 determines a tension experienced by the actuators 304 and 306 at the joint 308 , for example.
  • the differential 308 connects to a bicep link that comprises side-by-side actuators 316 and 318 coupled to a joint 320 .
  • the joint 320 includes a differential 322 , a strain gauge 324 , and an indicator 326 .
  • the strain gauge 324 determines a tension experienced by the actuators 316 and 318 at the joint 320 , for example.
  • the differential 322 connect to a forearm link that comprises actuators 328 and 330 coupled to a joint 332 .
  • the joint includes a differential 334 , a strain gauge 336 , and an indicator 338 .
  • the strain gauge 336 determines a tension experienced by the actuators 328 and 330 at the joint 332 , for example.
  • the differential 334 outputs to end-of-arm-tooling (EOAT) 340 .
  • EOAT end-of-arm-tooling
  • the base link, trunk link, bicep link, and forearm link may be modular links coupled to form a seven degree-of-freedom (DOF) robotic arm.
  • DOF degree-of-freedom
  • the system may also include a controller 342 coupled to the base link, the trunk link, the bicep link, and the forearm link to handle control at a deterministic rate and command desired torques to actuators.
  • the controller 342 may include torque controllers 344 and impedance controllers 346 to determine motor currents for the actuators of the links based on output torques in a feedback control loop.
  • Other hybrid force-position controllers and position controllers as would be suitable for a robotic arm may be used as well.
  • a dynamic model 348 may also be used based on measured link inertias, accelerations, and velocities, to compute expected instantaneous torques during a known task.
  • a contact detector 350 may compare such expectations against measured torques provided by sensors (e.g., strain gauges 312 , 324 , and 336 ), and a difference above a specified threshold can be flagged as unexpected contact with a person, object, or environment.
  • a safety reflex 352 may adapt control policy to react appropriately to the contact. Reactions may include reversal of torques, locking of motor brakes, changing to a low impedance gravity compensation mode, or modification of the arm trajectory, for example.
  • the controller 342 may further be programmable to control the indicators 314 , 326 , and 338 that are co-located proximal to the joints 308 , 320 , and 332 to provide visual feedback indicative of the operating status of the joints 308 , 320 , and 332 .
  • the controller 342 may receive information indicating a status of respective joints, and may be coupled to the indicators 314 , 326 , and 338 to control the indicators 314 , 326 , and 338 co-located proximal to the joints 308 , 320 , and 332 to provide the visual feedback indicative of the status of the respective joints.
  • the controller 342 may also be coupled to the strain gauges 312 , 324 , and 336 for receiving information indicating a tension applied by or experienced by the respective actuators (or information from which a tension applied by or experienced by the respective actuators may be derived), and may control the indicators 314 , 326 , and 338 to provide the visual feedback indicative of the tension experienced.
  • the controller 342 may cause the indicator 314 to illuminate red when the joint 308 is experiencing a torque outside of an operating limit, and in such a scenario, the joints 320 and 332 may be at or outside of operating limits as well causing the indicators 326 and 338 to be illuminated yellow (e.g., at or near a limit) or red (e.g., outside the limit).
  • visual feedback can be provided that includes respective information indicative of whether any of the joints 308 , 320 , and 332 is experiencing a given amount of torque outside, within, or near operating limits based on a respective indicator at a respective joint being activated.
  • the indicators 314 , 326 , and 338 may include LEDs for lighting at a given color to indicate whether respective joints are experiencing a given amount of torque outside or within operating limits of the respective joints.
  • the indicators 314 , 326 , and 338 may not be illuminated when the robotic device is not being operated (or is off), or the indicators 314 , 326 , and 338 may also not be illuminated when the operating status of the joints 308 , 320 , and 332 is acceptable or within operating limits.
  • the controller 342 may be programmed to illuminate the indicators at times at which the joints 308 , 320 , and 332 are experiencing strain above threshold limits, for example.
  • FIG. 3 illustrates one controller 342
  • multiple controllers may be provided for each joint to couple to the strain gauges 312 , 324 , and 336 for receiving information indicating respective tensions and for controlling the indicators 314 , 326 , and 338 co-located proximal to the joints to provide the visual feedback indicative of the respective tensions experienced.
  • the controller 342 may take the form of a computing device, such as illustrated in FIG. 4 , which illustrates a schematic drawing of an example computing device 400 .
  • FIG. 4 illustrates a schematic drawing of an example computing device 400 .
  • some components illustrated in FIG. 4 may be distributed across multiple computing devices. However, for the sake of example, the components are shown and described as part of one example device 400 .
  • the device 400 may be or include a mobile device, desktop computer, tablet computer, or similar device that may be configured to perform the functions described herein.
  • the device 400 may include an interface 402 , sensor(s) 404 , data storage 406 , and a processor 408 . Components illustrated in FIG. 4 may be linked together by a communication link 410 .
  • the communication link 410 is illustrated as a wired connection; however, wireless connections may also be used.
  • the device 400 may also include hardware to enable communication within the device 400 and between the device 400 and other computing devices (not shown), such as a server entity.
  • the hardware may include transmitters, receivers, and antennas, for example.
  • the interface 402 may be configured to allow the device 400 to communicate with another computing device (not shown), such as a server.
  • the interface 402 may be configured to receive input data from one or more computing devices, and may also be configured to send output data to the one or more computing devices.
  • the interface 402 may also be configured to receive input from and provide output to an actuator, a modular link of a robot arm, or indicators of a joint of the arm, for example.
  • the interface 402 may include a receiver and transmitter to receive and send data, or may be hard-wired to components to send and receive data.
  • the interface 402 may also include a user-interface, such as a keyboard, microphone, touchscreen, etc., to receive inputs as well.
  • the sensor 404 may include one or more sensors, or may represent one or more sensors included within the device 400 .
  • Example sensors include an accelerometer, gyroscope, pedometer, light sensors, microphone, camera, contact rollers, load cells, strain gauges or other context-aware sensors that may collect data of the actuators or joints and provide the data to the data storage 406 or processor 408 .
  • the processor 408 may be configured to receive data from the interface 402 , the sensor 404 , and the data storage 406 .
  • the data storage 406 may store program logic 412 that can be accessed and executed by the processor 408 to perform functions executable to determine instructions for operation of actuators or indicators on joints, for example.
  • Example functions include determination of tension experienced by the actuators or output torque of the system, and control of indicators to light or provide visual feedback information at the joints and about operating characteristics of the joints. Any functions described herein, or other example functions for the robotic device may be performed by the device 400 or one or more processors 408 of the device via execution of instructions stored on the data storage 406 or otherwise received.
  • the device 400 is illustrated to include an additional processor 414 .
  • the processor 414 may be configured to control other aspects of the device 400 including displays or outputs of the device 400 (e.g., the processor 414 may be a GPU).
  • Example methods described herein may be performed individually by components of the device 400 , or in combination by one or all of the components of the device 400 .
  • portions of the device 400 may process data and provide an output internally in the device 400 to the processor 414 , for example.
  • portions of the device 400 may process data and provide outputs externally to other computing devices.
  • a robotic device can include multiple actuators coupled through joints to form a robotic manipulator, and indicators are co-located in proximity to the joints to provide visual feedback on the robotic manipulator at the joints and about the one or more joints.
  • the actuators may be coupled through joints to form a multiple degree-of-freedom (DOF) robotic manipulator, and the indicators can be independently operated to provide visual feedback on the robotic manipulator at the joints and about the joints, respectively.
  • DOF degree-of-freedom
  • operations may be described as methods for performing functions, and methods may be embodied on a computer program product (e.g., a tangible computer readable storage medium or non-transitory computer readable medium) that includes instructions executable to perform the functions.
  • a computer program product e.g., a tangible computer readable storage medium or non-transitory computer readable medium
  • FIG. 5 is a flowchart illustrating an example method 500 for operating a robotic device.
  • the method 500 may be embodied as computer executable instructions stored on non-transitory media, for example.
  • the method 500 includes providing one or more actuators coupled through one or more joints to form a robotic manipulator.
  • the method 500 includes receiving information indicative of an operating status of the one or more joints including an indication of an amount of torque that the one or more joints is experiencing.
  • the method 500 includes controlling an indicator co-located proximal to the one or more joints to provide visual feedback indicative of the operating status of the one or more joints on the robotic manipulator at the one or more joints and about the one or more joints.
  • the robotic device 100 described in FIG. 1 or the system described in FIG. 3 above may be used in many implementations.
  • Example implementations include within a modular robot link or actuator system.

Landscapes

  • Engineering & Computer Science (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • Manipulator (AREA)

Abstract

In examples, robotic devices with ambient indications of joint status are provided. An example robotic device includes one or more actuators coupled through one or more joints to form a robotic manipulator, and an indicator co-located in proximity to the one or more joints to provide visual feedback on the robotic manipulator at the one or more joints and about the one or more joints. The visual feedback may include information indicative of an operating status of the one or more joints including an indication of an amount of torque that the one or more joints is experiencing. The robotic device also includes a controller programmable to control the indicator co-located proximal to the one or more joints to provide the visual feedback indicative of the operating status of the one or more joints.

Description

BACKGROUND
Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.
A robotic device includes a mechanical agent, usually an electro-mechanical machine that is guided by a computer program or electronic circuitry. Robots can be autonomous or semi-autonomous and range from humanoid designs, to large industrial designs with jointed arms and end effectors to perform specialized tasks.
Many interfaces have been designed to enable control of robotic devices, and to receive feedback information from the robotic devices. Such interfaces are often provided remotely from the robotic devices, such as in a control center for an operator, for example. The interfaces may provide numerous amounts of information of the robotic devices, including factors related to health of the robotic device.
Robotic devices may also include various indicators to provide notifications to a user. Example indicators include lights to inform that power is on, a battery is charging, a connection is in place with a remote network, or a strength of signal in place with the remote network.
SUMMARY
In examples, robotic devices with ambient indications of joint status are provided. An example robotic device comprises one or more actuators coupled through one or more joints to form a robotic manipulator, and an indicator co-located in proximity to the one or more joints to provide visual feedback on the robotic manipulator at the one or more joints and about the one or more joints. The visual feedback includes information indicative of an operating status of the one or more joints. The robotic device also includes a controller programmable to control the indicator co-located proximal to the one or more joints to provide the visual feedback indicative of the operating status of the one or more joints.
In another example, a robotic device is provided that comprises one or more actuators coupled through one or more joints to form a robotic manipulator, an indicator co-located in proximity to the one or more joints to provide visual feedback on the robotic manipulator at the one or more joints and about the one or more joints, and a strain gauge for determining a tension experienced by the one or more actuators at the one or more joints. The robotic device also comprises a controller coupled to the strain gauge for receiving information indicating the tension and for controlling the indicator co-located proximal to the one or more joints to provide the visual feedback indicative of the tension experienced.
In still another example, a robotic device is provided that comprises a plurality of actuators coupled through a plurality of joints to form a robotic manipulator, a plurality of indicators co-located in proximity to the plurality of joints to provide visual feedback on the robotic manipulator at the plurality of joints and about the plurality of joints, and a controller coupled to the plurality of indicators for receiving information indicating a status of respective joints of the plurality of joints and for controlling the plurality of indicators co-located proximal to the plurality of joints to provide the visual feedback indicative of the status of the respective joints.
In another example, a system that includes one or more actuators coupled through one or more joints to form a robotic manipulator is provided that includes a means for receiving information indicative of an operating status of the one or more joints including an indication of an amount of torque that the one or more joints is experiencing, and a means for controlling an indicator co-located proximal to the one or more joints to provide visual feedback indicative of the operating status of the one or more joints on the robotic manipulator at the one or more joints and about the one or more joints.
In yet another example, a method is provided that comprises providing one or more actuators coupled through one or more joints to form a robotic manipulator, receiving information indicative of an operating status of the one or more joints including an indication of an amount of torque that the one or more joints is experiencing, and controlling an indicator co-located proximal to the one or more joints to provide visual feedback indicative of the operating status of the one or more joints on the robotic manipulator at the one or more joints and about the one or more joints.
In still other examples, further methods and computer program products including instructions executable by a device or by one or more processors to perform functions of the methods are provided. The methods may be executable for operating a robotic device, for example. The methods may also be executable to control indicators of the robotic device so as to provide visual feedback information about operating characteristics of portions of the robotic device.
These as well as other aspects, advantages, and alternatives, will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying figures.
BRIEF DESCRIPTION OF THE FIGURES
FIG. 1 illustrates an example robotic device in a form of a robotic manipulator that includes actuators coupled through joints.
FIG. 2A illustrates an example faceplate to couple to a joint including a light emitting diode (LED).
FIG. 2B illustrates another example faceplate to couple to a joint that includes multiple lights.
FIG. 2C illustrates another example faceplate to couple to a joint that includes an indicator in a form of a slider bar with markings to show an operating range of the joint.
FIG. 2D illustrates another example faceplate to couple to a joint that includes an indicator in a form of another slider bar.
FIG. 2E illustrates another example faceplate to couple to a joint that includes a digital readout display.
FIG. 2F illustrates another example faceplate to couple to a joint that includes a display.
FIG. 3 is a block diagram illustrating an example system for control of a robotic device.
FIG. 4 illustrates a schematic drawing of an example computing device.
FIG. 5 is a flowchart illustrating an example method for operating a robotic device.
DETAILED DESCRIPTION
The following detailed description describes various features and functions of the disclosed systems and methods with reference to the accompanying figures. In the figures, similar symbols identify similar components, unless context dictates otherwise. The illustrative system and method embodiments described herein are not meant to be limiting. It may be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a wide variety of different configurations, all of which are contemplated herein.
Within examples, a robotic device may include multiple actuators coupled through joints, and the joints may be configured as multiple degree of freedom (DOF) joints. The robotic device may be able to communicate to an interface data regarding operation of the device, such as a load or tension being applied to or experienced by a joint, for example. However, the interface may not be co-located with the robotic device.
Within examples, indicators may be provided at joints of a robotic device to provide information of the robotic device at the joint. As an example, a tri-color LED can be located at each joint to indicate may types of joint-specific information, such as to light up a certain color to show a joint “effort” and operators can easily determine how close the robotic device is to an operating limits. The LED color could be diffused or piped through a cover or through a specific feature to blend the color for overall aesthetics.
As a specific example, for an industrial robotic arm, a tri-color LED may be used to indicate status of the arm. With the arm, the light may be red to indicate a maximum torque is being applied due to a heavy load, or the light may be green when the arm is operating within the operating limits. Other types of indicating can be used such as blinking lights, light patterns, color patterns, etc., all or any of which may be indicative of a certain status of the joint.
The LEDs are co-located at joints to provide feedback on the robotic at the joint and about the joint itself. Thus, an operator in view of the robotic device can infer a status of the joint based on the indicators at the joint.
Referring now to the figures, FIG. 1 illustrates an example robotic device 100 in a form of a robotic manipulator that includes actuators 102, 104, and 106 coupled through joints 108 and 110. The robotic device 100 further includes an end-of-arm tool (EOAT) 112 coupled to the actuator 102 through a joint 114.
Within examples, the actuators 102, 104, and 106 may include a torque controlled actuator that has an electric motor attached to a transmission, and the transmission may include an N-stage belt transmission arranged in a serial fashion that connects rotation of the motor to rotation of an output. Each stage of the transmission may include a timing belt, a belt tensioning mechanism, and a tension measuring system, for example. Other types of actuators may also be used as well, and any number of actuators may be coupled together to form the robotic device 100.
The robotic device 100 further includes indicators 116, 118, and 120 co-located in proximity to the joints 110, 108, and 114, respectively to provide visual feedback on the robotic device 100 at the joints 110, 108, and 114 and about the joints 110, 108, and 114. The indicators 116, 118, and 120 are provided on an exterior surface of the joints 110, 108, and 114. In other examples, the indicators 116, 118, and 120 may be on an interior surface of the joints 110, 108, and 114, and diffused or piped through a cover or through a specific feature to blend colors for overall aesthetics. In still further examples, the joints 110, 108, and 114 may include respective interface elements that connect the actuators 102, 104, 106, and the EOAT 112, and the indicators 116, 118, and 120 may be provided on the interface elements so as to be located at or near the joints 110, 108, and 114.
The indicators 116, 118, and 120 may be or include any number or type of indicators, such as lights, light emitting diodes (LEDs), displays, digital read-outs, or other types of electrical indicators. The indicators 116, 118, and 120 may further includes multiple indicators, such as multiple LEDs as shown in the indicator 116, or single indicators as shown by the indicators 118 and 120.
The robotic device 100 may further include a controller (not shown) programmable to control the indicators 116, 118, and 120, respectively that are co-located proximal to the joints 110, 108, and 114 to provide the visual feedback indicative of the operating status of the joints 110, 108, and 114. Thus, the controller may be electronically connected or in communication with the indicators 116, 118, and 120 to control operation of the indicators 116, 118, and 120.
In one example, the visual feedback includes information indicative of an operating status of the joints 110, 108, and 114 including an indication of an amount of torque that the respective joints 110, 108, and 114 are experiencing. The visual feedback may include or indicate other information as well, such as an operating status of the joints 110, 108, and 114 or an operating status of the actuators 102, 104, 106, and the EOAT 112. An operating status may include normal, medium, or overload, or may provide more detailed information such as related to wear and tear of the joint (based on an age of the joint, an amount of movement of the joint over time, etc.), information related to metal fatigue (e.g., derived from performance measurements of the joint due to speed of rotation), or other information related to any type of operating characteristic. The visual feedback information may also indicate whether any of the joints 110, 108, and 114 is experiencing a given amount of torque outside or within operating limits, or above or below a given threshold amount.
The robotic device 100 in FIG. 1 is in a form of a robotic manipulator, such as an arm of a robot. However, the robotic device 100 may take other forms, and any combination of actuators coupled together through joints may be provided that include one or more indicators at the joints to provide the visual feedback information.
FIGS. 2A-2F illustrate example indicators and indicator displays. In one example, FIG. 2A illustrates a faceplate 202 to couple to a joint including a light emitting diode (LED) 204. The LED 204 may be a multi-color, and may be configured to light at a given color to indicate whether a joint is experiencing a given amount of torque outside or within operating limits. As an example, the LED 204 may light red to indicate a torque outside of an operating limit, green to indicate a torque within operating limits, and yellow to indicate torques at about the limit. In further examples, the LED 204 may be configured to light at an intensity level based on the amount of torque that the one or more joints is experiencing, such that the LED 204 is brighter for higher levels of torque. Still further, in other examples, the LED 204 may provide the visual feedback using blinking patterns, light patterns, or color patterns that map to a level of torque being applied or experienced by the joint.
FIG. 2B illustrates another example faceplate 206 to couple to a joint that includes multiple lights 208, 210, and 212. The lights 208, 210, and 212 may be LEDs or other lights, and the faceplate 206 may include textual indications of “LOW”, “MEDIUM”, and “HIGH” as well for indication of an amount of torque applied by or experienced by the joint due to which of the lights 208, 210, and 212 is lit. Any of the lights 208, 210, and 212 may also be multi-color lights as well.
FIG. 2C illustrates another example faceplate 214 to couple to a joint that includes an indicator in a form of a slider bar 216 with markings to show an operating range of the joint. The operating range may refer to any number of operating characteristics including torque or tension, for example. The faceplate 214 also includes textual indications of “LOW” and “HIGH”, and the slider bar 216 may be digital and light up along the slider bar 216 to illustrate an operating status of the joint.
FIG. 2D illustrates another example faceplate 218 to couple to a joint that includes an indicator in a form of another slider bar 220, and the faceplate 218 includes textual indications of “LOW”, “MEDIUM”, and “HIGH”. The slider bar 220 may be digital and light up along the slide bar 220 to illustrate an operating status of the joint.
FIG. 2E illustrates another example faceplate 222 to couple to a joint that includes a digital readout display 224. The digital readout display 224 may be programmed to indicate any number of information related to operating characteristics of the joint, such as an amount of torque being applied by or experienced by the joint. Additional or alternative data may be displayed as well for operating status, such as an amount of time that the joint experiences torque outside of operating limits, a health status indication of the joint, an identification of the joint, an age of the joint, information indicating mileage of the joint (or amount of usage of the joint), or still other information.
FIG. 2F illustrates another example faceplate 226 to couple to a joint that includes a display 228. The display 228 may be programmed to display any number or type of information related to operating characteristics of the joint, such a display of an arm of the robotic device, and an illustration showing any joint that may be experiencing an amount of torque outside of an operating limit (e.g., an overload on a joint). Additional or alternative data may be displayed as well to visually indicate an operating status of the joint, and at the joint. Any types of information regarding characteristics of the joint, how the joint is currently operating, or how the joint has operated may be provided as well.
Within examples, any of the example indicators and indicator displays in FIGS. 2A-2E may be provided co-located at a joint of a robotic device to illustrate operating status of the joint, and at the joint. In further examples, any of the example indicators and indicator displays in FIGS. 2A-2E may be provided in proximity to joints, such as on or around a joint, or on a surface of an actuator that couples to the joint and is adjacent to the joint, for example. Within such examples, the indicators are providing information of the joint, and the indicators are located at or near the joint so as to provide the information at an area of the robotic device that the information pertains to.
FIG. 3 is a block diagram illustrating an example system for control of a robotic device. The system includes a base link comprising a single actuator 302, coupled to a trunk link comprising side-by- side actuators 304 and 306 coupled to a joint 308. The joint 308 includes a differential 310, a strain gauge 312, and an indicator 314. The strain gauge 312 determines a tension experienced by the actuators 304 and 306 at the joint 308, for example.
The differential 308 connects to a bicep link that comprises side-by- side actuators 316 and 318 coupled to a joint 320. The joint 320 includes a differential 322, a strain gauge 324, and an indicator 326. The strain gauge 324 determines a tension experienced by the actuators 316 and 318 at the joint 320, for example.
The differential 322 connect to a forearm link that comprises actuators 328 and 330 coupled to a joint 332. The joint includes a differential 334, a strain gauge 336, and an indicator 338. The strain gauge 336 determines a tension experienced by the actuators 328 and 330 at the joint 332, for example.
The differential 334 outputs to end-of-arm-tooling (EOAT) 340. Thus, the base link, trunk link, bicep link, and forearm link may be modular links coupled to form a seven degree-of-freedom (DOF) robotic arm.
The system may also include a controller 342 coupled to the base link, the trunk link, the bicep link, and the forearm link to handle control at a deterministic rate and command desired torques to actuators. The controller 342 may include torque controllers 344 and impedance controllers 346 to determine motor currents for the actuators of the links based on output torques in a feedback control loop. Other hybrid force-position controllers and position controllers as would be suitable for a robotic arm may be used as well. A dynamic model 348 may also be used based on measured link inertias, accelerations, and velocities, to compute expected instantaneous torques during a known task. A contact detector 350 may compare such expectations against measured torques provided by sensors (e.g., strain gauges 312, 324, and 336), and a difference above a specified threshold can be flagged as unexpected contact with a person, object, or environment. A safety reflex 352 may adapt control policy to react appropriately to the contact. Reactions may include reversal of torques, locking of motor brakes, changing to a low impedance gravity compensation mode, or modification of the arm trajectory, for example.
The controller 342 may further be programmable to control the indicators 314, 326, and 338 that are co-located proximal to the joints 308, 320, and 332 to provide visual feedback indicative of the operating status of the joints 308, 320, and 332. As an example, the controller 342 may receive information indicating a status of respective joints, and may be coupled to the indicators 314, 326, and 338 to control the indicators 314, 326, and 338 co-located proximal to the joints 308, 320, and 332 to provide the visual feedback indicative of the status of the respective joints. In one example, the controller 342 may also be coupled to the strain gauges 312, 324, and 336 for receiving information indicating a tension applied by or experienced by the respective actuators (or information from which a tension applied by or experienced by the respective actuators may be derived), and may control the indicators 314, 326, and 338 to provide the visual feedback indicative of the tension experienced.
As a specific example, the controller 342 may cause the indicator 314 to illuminate red when the joint 308 is experiencing a torque outside of an operating limit, and in such a scenario, the joints 320 and 332 may be at or outside of operating limits as well causing the indicators 326 and 338 to be illuminated yellow (e.g., at or near a limit) or red (e.g., outside the limit). Thus, visual feedback can be provided that includes respective information indicative of whether any of the joints 308, 320, and 332 is experiencing a given amount of torque outside, within, or near operating limits based on a respective indicator at a respective joint being activated. The indicators 314, 326, and 338 may include LEDs for lighting at a given color to indicate whether respective joints are experiencing a given amount of torque outside or within operating limits of the respective joints. The indicators 314, 326, and 338 may not be illuminated when the robotic device is not being operated (or is off), or the indicators 314, 326, and 338 may also not be illuminated when the operating status of the joints 308, 320, and 332 is acceptable or within operating limits. The controller 342 may be programmed to illuminate the indicators at times at which the joints 308, 320, and 332 are experiencing strain above threshold limits, for example.
Although FIG. 3 illustrates one controller 342, multiple controllers may be provided for each joint to couple to the strain gauges 312, 324, and 336 for receiving information indicating respective tensions and for controlling the indicators 314, 326, and 338 co-located proximal to the joints to provide the visual feedback indicative of the respective tensions experienced.
The controller 342 may take the form of a computing device, such as illustrated in FIG. 4, which illustrates a schematic drawing of an example computing device 400. In some examples, some components illustrated in FIG. 4 may be distributed across multiple computing devices. However, for the sake of example, the components are shown and described as part of one example device 400. The device 400 may be or include a mobile device, desktop computer, tablet computer, or similar device that may be configured to perform the functions described herein.
The device 400 may include an interface 402, sensor(s) 404, data storage 406, and a processor 408. Components illustrated in FIG. 4 may be linked together by a communication link 410. The communication link 410 is illustrated as a wired connection; however, wireless connections may also be used. The device 400 may also include hardware to enable communication within the device 400 and between the device 400 and other computing devices (not shown), such as a server entity. The hardware may include transmitters, receivers, and antennas, for example.
The interface 402 may be configured to allow the device 400 to communicate with another computing device (not shown), such as a server. Thus, the interface 402 may be configured to receive input data from one or more computing devices, and may also be configured to send output data to the one or more computing devices. The interface 402 may also be configured to receive input from and provide output to an actuator, a modular link of a robot arm, or indicators of a joint of the arm, for example. The interface 402 may include a receiver and transmitter to receive and send data, or may be hard-wired to components to send and receive data. In other examples, the interface 402 may also include a user-interface, such as a keyboard, microphone, touchscreen, etc., to receive inputs as well.
The sensor 404 may include one or more sensors, or may represent one or more sensors included within the device 400. Example sensors include an accelerometer, gyroscope, pedometer, light sensors, microphone, camera, contact rollers, load cells, strain gauges or other context-aware sensors that may collect data of the actuators or joints and provide the data to the data storage 406 or processor 408.
The processor 408 may be configured to receive data from the interface 402, the sensor 404, and the data storage 406. The data storage 406 may store program logic 412 that can be accessed and executed by the processor 408 to perform functions executable to determine instructions for operation of actuators or indicators on joints, for example. Example functions include determination of tension experienced by the actuators or output torque of the system, and control of indicators to light or provide visual feedback information at the joints and about operating characteristics of the joints. Any functions described herein, or other example functions for the robotic device may be performed by the device 400 or one or more processors 408 of the device via execution of instructions stored on the data storage 406 or otherwise received.
The device 400 is illustrated to include an additional processor 414. The processor 414 may be configured to control other aspects of the device 400 including displays or outputs of the device 400 (e.g., the processor 414 may be a GPU). Example methods described herein may be performed individually by components of the device 400, or in combination by one or all of the components of the device 400. In one instance, portions of the device 400 may process data and provide an output internally in the device 400 to the processor 414, for example. In other instances, portions of the device 400 may process data and provide outputs externally to other computing devices.
Thus, within examples described herein, a robotic device can include multiple actuators coupled through joints to form a robotic manipulator, and indicators are co-located in proximity to the joints to provide visual feedback on the robotic manipulator at the joints and about the one or more joints. As described, the actuators may be coupled through joints to form a multiple degree-of-freedom (DOF) robotic manipulator, and the indicators can be independently operated to provide visual feedback on the robotic manipulator at the joints and about the joints, respectively.
Within some examples herein, operations may be described as methods for performing functions, and methods may be embodied on a computer program product (e.g., a tangible computer readable storage medium or non-transitory computer readable medium) that includes instructions executable to perform the functions.
FIG. 5 is a flowchart illustrating an example method 500 for operating a robotic device. The method 500 may be embodied as computer executable instructions stored on non-transitory media, for example. At block 502, the method 500 includes providing one or more actuators coupled through one or more joints to form a robotic manipulator. At block 504, the method 500 includes receiving information indicative of an operating status of the one or more joints including an indication of an amount of torque that the one or more joints is experiencing. At block 506, the method 500 includes controlling an indicator co-located proximal to the one or more joints to provide visual feedback indicative of the operating status of the one or more joints on the robotic manipulator at the one or more joints and about the one or more joints.
The robotic device 100 described in FIG. 1 or the system described in FIG. 3 above may be used in many implementations. Example implementations include within a modular robot link or actuator system.
It should be understood that arrangements described herein are for purposes of example only. As such, those skilled in the art will appreciate that other arrangements and other elements (e.g. machines, interfaces, functions, orders, and groupings of functions, etc.) can be used instead, and some elements may be omitted altogether according to the desired results. Further, many of the elements that are described are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, in any suitable combination and location, or other structural elements described as independent structures may be combined.
While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope being indicated by the following claims, along with the full scope of equivalents to which such claims are entitled. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting.

Claims (18)

What is claimed is:
1. A robotic device comprising:
a robotic manipulator comprising a plurality of links connected by one or more joints, wherein the plurality of links include one or more actuators coupled through the one or more joints;
an indicator located on a surface of the one or more joints to provide visual feedback on the robotic manipulator at a location on the one or more joints and the visual feedback is about the one or more joints, wherein the visual feedback includes information indicative of an operating status of the one or more joints, and wherein the operating status includes indicating an amount of torque that the one or more joints is experiencing;
a sensor positioned at the one or more joints for determining a tension experienced by the one or more actuators at the one or more joints; and
a controller programmable to receive information indicating the tension and to control the indicator co-located proximal to the one or more joints to provide the visual feedback indicative of the amount of torque that the one or more joints is experiencing based on the tension experienced.
2. The robotic device of claim 1, wherein the visual feedback includes information indicative of whether the one or more joints is experiencing a given amount of torque outside or within operating limits.
3. The robotic device of claim 1, wherein the indicator includes a multi-color light emitting diode (LED).
4. The robotic device of claim 3, wherein the visual feedback provided by the LED has an intensity level based on the given amount of torque that the one or more joints is experiencing.
5. The robotic device of claim 1, wherein the indicator includes a multi-color light emitting diode (LED) for lighting at a given color to indicate whether the one or more joints is experiencing a given amount of torque outside or within operating limits.
6. The robotic device of claim 1, wherein the indicator provides the visual feedback using one or more of blinking lights, light patterns, or color patterns.
7. The robotic device of claim 1, wherein the indicator is provided on an exterior surface of the one or more joints.
8. The robotic device of claim 1, wherein the one or more actuators comprise a plurality of actuators coupled through a plurality of joints to form a multiple degree-of-freedom (DOF) robotic manipulator, and
the robotic device further comprises:
a plurality of indicators co-located in proximity to the plurality of joints to provide visual feedback on the robotic manipulator at the plurality of joints and about the plurality of joints.
9. A robotic device comprising:
a robotic manipulator comprising a plurality of links connected by one or more joints, wherein the plurality of links include one or more actuators coupled through the one or more joints;
an indicator located on a surface of the one or more joints to provide visual feedback on the robotic manipulator at a location on the one or more joints and the visual feedback is about the one or more joints, wherein the visual feedback includes information indicative of an operating status of the one or more joints, and wherein the operating status includes indicating an amount of torque that the one or more joints is experiencing;
a strain gauge for determining a tension experienced by the one or more actuators at the one or more joints; and
a controller coupled to the strain gauge for receiving information indicating the tension and for controlling the indicator co-located proximal to the one or more joints to provide the visual feedback indicative of the amount of torque that the one or more joints is experiencing based on the tension experienced.
10. The robotic device of claim 9, wherein the visual feedback includes information indicative of an operating status of the one or more joints.
11. The robotic device of claim 9, wherein the visual feedback includes information indicative of whether the one or more joints is experiencing a given amount of torque outside or within operating limits.
12. The robotic device of claim 9, wherein the indicator includes a multi-color light emitting diode (LED), and wherein the visual feedback provided by the LED has an intensity level based on the amount of torque that the one or more joints is experiencing.
13. The robotic device of claim 9, wherein the one or more actuators comprise a plurality of actuators coupled through a plurality of joints to form a multiple degree-of-freedom (DOF) robotic manipulator, and
the robotic device further comprises:
a plurality of indicators co-located in proximity to the plurality of joints to provide visual feedback on the robotic manipulator at the plurality of joints and about the plurality of joints.
14. A robotic device comprising:
a robotic manipulator comprising a plurality of links connected by a plurality of joints, wherein the plurality of links include a plurality of actuators coupled through the plurality of joints;
a plurality of indicators located on respective surfaces of the plurality of joints to provide visual feedback on the robotic manipulator at respective locations on the plurality of joints and the visual feedback is about the plurality of joints, wherein the visual feedback includes information indicative of operating statuses of the plurality of joints, and wherein the operating statuses include indicating respective amounts of torque that the plurality of joints are experiencing;
a plurality of sensors positioned at the plurality of joints for determining respective tensions experienced by the plurality of actuators at the plurality of joints, and
a controller coupled to the plurality of indicators for receiving information indicating the respective tensions and for and for controlling the plurality of indicators co-located proximal to the plurality of joints to provide the visual feedback indicative of the respective amounts of torque that the plurality of joints are experiencing based on the respective tensions experienced.
15. The robotic device of claim 14, wherein the visual feedback includes respective information indicative of whether any of the plurality of joints is experiencing a given amount of torque outside or within operating limits based on a respective indicator at a respective joint being activated.
16. The robotic device of claim 14, wherein the plurality of indicators include respective multi-color light emitting diodes (LEDs) for lighting at a given color to indicate whether respective joints of the plurality of joints are experiencing a given amount of torque outside or within operating limits of the respective joints.
17. The robotic device of claim 14, wherein the plurality of actuators are coupled to each other at the plurality of joints, and wherein the plurality of indicators are provided on exterior surfaces of the plurality of actuators.
18. The robotic device of claim 14, wherein the plurality of joints comprise respective interface elements that connect the plurality of actuators, and wherein the plurality of indicators are provided on the respective interface elements.
US14/288,438 2014-05-28 2014-05-28 Robotic devices with ambient indications of joint status Active US9375841B1 (en)

Priority Applications (7)

Application Number Priority Date Filing Date Title
US14/288,438 US9375841B1 (en) 2014-05-28 2014-05-28 Robotic devices with ambient indications of joint status
KR1020187023859A KR20180096818A (en) 2014-05-28 2015-04-13 Robotic devices with ambient indications of joint status
CN202210059416.5A CN114367987A (en) 2014-05-28 2015-04-13 Robot device with environmental indication of joint status
KR1020167032952A KR20160146979A (en) 2014-05-28 2015-04-13 Robotic devices with ambient indications of joint status
PCT/US2015/025506 WO2015183419A1 (en) 2014-05-28 2015-04-13 Robotic devices with ambient indictions of joint status
CN201580026444.5A CN107073709A (en) 2014-05-28 2015-04-13 Robot device with environmental indication of joint status
EP15799198.5A EP3148754A4 (en) 2014-05-28 2015-04-13 Robotic devices with ambient indictions of joint status

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US14/288,438 US9375841B1 (en) 2014-05-28 2014-05-28 Robotic devices with ambient indications of joint status

Publications (1)

Publication Number Publication Date
US9375841B1 true US9375841B1 (en) 2016-06-28

Family

ID=54699500

Family Applications (1)

Application Number Title Priority Date Filing Date
US14/288,438 Active US9375841B1 (en) 2014-05-28 2014-05-28 Robotic devices with ambient indications of joint status

Country Status (5)

Country Link
US (1) US9375841B1 (en)
EP (1) EP3148754A4 (en)
KR (2) KR20160146979A (en)
CN (2) CN114367987A (en)
WO (1) WO2015183419A1 (en)

Cited By (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160221193A1 (en) * 2015-01-29 2016-08-04 Canon Kabushiki Kaisha Robot apparatus, method for controlling robot, program, and recording medium
US20160263752A1 (en) * 2014-12-03 2016-09-15 Google Inc. Methods and Systems to Provide Feedback Based on a Motion Per Path Metric Indicative of an Effect of Motion Associated with Components of a Robotic Device
USD802041S1 (en) * 2017-04-27 2017-11-07 Engineering Services Inc Robotic arm
US10022861B1 (en) 2017-04-27 2018-07-17 Engineering Services Inc. Two joint module and arm using same
US20180251346A1 (en) * 2015-10-08 2018-09-06 Verton Technologies Australia Pty Ltd Materials management systems and methods
WO2019023378A1 (en) * 2017-07-27 2019-01-31 Intuitive Surgical Operations, Inc. Light displays in a medical device
DE102017009641A1 (en) * 2017-10-17 2019-04-18 Kuka Deutschland Gmbh Method and system for operating a robot arm
JP2019118993A (en) * 2017-12-29 2019-07-22 富士通株式会社 Robot, self-diagnostic program and self-diagnostic method
USD890238S1 (en) * 2018-03-02 2020-07-14 Abb Schweiz Ag Joint for an industrial robot
WO2020200468A1 (en) 2019-04-05 2020-10-08 Abb Schweiz Ag System for optimizing robot performance
US10821613B2 (en) * 2017-05-08 2020-11-03 Seiko Epson Corporation Robot
US20200376654A1 (en) * 2019-05-27 2020-12-03 Ati Industrial Automation, Inc. Robotic Tool Holder with Passive Compliance
US20200391394A1 (en) * 2019-06-12 2020-12-17 Flexiv Ltd. Robot and industrial robot with light device for indicating the state of actuators
US20210001484A1 (en) * 2018-03-27 2021-01-07 Productive Robotics, Inc. Collaborative Robot System Incorporating Enhanced Human Interface
CN112384341A (en) * 2018-07-17 2021-02-19 索尼公司 Control device, control method, and control system
US10933541B2 (en) 2016-10-17 2021-03-02 Fanuc Corporation Robot and method of installing signal lamp in robot
RU2745228C2 (en) * 2016-12-23 2021-03-22 Комау С.п.А. Functional device such as a robot with interconnected modules for educational applications
US11104002B2 (en) * 2014-06-10 2021-08-31 Canon Kabushiki Kaisha Robot apparatus
USD929481S1 (en) * 2018-04-27 2021-08-31 Staubli Faverges Industrial robots
US20220048192A1 (en) * 2018-12-05 2022-02-17 Siemens Aktiengesellschaft Articulated-arm robot
DE102019001525B4 (en) 2018-03-05 2022-09-01 Fanuc Corporation ROBOT STATUS NOTIFICATION DEVICE FOR NOTIFYING AN OPERATIONAL STATUS OF A ROBOT
US11685042B2 (en) * 2017-07-20 2023-06-27 Canon Kabushiki Kaisha Working robot and control method for working robot
US11834305B1 (en) 2019-04-12 2023-12-05 Vita Inclinata Ip Holdings Llc Apparatus, system, and method to control torque or lateral thrust applied to a load suspended on a suspension cable
US11932402B2 (en) 2019-04-12 2024-03-19 Vita Inclinata Ip Holdings Llc State information and telemetry for suspended load control equipment apparatus, system, and method
US11945697B2 (en) 2018-02-08 2024-04-02 Vita Inclinata Ip Holdings Llc Multiple remote control for suspended load control equipment apparatus, system, and method
US11992444B1 (en) 2023-12-04 2024-05-28 Vita Inclinata Ip Holdings Llc Apparatus, system, and method to control torque or lateral thrust applied to a load suspended on a suspension cable
US12099337B1 (en) * 2019-12-06 2024-09-24 Vita Inclinata Ip Holdings Llc Control moment gyroscope hoist stabilization system, method, and apparatus

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102591805B1 (en) 2016-11-04 2023-10-23 삼성전자주식회사 Antenna for Wearable Device
USD837294S1 (en) 2016-12-23 2019-01-01 Comau S.P.A. Educational robot
DE102017118982A1 (en) * 2017-04-23 2018-10-25 Franka Emika Gmbh Robot and method for operating a robot
KR102076907B1 (en) 2018-04-12 2020-02-12 한국기계연구원 Robot manipulator
CN109551522A (en) * 2019-01-03 2019-04-02 合肥安达创展科技股份有限公司 A kind of the science popularization device and demenstration method of robot automatic demonstrating gyro
CN111655434B (en) * 2019-06-12 2024-06-25 上海非夕机器人科技有限公司 Robot and industrial robot with lamp device for indicating state of actuator
KR20210043795A (en) 2019-10-11 2021-04-22 한국기계연구원 Structure for sensing external force applied to robot
DE102019128583B4 (en) * 2019-10-23 2022-03-10 Franka Emika Gmbh Optimization modes for control programs of a robot manipulator
WO2022041064A1 (en) * 2020-08-27 2022-03-03 Rethink Robotics Gmbh Method and apparatus for robot joint status monitoring
WO2024087108A1 (en) * 2022-10-27 2024-05-02 Shanghai Flexiv Robotics Technology Co., Ltd. Robot system and color control method

Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4786847A (en) 1986-11-20 1988-11-22 Unimation Inc. Digital control for multiaxis robots
US5299288A (en) 1990-05-11 1994-03-29 International Business Machines Corporation Image-directed robotic system for precise robotic surgery including redundant consistency checking
US5917428A (en) * 1996-11-07 1999-06-29 Reliance Electric Industrial Company Integrated motor and diagnostic apparatus and method of operating same
US6297742B1 (en) * 1996-08-22 2001-10-02 Csi Technology, Inc. Machine monitor with status indicator
US6785572B2 (en) 2001-11-21 2004-08-31 Koninklijke Philips Electronics, N.V. Tactile feedback and display in a CT image guided robotic system for interventional procedures
US20070163136A1 (en) 2004-01-14 2007-07-19 Homer Eaton Automated robotic measuring system
US20070182551A1 (en) * 2005-03-07 2007-08-09 Frank Yang Trash can with power operated lid
US20080013958A1 (en) 2006-03-30 2008-01-17 Kabushiki Kaisha Toshiba Information communication system for use in robot
US20080255772A1 (en) * 2007-02-06 2008-10-16 Abb Research Ltd. Method and a control system for monitoring the condition of an industrial robot
JP2011062792A (en) * 2009-09-18 2011-03-31 Denso Wave Inc Robot system
US8100133B2 (en) 2005-06-30 2012-01-24 Intuitive Surgical Operations Indicator for tool state and communication in multi-arm robotic telesurgery and method of use
US8145492B2 (en) 2004-04-07 2012-03-27 Sony Corporation Robot behavior control system and method, and robot apparatus
JP2012218139A (en) 2011-04-14 2012-11-12 Seiko Epson Corp Motor unit and robot
US8369992B2 (en) 2009-09-22 2013-02-05 GM Global Technology Operations LLC Embedded diagnostic, prognostic, and health management system and method for a humanoid robot
US20130274894A1 (en) 2012-03-14 2013-10-17 Vanderbilt Uninersity Coordinating operation of multiple lower limb devices

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2000141275A (en) * 1998-11-13 2000-05-23 Kawasaki Heavy Ind Ltd Life testing method and device for industrial robot
US7930065B2 (en) * 2005-12-30 2011-04-19 Intuitive Surgical Operations, Inc. Robotic surgery system including position sensors using fiber bragg gratings
WO2009128104A1 (en) * 2008-04-18 2009-10-22 Sea Vision S.R.L. Manipulator and related operation
CH701886A2 (en) * 2009-09-23 2011-03-31 Stefan Fischer Neuronics Ag Handling automation or robot arm, has set of visible illuminant elements arranged as matrix, where illuminant elements represent symbols that are formed as single color or multicolor pictograph or alphanumeric symbols
WO2012018821A2 (en) * 2010-08-02 2012-02-09 The Johns Hopkins University Method for presenting force sensor information using cooperative robot control and audio feedback
DE102010051633A1 (en) * 2010-11-17 2012-05-24 Ondal Medical Systems Gmbh Carrying system with operator assistance

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4786847A (en) 1986-11-20 1988-11-22 Unimation Inc. Digital control for multiaxis robots
US5299288A (en) 1990-05-11 1994-03-29 International Business Machines Corporation Image-directed robotic system for precise robotic surgery including redundant consistency checking
US6297742B1 (en) * 1996-08-22 2001-10-02 Csi Technology, Inc. Machine monitor with status indicator
US5917428A (en) * 1996-11-07 1999-06-29 Reliance Electric Industrial Company Integrated motor and diagnostic apparatus and method of operating same
US6785572B2 (en) 2001-11-21 2004-08-31 Koninklijke Philips Electronics, N.V. Tactile feedback and display in a CT image guided robotic system for interventional procedures
US20070163136A1 (en) 2004-01-14 2007-07-19 Homer Eaton Automated robotic measuring system
US8145492B2 (en) 2004-04-07 2012-03-27 Sony Corporation Robot behavior control system and method, and robot apparatus
US20070182551A1 (en) * 2005-03-07 2007-08-09 Frank Yang Trash can with power operated lid
US8100133B2 (en) 2005-06-30 2012-01-24 Intuitive Surgical Operations Indicator for tool state and communication in multi-arm robotic telesurgery and method of use
US20080013958A1 (en) 2006-03-30 2008-01-17 Kabushiki Kaisha Toshiba Information communication system for use in robot
US20080255772A1 (en) * 2007-02-06 2008-10-16 Abb Research Ltd. Method and a control system for monitoring the condition of an industrial robot
JP2011062792A (en) * 2009-09-18 2011-03-31 Denso Wave Inc Robot system
US8369992B2 (en) 2009-09-22 2013-02-05 GM Global Technology Operations LLC Embedded diagnostic, prognostic, and health management system and method for a humanoid robot
JP2012218139A (en) 2011-04-14 2012-11-12 Seiko Epson Corp Motor unit and robot
US20130274894A1 (en) 2012-03-14 2013-10-17 Vanderbilt Uninersity Coordinating operation of multiple lower limb devices

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
International Search Report and Written Opinion prepared by the Korean Patent Office in International Application No. PCT/US2015/025506 mailed Jun. 30, 2015.

Cited By (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11104002B2 (en) * 2014-06-10 2021-08-31 Canon Kabushiki Kaisha Robot apparatus
US20160263752A1 (en) * 2014-12-03 2016-09-15 Google Inc. Methods and Systems to Provide Feedback Based on a Motion Per Path Metric Indicative of an Effect of Motion Associated with Components of a Robotic Device
US9744678B2 (en) * 2014-12-03 2017-08-29 X Development Llc Methods and systems to provide feedback based on a motion per path metric indicative of an effect of motion associated with components of a robotic device
US10201901B2 (en) * 2015-01-29 2019-02-12 Canon Kabushiki Kaisha Robot apparatus, method for controlling robot, program, and recording medium
US20160221193A1 (en) * 2015-01-29 2016-08-04 Canon Kabushiki Kaisha Robot apparatus, method for controlling robot, program, and recording medium
US20210253404A1 (en) * 2015-10-08 2021-08-19 Verton IP Pty Ltd Materials management systems and methods
JP2018531201A (en) * 2015-10-08 2018-10-25 バートン テクノロジーズ オーストラリア プロプリエタリー リミテッド Material management system and method
US12091290B2 (en) * 2015-10-08 2024-09-17 Verton IP Pty Ltd Materials management systems and methods
US20180251346A1 (en) * 2015-10-08 2018-09-06 Verton Technologies Australia Pty Ltd Materials management systems and methods
JP7007731B2 (en) 2015-10-08 2022-01-25 バートン アイピー プロプリエタリー リミテッド Material management system and method
US11021347B2 (en) * 2015-10-08 2021-06-01 Verton Holdings Pty Ltd Materials management systems and methods
US10933541B2 (en) 2016-10-17 2021-03-02 Fanuc Corporation Robot and method of installing signal lamp in robot
RU2745228C2 (en) * 2016-12-23 2021-03-22 Комау С.п.А. Functional device such as a robot with interconnected modules for educational applications
US10022861B1 (en) 2017-04-27 2018-07-17 Engineering Services Inc. Two joint module and arm using same
USD802041S1 (en) * 2017-04-27 2017-11-07 Engineering Services Inc Robotic arm
US10821613B2 (en) * 2017-05-08 2020-11-03 Seiko Epson Corporation Robot
US11685042B2 (en) * 2017-07-20 2023-06-27 Canon Kabushiki Kaisha Working robot and control method for working robot
US11672621B2 (en) * 2017-07-27 2023-06-13 Intuitive Surgical Operations, Inc. Light displays in a medical device
WO2019023378A1 (en) * 2017-07-27 2019-01-31 Intuitive Surgical Operations, Inc. Light displays in a medical device
US20210085424A1 (en) * 2017-07-27 2021-03-25 Intuitive Surgical Operations, Inc. Light displays in a medical device
US20230404693A1 (en) * 2017-07-27 2023-12-21 Intuitive Surgical Operations, Inc. Light displays in a medical device
US11751966B2 (en) 2017-07-27 2023-09-12 Intuitive Surgical Operations, Inc. Medical device handle
US11618165B2 (en) 2017-10-17 2023-04-04 Kuka Deutschland Gmbh Method and system for operating a robotic arm
DE102017009641A1 (en) * 2017-10-17 2019-04-18 Kuka Deutschland Gmbh Method and system for operating a robot arm
JP2019118993A (en) * 2017-12-29 2019-07-22 富士通株式会社 Robot, self-diagnostic program and self-diagnostic method
US11945697B2 (en) 2018-02-08 2024-04-02 Vita Inclinata Ip Holdings Llc Multiple remote control for suspended load control equipment apparatus, system, and method
USD890238S1 (en) * 2018-03-02 2020-07-14 Abb Schweiz Ag Joint for an industrial robot
DE102019001525B4 (en) 2018-03-05 2022-09-01 Fanuc Corporation ROBOT STATUS NOTIFICATION DEVICE FOR NOTIFYING AN OPERATIONAL STATUS OF A ROBOT
US20210001484A1 (en) * 2018-03-27 2021-01-07 Productive Robotics, Inc. Collaborative Robot System Incorporating Enhanced Human Interface
USD929481S1 (en) * 2018-04-27 2021-08-31 Staubli Faverges Industrial robots
US11850751B2 (en) * 2018-07-17 2023-12-26 Sony Corporation Control device, control method, and control system
US20210323148A1 (en) * 2018-07-17 2021-10-21 Sony Corporation Control device, control method, and control system
CN112384341A (en) * 2018-07-17 2021-02-19 索尼公司 Control device, control method, and control system
US12070866B2 (en) * 2018-12-05 2024-08-27 Siemens Aktiengesellschaft Articulated-arm robot
US20220048192A1 (en) * 2018-12-05 2022-02-17 Siemens Aktiengesellschaft Articulated-arm robot
WO2020200468A1 (en) 2019-04-05 2020-10-08 Abb Schweiz Ag System for optimizing robot performance
US12053891B2 (en) 2019-04-05 2024-08-06 Abb Schweiz Ag System for optimizing robot performance
US11932402B2 (en) 2019-04-12 2024-03-19 Vita Inclinata Ip Holdings Llc State information and telemetry for suspended load control equipment apparatus, system, and method
US11834305B1 (en) 2019-04-12 2023-12-05 Vita Inclinata Ip Holdings Llc Apparatus, system, and method to control torque or lateral thrust applied to a load suspended on a suspension cable
US20200376654A1 (en) * 2019-05-27 2020-12-03 Ati Industrial Automation, Inc. Robotic Tool Holder with Passive Compliance
CN113134830A (en) * 2019-05-27 2021-07-20 奥腾工业自动化(廊坊)有限公司 Robot tool holder with passive compliance
US11724386B2 (en) * 2019-05-27 2023-08-15 Ati Industrial Automation, Inc. Robotic tool holder with passive compliance
US20200391394A1 (en) * 2019-06-12 2020-12-17 Flexiv Ltd. Robot and industrial robot with light device for indicating the state of actuators
US10933542B2 (en) * 2019-06-12 2021-03-02 Flexiv Ltd. Robot and industrial robot with light device for indicating the state of actuators
US12099337B1 (en) * 2019-12-06 2024-09-24 Vita Inclinata Ip Holdings Llc Control moment gyroscope hoist stabilization system, method, and apparatus
US11992444B1 (en) 2023-12-04 2024-05-28 Vita Inclinata Ip Holdings Llc Apparatus, system, and method to control torque or lateral thrust applied to a load suspended on a suspension cable

Also Published As

Publication number Publication date
EP3148754A1 (en) 2017-04-05
KR20160146979A (en) 2016-12-21
EP3148754A4 (en) 2018-04-18
KR20180096818A (en) 2018-08-29
CN107073709A (en) 2017-08-18
WO2015183419A1 (en) 2015-12-03
CN114367987A (en) 2022-04-19

Similar Documents

Publication Publication Date Title
US9375841B1 (en) Robotic devices with ambient indications of joint status
US10031511B2 (en) Arrangement and method for monitoring a position of a hand-held tool
KR102386763B1 (en) Safety system for industrial robot
US8090475B2 (en) Method and device for monitoring the condition of an industrial robot
US6813542B2 (en) Modules for use in an integrated intelligent assist system
US9044856B2 (en) Robot apparatus, method of controlling the same, and computer program
US20020112016A1 (en) System and architecture for providing a modular intelligent assist system
US9346165B1 (en) Robotic devices with multi-degree of freedom (DOF) load cell for shear beam sensing
JP2018103309A (en) Mechanical learning device and part assembly system for learning assembly operation
TWI748181B (en) System and method for measuring torque and angle
JP2011088260A (en) Robot control system
WO2017115385A2 (en) System and method for operating and controlling a hyper configurable humanoid robot to perform multiple applications in various work environments
US20210053218A1 (en) Robot controller
CN113165180B (en) Articulated arm robot
US20240299783A1 (en) Mounting device for a fall protection device and methods of using the same
US10933542B2 (en) Robot and industrial robot with light device for indicating the state of actuators
KR102381067B1 (en) Mobile remote work robot, its control system and control method
Hietanen et al. Proof of concept of a projection-based safety system for human-robot collaborative engine assembly
CN111406239A (en) Method for operating a processing device and configuration of a processing device
Zaman et al. Design and construction of a multipurpose robot
GB2567214A (en) Process performance measurement
CN113631326A (en) System for optimizing performance of robot
CN111655434B (en) Robot and industrial robot with lamp device for indicating state of actuator
KR20160066259A (en) High-payload supporting driving robot adopting contactless driver motion detecting system
US11774938B2 (en) Monitoring device and method for operating a monitoring device

Legal Events

Date Code Title Description
AS Assignment

Owner name: GOOGLE INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KEMPER, KEVIN;REEL/FRAME:032977/0248

Effective date: 20140527

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: X DEVELOPMENT LLC, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:GOOGLE INC.;REEL/FRAME:039900/0610

Effective date: 20160901

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:044144/0001

Effective date: 20170929

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE CORRECTIVE BY NULLIFICATION TO CORRECT INCORRECTLY RECORDED APPLICATION NUMBERS PREVIOUSLY RECORDED ON REEL 044144 FRAME 0001. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:047894/0508

Effective date: 20170929

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 4

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:X DEVELOPMENT LLC;REEL/FRAME:064658/0001

Effective date: 20230401

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

AS Assignment

Owner name: GOOGLE LLC, CALIFORNIA

Free format text: CORRECTIVE ASSIGNMENT TO CORRECT THE THE REMOVAL OF THE INCORRECTLY RECORDED APPLICATION NUMBERS 14/149802 AND 15/419313 PREVIOUSLY RECORDED AT REEL: 44144 FRAME: 1. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME;ASSIGNOR:GOOGLE INC.;REEL/FRAME:068092/0502

Effective date: 20170929