JP7530134B1 - Information processing device, program, and information processing method - Google Patents

Abstract

An information processing device for improving the reliability of a content generation system using generative AI is provided.
[Solution] According to the present invention, an information processing device for verifying a content generation system using a generative AI is provided, comprising a control means and a memory means, wherein the control means comprises an output information acquisition unit, an output information verification unit, a feedback information acquisition unit, a history information recording unit, and a reward model learning unit, wherein the output information acquisition unit acquires AI output information from the generative AI in response to input from an inputter who inputs a prompt to the content generation system, the output information verification unit verifies the AI output information, the feedback information acquisition unit acquires feedback information from the inputter in response to system output information from the content generation system, the history information recording unit records the AI output information and the feedback information in the memory means as history information, and the reward model learning unit learns a reward model based on the history information.
[Selected Figure] Figure 1

Description

The present invention relates to an information processing device, a program, and an information processing method.

In recent years, applications and services that use large-scale language models (LLMs) to generate text have been developed. For example, Patent Literature 1 discloses a device that generates questions (prompts) to be input to an LLM in order to improve the accuracy of answers from the LLM.

Patent No. 7313757

However, even if the prompts were improved, there was still a risk that the LLM or a sentence generation system using the LLM would generate an inappropriate answer sentence, making it impossible to provide or use the sentence generation system safely. Furthermore, the problem of outputting such undesired results was not limited to the LLM and the corresponding sentence generation system, but also occurred in other generation AIs such as image and video generation AI and corresponding content generation systems.

The present invention has been made in consideration of these circumstances, and provides an information processing device for improving the reliability of a content generation system that uses generation AI.

According to the present invention, the following inventions are provided.
[1] An information processing device for verifying a content generation system using a generative AI, comprising a control means and a storage means, wherein the control means comprises an output information acquisition unit, an output information verification unit, a feedback information acquisition unit, a history information recording unit, and a reward model learning unit, wherein the output information acquisition unit acquires AI output information from the generative AI in response to input from an input person who inputs a prompt to the content generation system, the output information verification unit verifies the AI output information, the feedback information acquisition unit acquires feedback information from the input person in response to system output information from the content generation system, the history information recording unit records the AI output information and the feedback information as history information in the storage means, and the reward model learning unit learns a reward model based on the history information.
[2] An information processing device as described in [1], wherein the control means further includes an input information acquisition unit, which acquires at least one of user input information from the user to the content generation system and system input information from the content generation system to the generation AI, the history information recording unit records at least one of the user input information and the system input information as the history information in the storage means in addition to the AI output information and the feedback information, and the reward model learning unit learns the reward model based on the history information.
[3] An information processing device as described in [2], wherein the input information acquisition unit and the feedback information acquisition unit provide an API to the content generation system in advance, and use the API to acquire at least one of the user input information and the system input information, and the feedback information, from the content generation system, respectively.
[4] An information processing device according to any one of [1] to [3], wherein the output information acquisition unit provides an API to the content generation system in advance and acquires the AI output information from the content generation system using the API.
[5] An information processing device according to any one of [1] to [4], wherein the control means further includes an output information presentation unit, and the output information presentation unit presents the verification result to the person inputting information together with or in place of the system output information based on the verification result of the output information verification unit.
[6] An information processing device according to any one of [1] to [5], wherein the reward model learning unit, in a learning phase, pre-learns the reward model based on the AI output information and an evaluation of the AI output information, the output information verification unit, in a provision phase, verifies the AI output information using the reward model, and the reward model learning unit, in the provision phase, updates the reward model based on the history information.
[7] An information processing device as described in [6], wherein the control means further includes an input information acquisition unit, which acquires, during the learning phase, at least one of user input information from the user inputting to the content generation system and system input information from the content generation system to the generation AI, and the reward model learning unit, during the learning phase, learns the reward model based on at least one of the user input information and the system input information for a learning dataset input by the user, the AI output information corresponding thereto, and an evaluation of the AI output information.
[8] An information processing device according to [6] or [7], wherein the output information verification unit, in addition to verification using the reward model, further verifies at least one of harmfulness, accuracy, and emotion of the AI output information in the provision phase.
[9] An information processing device according to [2] or [3], wherein the control means further includes an input information verification unit, and the input information verification unit verifies at least one of harmfulness and emotions of the information input by the person entering the information.
[10] An information processing device according to any one of [1] to [9], wherein the reward model learning unit learns the reward model for each individual user who inputs data or for each of multiple users who belong to the same group.
[11] An information processing device according to any one of [1] to [10], wherein the generative AI is a large-scale language model, and the content generation system is a sentence generation system.
[12] An information processing method for verifying a content generation system using a generative AI, comprising the steps of: an output information acquisition process; an output information verification process; a feedback information acquisition process; a history information recording process; and a reward model learning process, in which the output information acquisition process acquires AI output information from the generative AI in response to input from an input user who inputs a prompt to the content generation system; the output information verification process verifies the AI output information; the feedback information acquisition process acquires feedback information from the input user in response to system output information from the content generation system; the history information recording process records the AI output information and the feedback information as history information in a storage means; and the reward model learning process learns a reward model based on the history information.
[13] A program for verifying a content generation system using a generative AI, the program causing a computer to execute an output information acquisition process, an output information verification process, a feedback information acquisition process, a history information recording process, and a reward model learning process, wherein the output information acquisition process acquires AI output information from the generative AI in response to input from an inputter who inputs a prompt to the content generation system, the output information verification process verifies the AI output information, the feedback information acquisition process acquires feedback information from the inputter in response to system output information from the content generation system, the history information recording process records the AI output information and the feedback information as history information in a storage means, and the reward model learning process learns a reward model based on the history information.

The present invention makes it possible to improve the reliability of a content generation system that uses generative AI.

1 is a diagram showing an overall configuration of a system including an information processing apparatus 100 according to a first embodiment of the present invention. 2A is a block diagram showing the hardware configuration of information processing device 100 in FIG. 1, and FIG. 2B is a block diagram showing the hardware configuration of user terminal 400. As shown in FIG. FIG. 2B is a block diagram showing a functional configuration of a control unit 1 of the information processing apparatus 100 of FIG. 2A. This is a sequence diagram showing the processing performed between the user terminal 400, the content generation system 200, the generation AI 300 and the information processing device 100 when a user U uses the content generation system 200. 1. This is an example D1 of a comparison screen output by the content generation system 200 and displayed on the display unit 405 of the user terminal 400 of FIG. 1. This is an example of a screen D2 output by the content generation system 200 and displayed on the display unit 405 of the user terminal 400 of FIG. 1. This is an example of a comparison screen D3 outputted from the content generation system 200 and displayed on the display unit 405 of the user terminal 400 in FIG. 1. This is an example of a screen D4 output by the information processing device 100 and displayed on the display unit 405 of the user terminal 400 in FIG.

The following describes the embodiments of the present invention. The various features shown in the embodiments below can be combined with each other. In addition, each feature can be an invention independently.

1. First embodiment 1.1 Overall configuration of a system including an information processing device 100 Fig. 1 is a diagram showing the overall configuration of a system including an information processing device 100 according to an embodiment of the present invention, a content generation system 200 to which the information processing device 100 is applied, a generation AI 300, and a user terminal 400 used by a user U who uses the content generation system 200. Here, the generation AI 300 to which the information processing device 100 according to this embodiment is applied is an LLM (large-scale language model), and the content generation system 200 is a text generation system that generates text as content using the generation AI 300.

More specifically, LLM is an AI that learns from large amounts of text data and generates text that sounds like something a human would write. For example, it can complete sentences, answer questions, write essays, and provide responses to instructions or questions entered as prompts.

More specifically, the content generation system 200 accepts user input information (input person input information) from a user U as an input person via a user terminal 400, appropriately processes the input person input information and transmits it to the generation AI 300 as system input information, obtains AI output information from the generation AI, and provides this to the user U as system output information. In addition, the content generation system 200 to which the information processing device 100 of this embodiment is applied may be created or customized for each organization of the same industry, such as a call center, a sales department, or a technical department. For customization, data unique to the organization, such as a response manual or a business manual, is imported and used when processing user input information to make it system input information or when processing AI output information to make it system output information. However, the content generation system 200 may be customized for each individual.

The information processing device 100 of this embodiment is a device used to verify whether such a content generation system 200 outputs an inappropriate answer sentence, that is, whether the generation AI 300 operating behind the content generation system 200 outputs an inappropriate answer sentence. The information processing device 100 of this embodiment will be described in detail below. In this embodiment, the information processing device 100, the content generation system 200, the generation AI 300, and the user terminal 400 are connected via a network N.

2A is a block diagram showing the hardware configuration of the information processing device 100. As shown in FIG. 2A, the information processing device 100 specifically includes a control unit 1, a storage unit 2, and a communication unit 3.

The control means 1 is composed of one or more processors such as a CPU (Central Processing Unit), and controls the operation of the entire information processing device 100 by executing a predetermined program stored in the storage means 2. The control means 1 may also include other processors such as a GPU (Graphics Processing Unit) or a DSP (Digital Signal Processor). At least a part of the control means 1 may be an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

A part of the storage means 2 is composed of, for example, RAM (Random Access Memory) or DRAM (Dynamic Random Access Memory), and is used as a work area when the control means 1 executes processes based on various programs. In addition, a part of the storage means 2 is, for example, a non-volatile memory such as ROM (Read Only Memory), HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various data and programs used in the processing of the control means 1. Note that at least a part of the storage means 2 may be composed of an external cloud or distributed storage.

The programs stored in the storage means 2 include, for example, an OS (Operating System) for implementing the basic functions of the information processing device 100, drivers for controlling various hardware, programs for implementing various functions, etc., and include the computer program according to this embodiment.

In addition, as shown in FIG. 2A, the storage means 2 of this embodiment also stores history information HS and reward model RM. The history information HS is information including system input information, AI output information, and feedback information which is feedback from the user U on the system output information. The reward model RM is a model for evaluating the output (answer) of the generation AI 300, and in this embodiment, is learned based on the history information HS. Details of the history information HS and reward model RM will be described later.

The communication means 3 is, for example, a NIC (Network Interface Controller) and has a function of connecting to the network N. Note that the communication means 3 may have a function of connecting to a wireless LAN (Local Area Network) instead of or together with the NIC, a function of connecting to a wireless WAN (Wide Area Network), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), and infrared communication, for example. The information processing device 100 is connected to the content generation system 200 and the user terminal 400 via the network N, and can transmit and receive various data.

The control means 1, storage means 2, and communication means 3 are electrically connected to each other via a bus 4. Therefore, the control means 1 can access the storage means 2 and communicate with the content generation system 200, the user terminal 400, etc. via the communication means 3.

The information processing device 100 does not have to be configured as a single device as shown in FIG. 2, but may be realized as a plurality of devices using so-called cloud or distributed computing technology. In addition, the various data (information) stored in the storage means 2 does not have to be configured as a single database as shown in the figure, but may be configured as a distributed database.

1.3 Hardware Configuration of User Terminal 400 Fig. 2B is a block diagram showing the hardware configuration of the user terminal 400. The user terminal 400 is a terminal used by a user U who uses the content generation system 200, and is, for example, an information processing terminal such as a personal computer (PC), a smartphone, a tablet terminal, or an in-vehicle terminal. As shown in Fig. 2B, the user terminal 400 specifically includes a control unit 401, a storage unit 402, a communication unit 403, an input unit 404, and a display unit 405. These components are electrically connected to each other via a bus 406.

The general configuration of the control means 401, storage means 402, and communication means 403 is the same as that of the information processing device 100 described above, so the description will be omitted.

The input means 404 is a device that accepts input from the user U, and is composed of various input means such as a mouse, keyboard, touch panel, microphone, etc. The display means 405 is a liquid crystal display, touch panel display, etc., and displays images, etc. to the user U.

1.4 Functional configuration of (control means 1) of information processing device 100 As shown in Fig. 3, the control means 1 of the information processing device 100 includes an input information acquisition unit 10, an output information acquisition unit 11, a feedback information acquisition unit 12, a history information recording unit 13, a reward model learning unit 14, an output information presentation unit 15, an input information verification unit 16, and an output information verification unit 17. With these functional configurations, the information processing device 100 of this embodiment verifies whether the content generation system 200 (generation AI 300) outputs an inappropriate answer sentence. Each functional configuration will be described below.

In this embodiment, the verification function by the information processing device 100 is provided by exchanging information with the content generation system 200 via an API. In addition, in FIG. 1 and the following description, "user input information," "system input information," "AI output information," "system output information," and "feedback information" may refer to the natural language itself input by or presented to the user U, depending on the context, or may refer to information that has been digitized (encoded) for transmission/reception or recording in the storage means 2. Furthermore, it is also preferable that this information is encrypted.

As an input information acquisition process (see step S4 in FIG. 4), the input information acquisition unit 10 acquires system input information from the content generation system 200 to the generation AI 300 using an API (Application Programming Interface). Specifically, the input information acquisition unit 10 provides the API to the content generation system 200 in advance, and when the content generation system 200 transmits system input information to the generation AI 300, the input information acquisition unit 10 causes the content generation system 200 to transmit the system input information to the information processing device 100 at the same time or around the same time.

As an output information acquisition process (see step S10 in FIG. 4), the output information acquisition unit 11 uses an API to acquire AI output information from the generation AI 300 to the content generation system 200. Specifically, the output information acquisition unit 11 provides the API to the content generation system 200 in advance, and when the content generation system 200 acquires the AI output information from the generation AI 300, the output information acquisition unit 11 causes the content generation system 200 to transmit the AI output information to the information processing device 100 at the same time or at a later timing.

The feedback information acquisition unit 12 acquires feedback information from the user U on the system output information using an API as a feedback information acquisition process (see step S18 in FIG. 4). Specifically, the feedback information acquisition unit 12 provides the API to the content generation system 200 in advance, and when the content generation system 200 acquires feedback information from the user terminal 400, the feedback information acquisition unit 12 causes the content generation system 200 to transmit the feedback information to the information processing device 100 at the same time or at a later timing. The feedback information is the system output information presented to the user U by the content generation system 200 via the display means 405 of the user terminal 400, in other words, information on the evaluation by the user U of the response to the prompt input by the user U. In this embodiment, the feedback information is specifically, for example, information on a two-level evaluation of "good/bad" (a three-level evaluation including no response). However, it may be a multi-level evaluation (for example, a numerical evaluation such as 0 points to 100 points) or a written evaluation.

As part of the history information recording process (see steps S5, S11, and S19 in FIG. 4), the history information recording unit 13 records the system input information acquired by the input information acquisition unit 10, the AI output information acquired by the output information acquisition unit 11, and the feedback information acquired by the feedback information acquisition unit 12 as history information HS in the storage means 2. More specifically, the history information recording unit 13 associates the system input information, AI output information, and feedback information acquired in the series of steps in which the user U inputs a prompt, obtains a response, and provides feedback on the response, and records them as history information HS.

The reward model learning unit 14 learns the reward model RM based on the history information HS recorded by the history information recording unit 13 as a reward model learning process (see step S20 in FIG. 4). In this embodiment, learning the reward model RM includes pre-learning in the learning phase and updating in the provision phase. Here, the pre-learning in the learning phase is mainly learning the reward model RM before the user U uses the content generation system 200, and the updating in the provision phase is updating the reward model RM while the user U actually uses the content generation system 200.

The pre-learning in the learning phase is specifically performed by having an inputter other than the user U, such as a developer of the content generation system 200, an operator who operates the content generation system 200, a beta version user of the content generation system 200, or even a provider of the information processing device 100 according to this embodiment, input a large number of prompts (at least several hundred prompts) as a learning dataset, and obtaining an evaluation (feedback information) of the AI output information and the system output information for each prompt (i.e., system input information) from the inputter. Here, the inputter who receives feedback may be one person, or two or more people. However, one reward model RM is created and learned for each single inputter or for multiple inputters belonging to an organization (group) that uses the content generation system 200 for the same purpose. It is also possible to have the reward model RM learn by having another generation AI evaluate instead of a human.

On the other hand, updates in the provision phase are specifically performed each time history information HS is added by user U using content generation system 200, that is, each time a series of steps is performed in which user U inputs a prompt, obtains a response, and provides feedback on the response, or each time a predetermined amount of history information HS is accumulated (for example, each time 100 pieces of feedback are obtained).

Based on the verification result of the output information verification unit 17, the output information presentation unit 15 presents the verification result by the output information verification unit 17 to the user U via the display means 405 of the user terminal 400 together with or instead of the system output information as an output information presentation process (see warning message W in FIG. 6). Here, "instead of the system output information" means that if the verification result of the output information verification unit 17 is unfavorable and it is determined that the system output information should not be presented to the user U, the system output information is not displayed on the display means 405. A specific method of presenting the verification result will be described later. In addition, the output information presentation unit 15 presents statistical information on the input and output of the content generation system 200 based on the history information HS (see FIG. 8).

The input information verification unit 16 verifies the user input information acquired by the input information acquisition unit 10 as an input information verification process (see step S6 in FIG. 4). Specifically, as shown in FIG. 3, the input information verification unit 16 of this embodiment is configured to include an input harmfulness filter 16a and an input emotion filter 16b, and to verify the harmfulness and emotion of the user input information. Here, the "harmfulness of the user input information" refers to a state in which the user input contains inappropriate terms or expressions such as discrimination or prejudice, and the "emotion of the user input information" refers to a positive or negative emotional expression inferred from the user input. The input information verification unit 16 may include only either the input harmfulness filter 16a or the input emotion filter 16b. The input information verification unit 16 may also include additional filters other than these.

The output information verification unit 17 verifies the AI output information acquired by the output information acquisition unit 11 as an output information verification process (see step S13 in FIG. 4). Specifically, as shown in FIG. 3, the output information verification unit 17 of this embodiment includes an output harmfulness filter 17a, an output emotion filter 17b, a fact check filter 17c, and a reward model filter 17d. The output harmfulness filter 17a verifies the harmfulness of the AI output information, and the output emotion filter 17b verifies the emotion of the AI output information. Here, the "harmfulness of the AI output information" refers to a state in which the AI output contains inappropriate terms or expressions such as discrimination or prejudice, and the "emotion of the AI output information" refers to a positive emotional expression or a negative emotional expression inferred from the AI output. In addition, the fact check filter 17c verifies whether the content of the AI output information is in line with the facts (accuracy), and outputs the degree of agreement with the facts. In addition, the reward model filter 17d verifies the AI output information using the reward model RM. Specifically, the reward model filter 17d evaluates (e.g., a numerical evaluation from 0 to 1 points) the system output information (which is essentially the same as the AI output information output by the generation AI 300) output by the content generation system 200 in response to the system input information. However, the evaluation of the system output information by the reward model filter 17d can also be classified into classes such as A, B, and C. Verification of the AI output information by the reward model filter 17d will be described later. Note that the output information verification unit 17 does not have to include at least one of the output emotion filter 17b, the fact check filter 17c, and the reward model filter 17d. The output information verification unit 17 may also include additional filters other than these.

The above-mentioned functional configurations and processes may be realized by software (including so-called apps) that is appropriately installed in the information processing device 100, or may be realized by hardware. When realized by software, the various functions can be realized by the control means 1 executing the programs that make up the software. Also, they may be realized by multiple pieces of software rather than a single piece of software.

When it is realized by executing a program, the program may be stored in the storage means 2 built into the information processing device 100, or may be stored in a non-transitory recording medium that is readable by a computer. It may also be realized by reading out a program stored in an external storage device, so-called cloud computing. Or, when it is realized by hardware, it can be realized by various circuits such as an ASIC, SOC, FPGA, or DRP. Also, at least a part of the functional configurations described above may be processed by software or hardware on a user terminal 400 that accepts input, etc.

The above-mentioned functional configuration may be realized by multiple computers, in which case each of the above-mentioned functional configurations may be distributed among multiple computers.

1.5 Information processing method by the information processing device 100 Next, an information processing method in which the information processing device 100 of this embodiment verifies the output of the content generation system 200 (output of the generation AI 300) will be described using the sequence diagram of Fig. 4 and screen examples D1 to D4 of Fig. 5 to Fig. 8. Note that the sequence diagram of Fig. 4 includes not only the processing of the information processing device 100, but also the processing of the user terminal 400, the content generation system 200, and the generation AI 300. Also, as described above, the verification by the information processing device 100 of this embodiment is realized by transmitting and receiving necessary information between the content generation system 200 and the information processing device 100 via an API.

Specifically, in the information processing method of this embodiment, first, in step S1, the user terminal 400 accepts an input, i.e., a prompt, from the user U to the content generation system 200 via the input means 404. Next, in step S2, the user terminal 400 transmits the accepted user input information to the content generation system 200. Next, in step S3, the content generation system 200 processes the user input information as necessary. Here, processing of the user input information by the content generation system 200 can be, for example, by adding supplemental information to the prompt entered by the user U using a mechanism of RAG (Retrieval Augmented Generation).

Next, in step S4, the content generation system 200 transmits the processed user input information to the information processing device 100 as system input information, and the input information acquisition unit 10 of the control means 1 of the information processing device 100 acquires the system input information.

Next, in step S5, the history information recording unit 13 of the control means 1 records the acquired history information HS by the input information acquisition unit 10 in the storage means 2. In step S6, the input information verification unit 16 of the control means 1 verifies the system input information, and in step S7, transmits the verification result of the system input information to the content generation system 200. Note that the content generation system 200 can freely set how to utilize the verification result by the information processing device 100. For example, if the input contains harmful words or sentences, it is conceivable to warn the user U or block the transmission of the input from the user U to the generation AI 300.

Next, in step S8, the content generation system 200 transmits the system input information to the generation AI 300 unless the verification result of the system input information indicates that the system input information should not be transmitted to the generation AI 300. In step S9, the generation AI 300 receives the system input information as a prompt and transmits the AI output information to the content generation system 200 as a response.

Next, in step S10, the content generation system 200 transmits the AI output information acquired from the generation AI 300 to the information processing device 100, and the output information acquisition unit 11 of the control means 1 of the information processing device 100 acquires the AI output information.

Next, in step S11, the history information recording unit 13 of the control means 1 records the AI output information acquired by the output information acquisition unit 11 in the storage means 2 as history information HS. In addition, in step S12, the output information verification unit 17 of the control means 1 verifies the AI output information, and in step S13, transmits the verification result to the content generation system 200.

Next, in step S14, the content generation system 200 processes the AI output information based on the verification result of the AI output information acquired from the information processing device 100, and generates system output information to be transmitted to the user terminal 400. In step S15, the generated system output information is transmitted to the user terminal 400 and displayed on the display means 405.

FIG. 5 shows a comparative screen example D1 in which a user U asks "What is the smallest prefecture in Japan?" in the prompt input field A1 of the content generation system 200, and the answer "The smallest prefecture is Hokkaido" is displayed in the response field A2. Here, since the smallest prefecture is not actually Hokkaido, this answer can be said to be an inappropriate answer. When the generation AI 300 outputs such an answer as the AI output information, the output information verification unit 17 of the information processing device 100 determines, for example, that the AI output information is an output that may be hallucination (i.e., plausible false information), and the output information presentation unit 15 transmits the verification result to the content generation system 200. As a result, the content generation system 200 can process the AI output information and generate system output information to which a warning statement W is added to warn that the output is a possible hallucination, as shown in the screen example D2 of FIG. 6.

Also, FIG. 7 is a comparison screen example D3 showing an example in which the generated AI 300 gives a harmful answer (bad word) to the user U as AI output information as a result of the user U pointing out an error in the prompt input field A1 in response to an incorrect answer from the generated AI 300 as shown in FIG. 5. When the generated AI 300 outputs such an answer as the AI output information, the output information verification unit 17 of the information processing device 100 determines, for example, that the AI output information is an output that should not be presented to the user U, and transmits the verification result to the content generation system 200. This enables the content generation system 200 to avoid presenting a harmful answer to the user U. In this case, the content generation system 200 can generate system output information instead of the AI output information, for example, indicating that the output content cannot be displayed because the output was problematic.

Next, in step S16, the user terminal 400 accepts feedback on the system output information (e.g., a two-level evaluation of good/bad) from the user U via the input means 404. Then, in step S17, the user terminal 400 transmits the accepted feedback information to the content generation system 200. In step S18, the content generation system 200 transmits the feedback information to the information processing device 100, and the feedback information acquisition unit 12 of the control means 1 acquires the feedback information.

Next, in step S19, the history information recording unit 13 of the control means 1 records the feedback information acquired by the feedback information acquisition unit 12 as history information HS in the storage means 2. Note that the history information recording unit 13 is configured to record the system input information in step S5, the AI output information in step S11, and the feedback information in step S19 in association with each other.

Finally, in step S20, the reward model learning unit 14 of the control means 1 updates the reward model RM based on the recorded history information HS.

By performing the above-described series of steps S1 to S20, in the information processing method of this embodiment, the reward model RM is updated each time the user U uses the content generation system 200 to provide feedback, or each time a predetermined amount of history information HS is accumulated.

In addition, as shown in FIG. 8, the output information presentation unit 15 of the information processing device 100 can also present the latest statistical information on the input and output of the content generation system 200 to the user U via a web page, email, etc., based on the recorded history information HS. In the example screen D4 shown in FIG. 8, statistical data on the harmfulness of the AI output information (harmfulness of text) is displayed.

1.6 Effects (1) According to the information processing device 100 of this embodiment, the output information acquisition unit 11 acquires the AI output information from the generation AI 300, and the output information verification unit 17 verifies the AI output information based on the reward model RM. The feedback information acquisition unit 12 acquires feedback information on the system output information from the user U via the content generation system 200, the history information recording unit 13 records the feedback information together with the AI output information as history information HS, and the reward model learning unit 14 learns and updates the reward model RM based on the history information HS. Since the information processing device 100 of this embodiment is configured in this way, the output information verification unit 17 can verify the content generation system 200 (system output information) based on the reward model RM that is updated daily by the feedback information, and it is possible to improve the reliability of the content generation system 200 in a society that changes daily.

(2) In the information processing device 100 according to this embodiment, the input information acquisition unit 10 acquires system input information and records it together with AI output information and feedback information as history information HS, and the reward model learning unit 14 learns and updates the reward model RM based on the history information HS. In this way, by also using the system input information to learn and update the reward model RM, it is possible to improve the accuracy of learning and updating the reward model RM.

(3) In the information processing device 100 according to this embodiment, the input information acquisition unit 10, the output information acquisition unit 11, and the feedback information acquisition unit 12 are each configured to acquire corresponding information from the content generation system 200 via an API. This makes it possible for the output information verification unit 17 to verify the content generation system 200 and learn and update the reward model RM even when information cannot be acquired directly from the generation AI 300 or the user terminal 400. It is also preferable to design the API so that when the content generation system 200 provides each piece of information to the information processing device 100, the information is encrypted (vectorized). This allows the developer and operator of the content generation system 200, and each user U, to use the information processing device 100 with peace of mind.

(4) In the information processing device 100 according to this embodiment, the output information presenting unit 15 displays the verification result by the output information verifying unit 17 on the display means 405 of the user terminal 400 used by the user U, together with or instead of the system output information by the content generation system 200. This allows the user U of the content generation system 200 to determine whether the response by the content generation system 200 (or the response by the generation AI) is trustworthy.

(5) The information processing device 100 according to this embodiment is configured such that the reward model learning unit 14 pre-learns the reward model RM using a learning dataset in the learning phase, and then in the provision phase, the output information verification unit 17 verifies the AI output information using the trained reward model RM. This makes it possible to output verification results customized for the user U from the beginning of the provision of the content generation system 200 to the user U, to which the information processing device 100 is applied.

(6) In the information processing device 100 according to this embodiment, the control means 1 includes an output information verification unit 17 that includes an output harmfulness filter 17a, an output emotion filter 17b, and a fact check filter 17c in addition to a reward model filter 17d. This makes it possible to perform comprehensive verification that is not limited to verification using the reward model filter 17d, which is tuned based on feedback from the user U.

(7) Furthermore, the control means 1 of this embodiment also includes an input information verification unit 16 equipped with an input harmfulness filter 16a and an input emotion filter 16b, and it is possible to prevent the reward model RM from inappropriately learning due to inappropriate user input information, AI output information (system output information) based on the inappropriate user input information, and feedback information thereto. In addition, it is possible to protect the content generation system 200 from inappropriate input.

(8) In this embodiment, the reward model learning unit 14 creates, pre-learns, and updates a reward model RM for each individual user U or for each group of multiple users U (inputters) who use the content generation system 200 for the same purpose. As a result, the reward model RM is generated and updated to suit the purpose of a single user or group, and the verification results are fine-tuned, making it possible to provide each user U with more accurate verification results than if all users U used the same reward model (general-purpose reward model).

(9) In this embodiment, the content generation system 200 to which the information processing device 100 is applied is a text generation system, and the generation AI 300 is an LLM (large-scale language model). Inappropriate responses (answers) by LLMs and text generation systems using them have become a particular problem in recent years because their inappropriateness is easily highlighted. When an inappropriate response is reported, developers and operators of the content generation system 200 using an external LLM service may be forced to suspend the provision of the system until the LLM service responds, or may be forced to deal with the problem on the system side. In this regard, according to the information processing device 100 of this embodiment, the output of the content generation system 200 and the generation AI 300 is verified from the standpoint of a third party that is neither the content generation system 200 nor the generation AI 300, and the reward model RM is updated daily. This makes it possible for developers and operators of the content generation system 200 to provide the system with peace of mind, even if they do not themselves respond to inappropriate responses.

3. Modifications The present invention can also be implemented in the following aspects.

In the above embodiment, the information processing device 100 was configured to verify a text generation system using an LLM (large-scale language model) as a content generation system 200 using the generative AI 300. However, the information processing device 100 of the present invention may also be used to verify an image generation system using AI to generate images, a video generation system using AI to generate videos, or even a music generation system using AI to generate music.

In the above embodiment, the information processing device 100 was configured such that the history information recording unit 13 records, as history information HS, the AI output information acquired by the output information acquisition unit 11 as well as the user input information acquired by the input information acquisition unit 10, together with the feedback information acquired by the feedback information acquisition unit 12, and the reward model learning unit 14 learns and updates the reward model RM based on the history information HS. However, the history information recording unit 13 may also learn and update the reward model RM based on the AI output information and feedback information, i.e., not based on the user input information.

The input information acquisition unit 10 in this embodiment is configured to acquire the processed prompt sent by the content generation system 200 to the generation AI 300 as AI input information. However, instead, the input information acquisition unit 10 may acquire the raw prompt (the prompt itself entered by the user U) sent by the user terminal 400 to the content generation system 200 as AI input information. In this case, the reward model RM learns and is updated based on this information in addition to the AI output information and feedback information.

In the above embodiment, the input information acquisition unit 10, the output information acquisition unit 11, and the feedback information acquisition unit 12 were each configured to acquire corresponding information from the content generation system 200 via an API. However, the method of acquiring information of the input information acquisition unit 10, the output information acquisition unit 11, and the feedback information acquisition unit 12 according to the present invention is not limited to this. For example, at least one of the input information acquisition unit 10 and the feedback information acquisition unit 12 may directly acquire corresponding information from the user terminal 400 via the network N. For example, it is possible to acquire the information via a web browser or by using an application resident in the terminal such as virus software. In addition, the output information acquisition unit 11 may acquire an answer directly from the generation AI 300. Furthermore, the information processing device 100 may directly accept an input from the content generation system 200 and transmit the input to the generation AI 300 as a prompt, and the information processing device 100 may directly accept a response from the generation AI 300 and transmit the response to the content generation system 200. In this case, the information processing device 100 is placed between the content generation system 200 and the generation AI 300, and functions as a sort of firewall for the content generation system 200 or the generation AI 300.

In the above embodiment, the output information acquisition unit 11 of the information processing device 100 acquired the AI output information, and recorded the AI output information together with the user input information and feedback information as history information HS for use in learning the reward model RM. However, it is also possible for the output information acquisition unit 11 to acquire system output information instead of the AI output information, record it as history information HS, and use it for learning the reward model RM. However, if the system output information already contains the verification result sent by the output information presentation unit 15 of the information processing device 100, a loop will occur if the output information verification unit 17 verifies it again, so it is necessary to extract only the output of the generation AI 300 from the system output information and perform verification, etc.

In the above embodiment, the output information presentation unit 15 was configured to transmit the verification result by the output information verification unit 17 to the content generation system 200, and process the AI output information on the content generation system 200 to display the warning message W (see FIG. 6). However, the verification result by the output information verification unit 17 can also be displayed separately from the content generation system 200, for example, by a web browser, a pop-up, a push notification, etc. It is also preferable to display the verification result on the content generation system 200 system and further enable confirmation together with separate statistical data on a web browser, etc.

In the above embodiment, the reward model learning unit 14 pre-learns the reward model RM after applying the information processing device 100 to the content generation system 200 and before the content generation system 200 is provided to the user U. However, the information processing device 100 according to the present invention can also provide the reward model RM to the user U without pre-learning it. In this case, the reward model RM is learned solely based on feedback from the user U.

In the above embodiment, the information processing device 100 acquires system input information by the input information acquisition unit 10 and acquires AI output information by the output information acquisition unit 11 at different times (step S4/step S10). However, it is also possible to acquire system input information and AI output information simultaneously. In this case, subsequent recording and verification of the system input information and AI output information as history information HS can also be performed simultaneously.

In the above embodiment, it was assumed that feedback would be received from the user U regarding the answer (AI output information) by the generation AI 300 among the system output information, but separately from this, feedback may also be received regarding the verification results by the output information verification unit 17 presented by the output information presentation unit 15.

In the above embodiment, the information processing device 100 is connected to the content generation system 200 via a network N, and verifies the content generation system 200 (generation AI 300) as a so-called web service. However, the information processing device 100 may be configured on-premise on the same local network or the same private cloud as the content generation system 200.

In the above embodiment, the content generation system 200 and the generation AI 300 are each connected to the user terminal 400 via the network N to provide various functions to the user U. However, the content generation system 200 and the generation AI 300 may each be provided as a system that runs on an integrated circuit mounted on the user terminal 400, specifically, a local device such as a PC, a mobile terminal, an automobile, or a medical device, so-called edge AI. In this case, at least some of the functional elements of the information processing device 100 can also be placed on-premise on the same user terminal 400 (local device). Also, even if the generation AI 300 is on the network N and only the content generation system 200 is on the user terminal 400, the information processing device 100 can be installed on-premise on the same device as the content generation system 200.

1: Control means 2: Storage means 3: Communication means 4: Bus 10: Input information acquisition unit 11: Output information acquisition unit 12: Feedback information acquisition unit 13: History information recording unit 14: Reward model learning unit 15: Output information presentation unit 16: Input information verification unit 16a: Input harmfulness filter 16b: Input emotion filter 17: Output information verification unit 17a: Output harmfulness filter 17b: Output emotion filter 17c: Fact check filter 17d: Reward model filter 100: Information processing device 200: Content generation system 300: Generation AI
400: User terminal 401: Control means 402: Storage means 403: Communication means 404: Input means 405: Display means 406: Bus A1: Prompt input field A2: Response field HS: History information N: Network RM: Reward model U: User W: Warning message

Claims (13)

An information processing device for verifying a content generation system using generation AI,
The content generation system accepts input information from an input user who inputs a prompt, transmits it to the generation AI as system input information, obtains AI output information from the generation AI for the system input information, and provides system output information to the input user based on the AI output information;
A control means and a storage means are provided,
The control means includes an output information acquisition unit, an output information verification unit, a feedback information acquisition unit, a history information recording unit, and a reward model learning unit,
The output information acquisition unit acquires the AI output information,
The output information verification unit verifies the AI output information,
the feedback information acquisition unit acquires feedback information from the person inputting the system output information,
The history information recording unit records the AI output information and the feedback information as history information in the storage means,
The reward model learning unit learns a reward model based on the history information. The information processing device according to claim 1 ,
The control means further includes an input information acquisition unit,
the input information acquisition unit acquires at least one of the person input information and the system input information,
The history information recording unit records at least one of the input person input information and the system input information as the history information in the storage means in addition to the AI output information and the feedback information,
The reward model learning unit learns the reward model based on the history information. 3. The information processing device according to claim 2,
The input information acquisition unit and the feedback information acquisition unit provide an API to the content generation system in advance, and use the API to acquire at least one of the user input information and the system input information, and the feedback information, from the content generation system, respectively. The information processing device according to claim 1 ,
The output information acquisition unit provides an API to the content generation system in advance and acquires the AI output information from the content generation system using the API. 2. The information processing device according to claim 1,
The control means further includes an output information presentation unit,
The information processing apparatus, wherein the output information presenting unit presents the verification result to the person inputting information together with the system output information or in place of the system output information based on the verification result of the output information verifying unit. The information processing device according to claim 1 ,
The reward model learning unit pre-learns the reward model based on the AI output information and an evaluation of the AI output information in a learning phase,
The output information verification unit verifies the AI output information using the reward model in the provision phase,
The reward model learning unit updates the reward model based on the history information in the provision phase. 7. The information processing device according to claim 6,
The control means further includes an input information acquisition unit,
the input information acquisition unit acquires at least one of the user input information and the system input information in the learning phase;
The reward model learning unit, during the learning phase, learns the reward model based on at least one of the user input information and the system input information for a learning dataset input by the user, the AI output information corresponding thereto, and an evaluation of the AI output information. 7. The information processing device according to claim 6,
The information processing device, wherein the output information verification unit further verifies at least one of harmfulness, accuracy, and emotion of the AI output information in addition to verification using the reward model in the provision phase. 3. The information processing device according to claim 2,
The control means further includes an input information verification unit,
The input information verification unit verifies at least one of harmfulness and emotion of the input user input information. 2. The information processing device according to claim 1,
The reward model learning unit is an information processing device that learns the reward model for each individual user who inputs data or for each of multiple users who belong to the same group. An information processing device according to any one of claims 1 to 10,
An information processing device, wherein the generative AI is a large-scale language model, and the content generation system is a sentence generation system. An information processing method for verifying a content generation system using generation AI, comprising:
The content generation system accepts input information from an input user who inputs a prompt, transmits it to the generation AI as system input information, obtains AI output information from the generation AI for the system input information, and provides system output information to the input user based on the AI output information;
performing an output information acquisition process, an output information verification process, a feedback information acquisition process, a history information recording process, and a reward model learning process;
In the output information acquisition process, the AI output information is acquired,
In the output information verification process, the AI output information is verified,
In the feedback information acquisition process, feedback information from the person inputting the system output information is acquired,
In the history information recording process, the AI output information and the feedback information are recorded as history information in a storage means;
In the reward model learning process, a reward model is learned based on the history information. A program for verifying a content generation system using generation AI,
The content generation system accepts input information from an input user who inputs a prompt, transmits it to the generation AI as system input information, obtains AI output information from the generation AI for the system input information, and provides system output information to the input user based on the AI output information;
causing a computer to execute an output information acquisition process, an output information verification process, a feedback information acquisition process, a history information recording process, and a reward model learning process;
In the output information acquisition process, the AI output information is acquired,
In the output information verification process, the AI output information is verified,
In the feedback information acquisition process, feedback information from the person inputting the system output information is acquired,
In the history information recording process, the AI output information and the feedback information are recorded as history information in a storage means;
In the reward model learning process, a reward model is learned based on the history information.