JP2023044336A - Learning apparatus, learning method, and program - Google Patents

Abstract

To provide a technology capable of obtaining a model with high interpretability and accuracy, while suppressing human cost.SOLUTION: A learning apparatus includes: an input unit which acquires first input data and a ground truth value of the first input data; an inference unit which outputs first inference values corresponding to each of a plurality of inference models, on the basis of the first input data and the inference models; an explanation unit which outputs first explanatory information corresponding to each of the inference models that indicates a degree of contribution of the first input data to the first inference values; an inference evaluation unit which obtains an inference evaluation result on the basis of the ground truth value and the first inference value; an explanation evaluation unit which obtains an explanation evaluation result on the basis of a matching degree between the pieces of first explanatory information corresponding to each of the inference models; and an update unit which updates a first weight parameter of the inference models, on the basis of the inference evaluation result and the explanation evaluation result.SELECTED DRAWING: Figure 1

Description

The present invention relates to a learning device, a learning method and a program.

A neural network (hereinafter also referred to as “NN”) has high performance in image recognition and the like. However, NNs are generally composed of a huge number of parameters and complicated models, and it is difficult to interpret the relationship between NN parameters and output results from NNs. In order to solve this problem, several techniques for obtaining NN with high interpretability have been proposed. Note that "highly interpretable" can also be translated into "highly consistent with human senses."

For example, a heat map that shows the areas that the NN model should pay attention to for judgment is manually labeled, and the model is trained so that it matches the heat map. is known (see, for example, Non-Patent Document 1). In addition, when the interpretability of the heat map obtained from the model is low, the model can be re-learned so as not to match the heat map, thereby obtaining a model with higher interpretability.

Also known is a method of improving the accuracy of the NN by introducing a mechanism into the network for extracting a region of interest for the NN to make judgments from the input data (see, for example, Non-Patent Document 2). Interpretability and accuracy of the NN can be improved by manually correcting the attention area obtained by such a method and re-learning the NN so that the corrected attention area and the NN attention area match.

Andrew Ross, 2 others, "Right for the Right Reasons: Training Differentiable Models by Constraining their Explanations", [online], [searched September 8, 2021], Internet <https://arxiv.org/abs/ 1703.03717> Masahiro Mitsuhara, 6 others, "Embedding Human Knowledge into Deep Neural Network via Attention Map", [online], [Searched on September 8, 2021], Internet <https://arxiv.org/abs/1905.03540> "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", [online], [searched September 8, 2021], Internet <https://arxiv.org/abs/1610.02391v3>

However, the method of manually preparing heat map labels described in Non-Patent Literature 1 and Non-Patent Literature 2 entails a large labor cost for labeling.

On the other hand, as a method that does not require labeling, there is a method of re-learning the model so that it does not match the heat map of the trained model described in Non-Patent Document 1. However, in such a method, there is a problem that there is a high possibility that the accuracy will be lowered by re-learning. Furthermore, in such a method, the degree of matching of the heat map uniformly decreases for all data, so the interpretability of the heat map may rather decrease for individual data. is an issue.

Therefore, it is desired to provide a technique that can obtain a highly interpretable and accurate model while suppressing human costs.

In order to solve the above problem, according to an aspect of the present invention, an input unit that acquires first input data and a correct value of the first input data; an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on a model; and the plurality of inferences that indicate a contribution magnitude of the first input data to the first inference value an explanation unit that outputs first explanation information corresponding to each model; an inference evaluation unit that obtains an inference evaluation result based on the correct value and the first inference value; and an inference evaluation unit that corresponds to each of the plurality of inference models. an explanation evaluation unit that obtains an explanation evaluation result based on the degree of matching between first explanation information; and an update of first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result. and an updating unit for performing.

The input unit acquires second input data, and the inference unit is based on the second input data and a plurality of trained models, which are a plurality of estimation models after updating the first weight parameters. and outputs a second inference value corresponding to each of the plurality of trained models, and the explanation unit outputs the plurality of learning models indicating the magnitude of contribution of the second input data to the second inference value. The learning device may include a presentation control unit that outputs second explanation information corresponding to each of the finished models, and controls presentation of the second explanation information to the user.

The presentation control unit may control presentation of the second inference value and the second explanatory information to the user.

The learning device may include a recording control unit that controls recording of information indicating one or a plurality of trained models selected by the user from the plurality of trained models.

The explanation evaluation result may take a smaller value as the degree of matching between the first explanation information corresponding to each of the plurality of inference models increases.

The explanation evaluation unit may obtain the explanation evaluation result based on an inner product of vectors obtained by normalizing the first explanation information corresponding to each of the plurality of inference models.

The explanation evaluation unit generates a mask by binarizing the first explanation information for each of the plurality of inference models, and generates a mask from the first explanation information corresponding to an inference model other than itself. A product of the mask and the first explanation information corresponding to its own inference model may be calculated, and the explanation evaluation result may be obtained based on the sum of the products for each of the plurality of inference models.

The explanation part may include a function capable of error backpropagation.

The explanation unit may have a second weight parameter, and the update unit may update the second weight parameter by error backpropagation.

At least one of the plurality of inference models may include a neural network. It should be noted that neural networks are just one example of machine learning algorithms. Therefore, other machine learning algorithms may be used instead of neural networks.

The updating unit may update the first weighting parameter based on an addition result of the inference evaluation result and the explanation evaluation result.

The first explanation information may be a heat map indicating the magnitude of contribution of the first input data to the first inference value.

According to another aspect of the present invention, obtaining first input data and a correct value of the first input data, and based on the first input data and a plurality of inference models, outputting a first inference value corresponding to each of a plurality of inference models; obtaining an inference evaluation result based on the correct value and the first inference value; obtaining an explanation evaluation result based on the inference evaluation result; and updating a first weight parameter of the plurality of inference models based on the inference evaluation result and the explanation evaluation result. be.

According to another aspect of the present invention, a computer comprises: an input unit that acquires first input data and correct values of the first input data; and the first input data and a plurality of inference models. and an inference unit that outputs a first inference value corresponding to each of a plurality of inference models, based on each of the plurality of inference models, and each of the plurality of inference models that indicates the magnitude of contribution of the first input data to the first inference value. an explanation unit that outputs first explanation information corresponding to the inference model, an inference evaluation unit that obtains an inference evaluation result based on the correct value and the first inference value, and a first inference model corresponding to each of the plurality of inference models an explanation evaluation unit that obtains an explanation evaluation result based on the degree of matching between the explanation information of the above; and an update that updates the first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result. and a program functioning as a learning device is provided.

As described above, according to the present invention, there is provided a technology capable of obtaining a model with high interpretability and accuracy while suppressing human costs.

1 is a diagram showing a functional configuration example of a learning device according to an embodiment of the present invention; FIG. FIG. 10 is a diagram for explaining a method of obtaining an explanation evaluation result by multiplying a mask obtained by binarizing a heat map and another heat map; 4 is a flow chart showing an example of the operation in the learning stage of the learning device according to the same embodiment; 6 is a flow chart showing an example of operation in a test stage of the learning device according to the same embodiment; 1 is a diagram showing a hardware configuration of an information processing device as an example of a learning device; FIG.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

In addition, in this specification and drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different numerals after the same reference numerals. However, when there is no particular need to distinguish between a plurality of constituent elements having substantially the same functional configuration, only the same reference numerals are used. Also, similar components in different embodiments may be distinguished by attaching different alphabets after the same reference numerals. However, when there is no particular need to distinguish between similar components of different embodiments, only the same reference numerals are used.

(0. Outline of embodiment)
An outline of an embodiment of the present invention will be described. In the embodiment of the present invention, a learning device that performs neural network learning based on a combination of input data (learning data) and correct values will be described. But neural networks are just one example of machine learning algorithms. Therefore, other machine learning algorithms may be used instead of neural networks. For example, SVM (Support Vector Machine) or the like may be used as another example of the machine learning algorithm.

(1. Details of embodiment)
Embodiments of the present invention will be described in detail.

(1.1. Configuration example of learning device)
FIG. 1 is a diagram showing a functional configuration example of a learning device 10 according to an embodiment of the present invention. As shown in FIG. 1, the learning device 10 according to the embodiment of the present invention includes an input unit 101, an inference unit 102, an explanation unit 103, an inference evaluation unit 104, an explanation evaluation unit 105, and an update unit 106. , presentation control unit 107 , recording control unit 108 , display unit 121 , and operation unit 122 .

In the embodiment of the present invention, it is mainly assumed that the reasoning unit 102 includes n (n is an integer greater than 1) reasoning models, that is, from "first reasoning model" to "nth reasoning model". . Moreover, in the embodiments of the present invention, it is mainly assumed that each of the first to n-th inference models includes a neural network. Below, a neural network is also written as "NN."

The NNs included in each of the first to n-th inference models use weight parameters 110 (first weight parameters). At this time, the NNs included in each of the first to n-th inference models may have a common structure and may use different weighting parameters 110 (first weighting parameters). Alternatively, the NNs included in each of the first to n-th inference models may have separate structures.

At least one of the first to n-th inference models may include NN. For example, part of the first to n-th inference model may include NN, and another part of the first to n-th inference model includes other machine learning algorithms instead of NN. It's okay.

Furthermore, in the embodiment of the present invention, it is mainly assumed that the explanation part 103 is configured including the NN. The NN included in the explanation part 103 uses a weight parameter (second weight parameter).

The data set 100, the weighting parameters 110 (first weighting parameters) from the first to n-th inference models, and the weighting parameters (second weighting parameters) of the explanation section 103 are stored in a storage section (not shown). . The storage unit may be composed of a memory such as a RAM (Random Access Memory), a hard disk drive, or a flash memory.

The input unit 101, the inference unit 102, the explanation unit 103, the inference evaluation unit 104, the explanation evaluation unit 105, the update unit 106, the presentation control unit 107, and the recording control unit 108 are implemented by a CPU (Central Processing Unit). ) or a GPU (Graphics Processing Unit), and a program stored in a ROM (Read Only Memory) is loaded into a RAM and executed by the arithmetic device to realize its function. At this time, a computer-readable recording medium recording the program may also be provided. Alternatively, these blocks may be composed of dedicated hardware, or may be composed of a combination of multiple pieces of hardware. Data necessary for calculation by the calculation device are appropriately stored in a storage unit (not shown).

In the initial state, the weight parameters 110 of the first to n-th inference models and the weight parameters of the explanation part 103 are set to initial values. For example, the initial values set to these may be random values, but may be any value. For example, the initial values set for these may be learned values obtained in advance through learning.

(Dataset 100)
Data set 100 includes a plurality of input data (first input data) used in the learning stage and correct values for each of the plurality of input data. A plurality of input data used in the learning stage may correspond to data for learning. Furthermore, data set 100 includes a plurality of input data (second input data) used in the test phase. A plurality of input data used in the test phase may correspond to test data.

In addition, it is mainly assumed that the test data is prepared as data different from the learning data. However, the test data may include part of the learning data.

Also, in the embodiments of the present invention, it is mainly assumed that input data is image data (particularly still image data). However, the type of input data is not particularly limited, and data other than image data can be used as input data. For example, the input data may be moving image data including a plurality of frames, or may be audio data.

(Input unit 101)
In the learning stage, the input unit 101 sequentially acquires combinations of input data and correct values used in the learning stage from the data set 100 . The input unit 101 sequentially outputs combinations of input data and correct values used in the learning stage to the inference unit 102 . Also, in the test stage, the input unit 101 sequentially acquires input data used in the test from the data set 100 . The input unit 101 sequentially outputs input data used in the test stage to the inference unit 102 .

Note that, for example, when the input unit 101 acquires and outputs all combinations of input data and correct values used in the learning stage from the data set 100, it acquires and outputs the combinations again from the beginning. The operation of performing may be repeated a predetermined number of times. In such a case, the blocks subsequent to the input unit 101 may sequentially repeat their processes based on the re-input. On the other hand, for example, the input unit 101 may terminate the acquisition of input data when all the input data used in the test stage have been acquired from the data set 100 and output.

(Inference unit 102)
In the learning stage, the inference unit 102 performs inference corresponding to each of the first to nth inference models based on the input data input from the input unit 101 and the first to nth inference models. Get a value (first inference value). Similarly, in the test stage, the inference unit 102, based on the input data input from the input unit 101 and the first inference model to the nth inference model, Obtain the corresponding inference value (second inference value).

The weight parameters 110 used by the first to n-th inference models are stored in a storage unit (not shown). Therefore, the inference unit 102 acquires the weight parameter 110 from the storage unit (not shown), and based on the acquired weight parameter 110 and the input data input from the input unit 101, the first to n-th inference models make inferences.

In this specification, obtaining an output from a NN based on an input to the NN is broadly called "inference".

As an example, if the function representing the i-th inference model is Fi (i is an integer from 1 to n) and the input to the i-th inference model is x, the output from the i-th inference model is Fi (x ).

As will be described later, the explanation method (that is, the method of generating explanation information) used by the explanation unit 103 includes, in addition to the inference values, the feature values output from each of the first to nth inference models. There may be cases where there are explanation methods that require information such as quantities (intermediate features). In such a case, the inference unit 102 may output to the explanation unit 103 feature amounts output from the respective intermediate layers from the first inference model to the n-th inference model together with the inference value.

The specific configurations of the first to n-th inference models are not particularly limited. However, it is preferable that the output format of each of the first to n-th inference models is set together with the format of the correct value corresponding to the input data. For example, if the correct answer is a class of a classification problem, each output from the first inference model to the nth inference model may be a one-hot vector having a length equal to the number of classes.

Inference unit 102 outputs inference values corresponding to each of the first to n-th inference models to explanation unit 103 and inference evaluation unit 104 in the learning stage. On the other hand, inference unit 102 outputs inference values corresponding to the first to n-th inference models to explanation unit 103 and presentation control unit 107 in the test stage.

(Description part 103)
The explanation unit 103 generates explanation information explaining the basis for judgment of the inference value input from the inference unit 102 for each of the first to n-th inference models.

Here, the explanation information is information indicating the degree of contribution of the input data to the inference value input from the inference unit 102 . In the following, a case where the explanation information is a heat map showing the magnitude of contribution of input data to an inference value for each region (for example, pixels constituting an image) or for each variable will be mainly explained. A heat map can show the important areas or variables of the input data that contributed to the decision.

If the input data is image data or the like, the heatmap can be represented by a two-dimensional vector. Alternatively, if the input data is tabular data or the like, the heatmap can be represented by a one-dimensional vector.

A heatmap may be generated in any way. For example, the explanation unit 103 may generate a heat map based on the inference values input from the inference unit 102 . Alternatively, as described above, not only the inference value but also the feature amount may be input from the inference unit 102 to the description unit 103 . In such a case, the explanation unit 103 may generate a heat map based on the inference values and feature amounts input from the inference unit 102 .

For example, the explanation part 103 may include a function capable of back propagation. At this time, as will be described later, the updating unit 106 can update the weighting parameters of the explanation unit 103 by the error backpropagation method. That is, the explanation unit 103 may generate a heat map using the weight parameters updated by the backpropagation method.

The so-called Grad-CAM described in Non-Patent Document 3 can be applied as an explanation method for generating a heat map using weight parameters after being updated by the error backpropagation method. Grad-CAM is an explanation method that outputs a heat map showing regions of inputs to the NN that have a high degree of contribution to inference values. In addition, various explanation methods such as Vanilla Gradient and SmoothGrad can be applied.

As described above, the inference value corresponding to the i-th inference model can be expressed as Fi(x). Therefore, as an example, assuming that the function indicating the heat map generation process is G, the i A heat map Ti(x) corresponding to the th inference model can be expressed as in Equation (1) below.

Ti(x)=G(Fi(x)) (1)

Explanation unit 103 outputs the generated n heat maps (first explanation information) to explanation evaluation unit 105 in the learning stage. On the other hand, the explanation unit 103 outputs the generated n heat maps (second explanation information) to the presentation control unit 107 in the test stage.

(Inference evaluation unit 104)
The inference evaluation unit 104 obtains an inference evaluation result based on the inference values corresponding to the first to n-th inference models input from the inference unit 102 and the correct values obtained by the input unit 101. . More specifically, the inference evaluation unit 104 obtains an inference evaluation result by comparing the inference values corresponding to the first to n-th inference models with the correct values obtained by the input unit 101 .

In the embodiment of the present invention, the inference evaluation unit 104 uses the sum of the first to n-th inference models of the loss functions corresponding to the inference value and the correct value as the loss function L1 as an example of the inference evaluation result. Assume the case of calculation. Here, the loss function according to the inference value and the correct value is not limited to a specific function, and a loss function similar to loss functions used in general neural networks may be used. For example, the loss function according to the inferred value and the correct value may be a cross-entropy error based on the difference between the correct value and the inferred value.

The inference evaluation unit 104 outputs the inference evaluation result to the update unit 106 .

(Description evaluation unit 105)
Explanation evaluation unit 105 obtains explanation evaluation results based on heat maps corresponding to the first to n-th inference models input from explanation unit 103 . More specifically, the explanation evaluation unit 105 compares heat maps corresponding to the first to n-th inference models. Then, the description evaluation unit 105 obtains a description evaluation result based on the degree of matching between the n heat maps as the comparison result.

In the embodiment of the present invention, it is mainly assumed that the loss function takes a smaller value for the explanation evaluation result as the degree of matching between the n heat maps increases. Note that the degree of matching between the n heat maps may be rephrased as the degree of divergence indicating how much the n heat maps diverge from each other. In such a case, the loss function may be such that the smaller the degree of divergence between the n heat maps, the smaller the explanation evaluation result.

The method of obtaining explanatory evaluation results from n heatmaps is not limited. Here, as a method to obtain explanation evaluation results, a method to obtain explanation evaluation results by multiplying a heat map binarized mask with another heat map, and a method to obtain explanation evaluation results by using the inner product of normalized heat maps The method for obtaining the results will be described in order.

FIG. 2 is a diagram for explaining a method of obtaining an explanation evaluation result by multiplying a mask obtained by binarizing a heat map and another heat map. In the example shown in FIG. 2, to simplify the explanation, it is assumed that n=2, that is, the reasoning unit 102 has the first reasoning model and the second reasoning model.

Referring to FIG. 2, the inference values and the heat map H1 are output from the first inference model. On the other hand, an inference value and a heat map H2 are output from the second inference model. In FIG. 2, in the heat map H1 and the heat map H2, areas of the input data that contribute more to the inference value are indicated by darker colors.

The explanation evaluation unit 105 binarizes the heat map H1 to generate the mask M1, and binarizes the heat map H2 to generate the mask M2. Note that binarization can be performed by assigning 1 to the values of elements that are equal to or greater than the threshold c (for example, pixels that make up the heatmap) and to 0 to the values of elements that are smaller than the threshold c. In FIG. 2, 1's of the binary are indicated by black and 0's by white.

The explanation evaluation unit 105 calculates the product of the heat map H1 output from the first inference model and the mask M2 generated from the heat map H2 output from the second inference model for each element. Similarly, the explanation evaluation unit 105 calculates the product of the heat map H2 output from the second inference model and the mask M1 generated from the heat map H1 output from the first inference model for each element. This yields a set of products corresponding to each element for each inference model.

The explanation evaluation unit 105 calculates the sum of the products by adding the products corresponding to each element for all the inference models. Then, the explanation evaluation unit 105 calculates a total value by adding the sums of products calculated in this way for all the elements. The explanation evaluation unit 105 uses this total value as a loss function L2 as an example of the explanation evaluation result.

The case of n=2 has been described with reference to FIG. If n is an arbitrary integer greater than 1, it is as follows.

That is, the explanation evaluation unit 105 generates a mask Mi(x) by binarizing the values of the elements of the heat map Ti(x) for i=1 to n. Next, for each inference model, the explanation evaluation unit 105 generates masks M1(x) to Mi Compute the product of −1(x) and the sum of Mi+1(x) to Mn(x) element by element.

The explanation evaluation unit 105 calculates the sum of the products by adding the products corresponding to the respective elements for the first to n-th inference models. Then, the explanation evaluation unit 105 obtains an explanation evaluation result based on the sum of products calculated in this way. More specifically, the explanation evaluation unit 105 calculates the total value by adding the sum of products for all the elements. The explanation evaluation unit 105 uses this total value as a loss function L2 as an example of the explanation evaluation result.

This loss function L2 is the total value of regions having values equal to or greater than the threshold value in heat maps other than the heat map itself. By performing learning so as to reduce the value of this loss function L2, n inference models with a low heat map matching degree are obtained. It should be noted that the loss function L2 at this time can be expressed as in Equation (2) below. In formula (2), e indicates an element number. Here, the heat map Ti(x) may be normalized by dividing by the size |Ti(x)| of the heat map Ti(x). Also, the heat map Ti(x) may be multiplied by an activation function such as sigmoid.

With reference to FIG. 2, a method of obtaining explanatory evaluation results by multiplying a mask obtained by binarizing a heat map with another heat map has been described. Next, a method of obtaining an explanation evaluation result by an inner product of normalized heat maps will be described.

The explanation evaluation unit 105 normalizes the heat map Ti(x) for i = 1 to n by dividing it by the size of the heat map Ti(x) |Ti(x)| for i = 1 to n. Generate a normalized vector. Then, explanation evaluation section 105 obtains explanation evaluation results based on the inner product of normalized vectors for i=1 to n. More specifically, the explanation evaluation unit 105 calculates the total value by adding the inner products of all the elements. The explanation evaluation unit 105 uses this total value as a loss function L2 as an example of the explanation evaluation result.

The larger the inner product of the normalized vectors, the larger the value of this loss function L2. A large inner product of normalized vectors means a high degree of matching between the heat maps. Therefore, by performing learning so as to reduce the value of this loss function L2, n inference models with a small degree of matching in the heat map are obtained. It should be noted that the loss function L2 at this time can be expressed as in Equation (3) below. In formula (3), e indicates an element number.

Description evaluation section 105 outputs the description evaluation result to update section 106 .

(Update unit 106)
Based on the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105, the update unit 106 updates the weights used by the first to n-th inference models. Update the parameters 110 . As a result, the inference values output from each of the first inference model to the n-th inference model approach the correct values, and the degree of matching between the n heat maps output from the explanation unit 103 decreases. As such, the weight parameter 110 may be updated. Weight parameters 110 may be updated by backpropagation.

For example, the update unit 106 adds the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105, and updates the weight parameter 110 based on the addition result. good. At this time, the update unit 106 may update the weight parameter 110 by error back propagation using the calculated addition result as an error. As described above, when the inference evaluation result is expressed as a loss function L1 and the explanation evaluation result is expressed as a loss function L2, the addition result is L1+L2.

Furthermore, the updating unit 106 may update the weighting parameters that the explanation unit 103 has. More specifically, when the explanation part 103 includes a function that can be backpropagated, the update part 106 updates the explanation part 103 by error backpropagation based on the inference evaluation result and the explanation evaluation result. 103 may have a weight parameter updated.

Note that the learning end condition (that is, the weight parameter update end condition) is not particularly limited as long as it indicates that the learning from the first inference model to the nth inference model has been performed to some extent. Specifically, the learning termination condition may include a condition that the value of the loss function L1+L2 is smaller than a threshold. Alternatively, the learning end condition may include a condition that the change in the value of the loss function L1+L2 is smaller than a threshold (a condition that the value of the loss function L1+L2 has converged). Alternatively, the learning termination condition may include a condition that the weighting parameters have been updated a predetermined number of times. Alternatively, when the inference evaluation unit 104 calculates the accuracy (for example, the correct answer rate) based on the correct value and the inference value, the learning termination condition is that the accuracy exceeds a predetermined percentage (for example, 90%) may include the condition

(Presentation control unit 107)
In the test stage, the presentation control unit 107 compares the inference values corresponding to the first to n-th inference models input from the inference unit 102 and the n heat maps input from the explanation unit 103. , to be presented to the user. More specifically, the presentation control unit 107 controls the display unit 121 so that the inference values corresponding to the first to n-th inference models and n heat maps are displayed. It is also conceivable that the n heat maps are displayed but the inference values corresponding to the first to n-th inference models are not displayed.

(Display unit 121)
The display unit 121 is configured by a display, and has a function of displaying various information under the control of the presentation control unit 107 . For example, the display unit 121 can display n inference values and n heat maps. Here, the form of the display unit 121 is not particularly limited. For example, the display unit 121 may be a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, or a display device such as a lamp.

(Operation unit 122)
The operation unit 122 accepts user's operations. For example, while referring to n inference values and n heat maps, the user finds one or more inference models with high interpretability (hereinafter also referred to as "selected models") from n inference models. Suppose At this time, the user inputs information indicating the selected model (hereinafter also referred to as “selected model information”) to the operating section 122 , and the operating section 122 receives the selected model information 123 . For example, the selected model information 123 may be a number indicating the selected model.

Note that the embodiment of the present invention mainly assumes that the operation unit 122 is a mouse and a keyboard. However, the form of the operation unit 122 is not particularly limited. For example, the operation unit 122 may be a touch panel or other input device.

(Recording control unit 108)
The recording control unit 108 controls recording of the selected model information 123 received from the user through the operation unit 122 . More specifically, the recording control unit 108 causes the storage unit (not shown) to store the selected model information 123 received from the user through the operation unit 122 . The selection model information 123 is acquired later from a storage unit (not shown), and the selection model indicated by the selection model information 123 can be used as a highly interpretable trained model.

Note that the conditions for ending the test are not particularly limited, and may be any condition that indicates that the test has been performed a sufficient number of times for the user. Specifically, the end condition of the test may include a condition that the user confirms the inference result more than a predetermined number of times in the test stage.

The configuration example of the learning device 10 according to the embodiment of the present invention has been described above.

(1.2. Operation in the learning stage)
The flow of operations in the learning stage of the learning device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a flow chart showing an operation example in the learning stage of the learning device 10 according to the embodiment of the present invention.

First, as shown in FIG. 3, the input unit 101 acquires a combination of input data (that is, learning data) and correct values from the dataset 100 . Furthermore, the inference unit 102 acquires the weight parameters 110 corresponding to each of the n inference models (S11). The inference unit 102 makes an inference based on the input data acquired by the input unit 101 and the n inference models (S12). output to each.

Based on the n inference values input from the inference unit 102, the explanation unit 103 generates a heat map explaining the basis for judgment of each of the n inference values (S13). Explanation unit 103 outputs the generated n heat maps to explanation evaluation unit 105 .

The inference evaluation unit 104 evaluates n inference values input from the inference unit 102 based on the correct values obtained by the input unit 101 to obtain inference evaluation results. More specifically, the inference evaluation unit 104 calculates a loss function corresponding to the correct value and the n inference values as an inference evaluation result. The inference evaluation unit 104 outputs the calculated inference evaluation result to the update unit 106 .

Description evaluation unit 105 obtains a description evaluation result based on the matching degree of the n heat maps input from description unit 103 . More specifically, the explanation evaluation unit 105 calculates a loss function according to the degree of matching between the n heat maps input from the explanation unit 103 as the explanation evaluation result. The explanation evaluation unit 105 outputs the calculated explanation evaluation result to the update unit 106 (S14).

Based on the inference evaluation result input from the inference evaluation unit 104 and the explanation evaluation result input from the explanation evaluation unit 105, the updating unit 106 updates weights corresponding to the first to n-th inference models. The parameter 110 is updated (S15). More specifically, the updating unit 106 updates the weight parameter 110 by error backpropagation based on the inference evaluation result and the explanatory evaluation result. Furthermore, the updating unit 106 updates the weighting parameters of the explanation unit 103 by backpropagation based on the inference evaluation result and the explanation evaluation result.

The update unit 106 determines whether or not the learning termination condition is satisfied each time the update of the weight parameter based on the input data is completed (S16). If it is determined that the learning end condition is not satisfied ("NO" in S16), the operation proceeds to S11, the next input data is acquired by the input unit 101, the inference unit 102, the explanation unit 103, The inference evaluation unit 104, the explanation evaluation unit 105, and the update unit 106 each perform their respective processes again based on the next input data. On the other hand, when update unit 106 determines that the end condition for learning is satisfied ("YES" in S16), learning ends.

The flow of operations in the learning stage of the learning device 10 according to the embodiment of the present invention has been described above.

(1.3. Operation in the test stage)
The flow of operations in the test stage of the learning device 10 according to the embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flow chart showing an operation example in the test stage of the learning device 10 according to the embodiment of the present invention.

First, as shown in FIG. 4, the input unit 101 acquires a combination of input data (that is, test data) and correct values from the dataset 100 . Furthermore, the inference unit 102 acquires the weight parameters 110 corresponding to each of the n inference models (S21). The inference unit 102 performs inference based on the input data acquired by the input unit 101 and the n inference models (S22), and presents the n inference values obtained by the inference to the explanation unit 103 and the presentation control unit 107. output to each.

Based on the n inference values input from the inference unit 102, the explanation unit 103 generates a heat map explaining the basis for the determination of each of the n inference values (S23). The explanation unit 103 outputs the generated n heat maps to the presentation control unit 107 .

The presentation control unit 107 controls the display unit 121 so that the n inference values input from the inference unit 102 and the n heat maps input from the explanation unit 103 are presented to the user. The display unit 121 displays n inference values and n heat maps under the control of the presentation control unit 107 (S24).

The operation unit 122 receives information (selected model information 123) indicating one or a plurality of inference models determined to have high interpretability from the n inference models from the user. The recording control unit 108 controls recording of the selected model information 123 received from the user by the operation unit 122 (S25). A storage unit (not shown) stores the selected model information 123 under the control of the recording control unit 108 .

Each time recording control of the selected model information 123 based on the input data is finished, the recording control unit 108 determines whether or not the end condition of the test is satisfied (S26). If it is determined that the end condition of the test is not satisfied ("NO" in S26), the operation proceeds to S21, the next input data is acquired by the input unit 101, the inference unit 102, the explanation unit 103, Each of the presentation control unit 107 and the recording control unit 108 re-executes its own processing based on the next input data. On the other hand, when the recording control unit 108 determines that the end condition of the test is satisfied ("YES" in S26), the test is ended.

The flow of operations in the test stage of the learning device 10 according to the embodiment of the present invention has been described above.

(1.4. Effect of Embodiment)
As described above, according to the embodiment of the present invention, the inference values output from each of the first inference model to the n-th inference model are output as explanatory information so as to approach the correct value. Learning can be performed so that the degree of matching between the n heatmaps becomes small. This makes it possible to obtain an inference model that outputs a plurality of different heatmaps. This allows the user to select and use a model that outputs a more interpretable heat map from among the n models.

The effects of the embodiment of the present invention have been described above.

(2. Hardware configuration example)
Next, a hardware configuration example of the learning device 10 according to the embodiment of the present invention will be described. A hardware configuration example of the information processing device 900 will be described below as a hardware configuration example of the learning device 10 according to the embodiment of the present invention. Note that the hardware configuration example of the information processing device 900 described below is merely an example of the hardware configuration of the learning device 10 . Therefore, as for the hardware configuration of the learning device 10, unnecessary configurations may be deleted from the hardware configuration of the information processing device 900 described below, or a new configuration may be added.

FIG. 5 is a diagram showing the hardware configuration of an information processing device 900 as an example of the learning device 10 according to the embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, and an interface 907. , an input device 908 , an output device 909 , a storage device 910 and a communication device 911 .

The CPU 901 functions as an arithmetic processing device and a control device, and controls general operations within the information processing device 900 according to various programs. Alternatively, the CPU 901 may be a microprocessor. The ROM 902 stores programs, calculation parameters, and the like used by the CPU 901 . The RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like. These are interconnected by a host bus 904 comprising a CPU bus or the like.

The host bus 904 is connected via a bridge 905 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904, the bridge 905 and the external bus 906 do not necessarily have to be configured separately, and these functions may be implemented in one bus.

The input device 908 includes input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901 . etc. A user who operates the information processing apparatus 900 can input various data to the information processing apparatus 900 and instruct processing operations by operating the input device 908 .

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker.

The storage device 910 is a device for data storage. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 is configured by, for example, an HDD (Hard Disk Drive). The storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

The communication device 911 is, for example, a communication interface configured with a communication device or the like for connecting to a network. Also, the communication device 911 may support either wireless communication or wired communication.

The hardware configuration example of the learning device 10 according to the embodiment of the present invention has been described above.

(3. Summary)
Although the preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention belongs can conceive of various modifications or modifications within the scope of the technical idea described in the claims. It is understood that these also naturally belong to the technical scope of the present invention.

For example, in the above example, it is mainly assumed that the learning device 10 learns n inference models simultaneously. However, the learning device 10 does not have to learn all of the n inference models at the same time. For example, a trained inference model may be used as part of the n inference models. At this time, the weight parameter of the learned inference model can be fixed at a constant value without being updated.

Further, in the above example, it is mainly assumed that the number of types of heat map generation methods in the explanation unit 103 is one. However, a plurality of types of heat map generation techniques may be used in the explanation section 103 . At this time, the explanation unit 103 may output to the update unit 106, as an example of the explanation evaluation result, a total value for a plurality of types of loss heat map generation methods based on the degree of matching between the heat maps.

10 learning device 100 data set 101 input unit 102 inference unit 103 explanation unit 104 inference evaluation unit 105 explanation evaluation unit 106 update unit 107 presentation control unit 108 recording control unit 110 weight parameter 121 display unit 122 operation unit 123 selected model information

Claims (14)

an input unit that acquires first input data and a correct value of the first input data;
an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and the plurality of inference models;
an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models indicating the magnitude of contribution of the first input data to the first inference value;
an inference evaluation unit that obtains an inference evaluation result based on the correct value and the first inference value;
an explanation evaluation unit that obtains an explanation evaluation result based on a degree of matching between first explanation information corresponding to each of the plurality of inference models;
an updating unit that updates first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result;
A learning device comprising: The input unit acquires second input data,
The inference unit, based on the second input data and a plurality of trained models that are a plurality of estimation models after the update of the first weight parameter, a second model corresponding to each of the plurality of trained models. outputs the inferred value of
The explanation unit outputs second explanation information corresponding to each of the plurality of trained models indicating the magnitude of contribution of the second input data to the second inference value,
The learning device includes a presentation control unit that controls presentation of the second explanatory information to the user.
A learning device according to claim 1. The presentation control unit controls presentation to the user of the second inference value and the second explanatory information.
3. A learning device according to claim 2. The learning device
a recording control unit that controls recording of information indicating one or more trained models selected by the user from the plurality of trained models;
4. The learning device according to claim 2 or 3. The explanation evaluation result takes a smaller value as the degree of matching between the first explanation information corresponding to each of the plurality of inference models increases.
A learning device according to any one of claims 1 to 4. The explanation evaluation unit obtains the explanation evaluation result based on an inner product of vectors obtained by normalizing the first explanation information corresponding to each of the plurality of inference models.
The learning device according to claim 5. The explanation evaluation unit generates a mask by binarizing the first explanation information for each of the plurality of inference models, and generates a mask from the first explanation information corresponding to an inference model other than itself. calculating the product of the mask and the first explanation information corresponding to its own inference model, and obtaining the explanation evaluation result based on the sum of the products for each of the plurality of inference models;
The learning device according to claim 5. The explanation part includes a function capable of backpropagation,
A learning device according to any one of claims 1 to 7. The description part has a second weighting parameter,
The update unit updates the second weight parameter by error backpropagation.
9. A learning device according to claim 8. at least one of the plurality of inference models includes a neural network;
A learning device according to any one of claims 1 to 9. The update unit updates the first weight parameter based on the addition result of the inference evaluation result and the explanation evaluation result.
A learning device according to any one of claims 1 to 10. The first explanatory information is a heat map showing the magnitude of contribution of the first input data to the first inference value,
A learning device according to any one of claims 1 to 11. obtaining first input data and a correct value of the first input data;
outputting a first inference value corresponding to each of a plurality of inference models based on the first input data and a plurality of inference models;
outputting first explanation information corresponding to each of the plurality of inference models indicating the magnitude of contribution of the first input data to the first inference value;
obtaining an inference evaluation result based on the correct value and the first inference value;
Obtaining an explanation evaluation result based on a degree of matching between first explanation information corresponding to each of the plurality of inference models;
updating first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result;
How to learn, including the computer,
an input unit that acquires first input data and a correct value of the first input data;
an inference unit that outputs a first inference value corresponding to each of a plurality of inference models based on the first input data and the plurality of inference models;
an explanation unit that outputs first explanation information corresponding to each of the plurality of inference models indicating the magnitude of contribution of the first input data to the first inference value;
an inference evaluation unit that obtains an inference evaluation result based on the correct value and the first inference value;
an explanation evaluation unit that obtains an explanation evaluation result based on a degree of matching between first explanation information corresponding to each of the plurality of inference models;
an updating unit that updates first weight parameters of the plurality of inference models based on the inference evaluation result and the explanation evaluation result;
A program that functions as a learning device.

Images