JP7504670B2 - Food Control System - Google Patents

Description

The technology disclosed in this specification relates to a food management system that manages food stored in a refrigerator.

Patent Document 1 discloses a refrigerator equipped with a camera that captures images of food stored inside the refrigerator. Image data of the food captured by the camera is sent to a server via the Internet. The server uses the image data received from the refrigerator to estimate the name of the captured food.

JP 2018-146177 A

Patent Document 1 does not give any consideration to improving the accuracy of estimating the food stored in a refrigerator used in a single household. This specification provides technology that can improve the accuracy of estimating the food stored in a refrigerator used in a single household.

The technology disclosed in this specification relates to a food management system that manages food stored in a refrigerator. The food management system may include an input unit to which food information, which is information indicating the type of food corresponding to the acquired target data, is input, a storage unit that stores related information in which the acquired target data or information based on the acquired target data is associated with the input food information, as for the specific refrigerator, and an estimation unit that, when the acquisition unit acquires specific target data while one or more pieces of related information are stored in the storage unit, analyzes the specific target data using the one or more pieces of related information stored in the storage unit and estimates specific food information corresponding to the specific target data.

The configuration of a food control system is shown. 13 is a flowchart of a terminal process executed by the terminal device. FIG. 11 is a sequence diagram of case A when a first user stores the food product "tangerines." FIG. 11 is a sequence diagram of case B when the first user additionally stores the food item "tangerine." FIG. 11 is a sequence diagram of case C when the first user stores the food product "persimmon." FIG. 13 is a sequence diagram of case D when a second user stores the food item "apple." FIG. 11 is a sequence diagram of case E when the first user stores the food item "apple."

(Example)
(Configuration of food control system 2)
As shown in Fig. 1, food management system 2 includes a refrigerator 10, a first terminal device 100, and a second terminal device 200. Refrigerator 10 is installed in house 4. First terminal device 100 and second terminal device 200 are devices used by residents of house 4. Refrigerator 10, first terminal device 100, and second terminal device 200 are capable of communicating with each other.

(Configuration of refrigerator 10)
The refrigerator 10 has storage compartments such as a refrigerator compartment, a freezer compartment, a vegetable compartment, an ice maker compartment, etc. (not shown). A plurality of foods are stored inside the refrigerator 10. For example, eggs, drinking water, etc. are stored in the refrigerator compartment of the refrigerator 10, and carrots, etc. are stored in the vegetable compartment. The refrigerator 10 cools the food ingredients stored inside. The refrigerator 10 is assigned a device ID "RF1" that identifies the refrigerator 10.

(Configuration of the first terminal device 100 and the second terminal device 200)
The first terminal device 100 is, for example, a portable terminal device such as a smartphone or a tablet terminal, and is a device separate from the refrigerator 10. In a modified example, the first terminal device 100 may be a device integrated with the refrigerator 10. The first terminal device 100 includes an operation unit 112, a display unit 114, a camera 116, a microphone 118, a communication interface (hereinafter, referred to as “I/F”) 120, a control unit 130, and a memory 132.

The operation unit 112 has a plurality of keys. A user can input various instructions to the first terminal device 100 by operating the operation unit 112. The display unit 114 is a display for displaying various information. The display unit 114 also functions as a so-called touch panel. In other words, the display unit 114 also functions as an operation unit.

The camera 116 is a device for photographing an object, and in this embodiment, is used in particular to photograph food stored inside the refrigerator 10. The microphone 118 is an interface for receiving sound wave signals. The communication I/F 120 is an interface for communicating with an external device (e.g., the second terminal device 200, etc.).

The control unit 130 controls the operation of the first terminal device 100 according to a program (not shown) stored in the memory 132, which is composed of a volatile memory, a non-volatile memory, etc. The memory 132 stores a management application (hereinafter, simply referred to as a "management app") 134 and an estimation table 136. The management app 134 is an application provided by the vendor of the refrigerator 10, and is installed in the first terminal device 100 from a server on the Internet, for example. The management app 134 is an application for managing food stored in the refrigerator 10. The first terminal device 100 stores the estimation table 136 in association with the device ID "RF1" of the refrigerator 10. That is, the control unit 130 stores the estimation table 136 as being in the refrigerator 10. The memory 132 also stores a storage list SL indicating the food stored in the refrigerator 10.

The operation unit 212, display unit 214, camera 216, microphone 218, communication I/F 220, control unit 230, memory 232, and management application 234 of the second terminal device 200 are similar to the operation unit 112, display unit 114, camera 116, microphone 118, communication I/F 120, control unit 130, memory 132, and management application 134 of the first terminal device 100, respectively. An estimation table 236 is stored in the memory 232. The second terminal device 200 stores the estimation table 136 in association with the device ID "RF1" of the refrigerator 10. In other words, the first terminal device 100 and the second terminal device 200 share the device ID "RF1" of the refrigerator 10.

(Explanation of Estimation Tables 136, 236)
The estimation tables 136, 236 are information used to estimate the food stored in the refrigerator 10 in the estimation process (S20, S54 in FIG. 2) described below. The estimation tables 136, 236 store one or more pieces of estimation information. The estimation information includes the name of the food and feature information. The feature information includes a shape feature, which is information that quantifies the shape of the food, and a color feature, which is information that quantifies the color of the food. Note that the feature information is different for the same food when the angle, brightness, etc. at which the food is imaged are different.

(Terminal processing; Figure 2)
The terminal processing executed by the control unit 130 of the first terminal device 100 in accordance with the management application 134 will be described with reference to Fig. 2. When the control unit 130 receives an operation for starting the management application 134, the control unit 130 causes the display unit 114 to display a top screen. The top screen is a screen for displaying a storage list screen showing a list of foods stored in the refrigerator 10 and for accepting an instruction to store foods in the refrigerator 10. When the control unit 130 receives a registration operation, which is an operation performed by a user when storing foods in the refrigerator 10, while the top screen is displayed, the control unit 130 starts the processing of Fig. 2. Note that when the control unit 130 receives a list selection operation, which is an operation for displaying the storage list screen on the first terminal device 100, while the top screen is displayed, the control unit 130 causes the display unit 114 to display the storage list screen without executing the processing of Fig. 2. Note that, although the information transmitted from the first terminal device 100 to an external device (for example, the second terminal device 200) always includes the device ID "RF1", in the following description and illustration of the device ID "RF1" will be omitted.

In S10, the control unit 130 activates the camera 116 and monitors whether an image capture operation has been accepted. If an image capture operation has been accepted, the control unit 130 judges YES in S10 and proceeds to S12. Note that if the control unit 130 judges YES in S10, it drives the camera 116 and acquires image data.

In S12, the control unit 130 performs image processing on the imaging data. Specifically, the control unit 130 analyzes the imaging data acquired in S10 and acquires feature information (hereinafter referred to as "target feature information") including shape features and color features corresponding to the food in the imaging data.

In S20, the control unit 130 executes an estimation process to estimate the food displayed in the imaging data using the target feature information and the estimation table 136 in the memory 132. Specifically, the control unit 130 identifies feature information that has a matching rate with the target feature information of a predetermined rate or more from among the feature information stored in the estimation table 136. In general, the matching rate of two feature information corresponding to the same food is high. However, even if there are two feature information corresponding to the same food, the matching rate is low when the angle or brightness at which the food is imaged is different, or when the image shown by one of the imaging data is unclear. Next, the control unit 130 identifies the food name corresponding to the identified feature information in the estimation table 136. When the control unit 130 identifies a plurality of feature information that has a matching rate with the target feature information of a predetermined rate or more, it identifies the food name corresponding to each of the feature information. Note that the control unit 130 does not identify (estimate) the food name if there is no feature information in the estimation table 136 that has a matching rate with the target feature information of a predetermined rate or more. In this case, the control unit 130 determines that the estimation process using the target feature information and the estimation table 136 has failed.

In S22, the control unit 130 determines whether or not a food name was identified in the estimation process of S20. If a food name was identified (YES in S22), the control unit 130 proceeds to S24. On the other hand, if a food name was not identified, i.e., if the estimation process failed (NO in S22), the control unit 130 proceeds to S50.

In S24, the control unit 130 causes the display unit 114 to display a selection screen including the food name identified in the estimation process in S20 and a cancel button. As shown in FIG. 4, the selection screen includes one or more food names identified (estimated) in the estimation process and a cancel button. In S26, the control unit 130 determines whether or not a food name in the selection screen has been selected. If a food name in the selection screen has been selected (YES in S26), the control unit 130 proceeds to S40. On the other hand, if a food name in the selection screen has not been selected, i.e., if the cancel button has been selected (NO in S26), the control unit 130 proceeds to S28.

In S28, the control unit 130 causes the display unit 114 to display an input screen. As shown in FIG. 3, the input screen includes a message for prompting the user to input the food name of the food that the user is going to store, a food name input field, and an OK button. In S30, the control unit 130 accepts an input operation for inputting the food name.

In S32, the control unit 130 stores inference information including the food name input in S30 and the target feature information identified in S12 in the inference table 136. This causes the inference information to be stored as information in the refrigerator 10.

In S40, the control unit 130 stores the food name selected in S26 or the food name input in S30 in a storage list SL in the memory 132. The control unit 130 also transmits the latest storage list SL to an external device. When S40 ends, the control unit 130 ends the processing of FIG. 2.

If the control unit 130 determines NO in S22 above, in S50, it transmits a data request to the external device. The data request is a signal requesting the transmission of the estimated information stored in the external device.

In S52, the control unit 130 determines whether or not estimated information has been received. If the control unit 130 receives estimated information from the external device (YES in S52), the control unit 130 proceeds to S54. On the other hand, if the control unit 130 does not receive estimated information from the external device even after a predetermined time (e.g., 5 seconds) has elapsed since sending the data request (NO in S52), the control unit 130 proceeds to S80.

In S54, the control unit 130 executes an estimation process using the target feature information and the estimation information received in S52. The content of this process is the same as the content of the estimation process in S20 described above.

In S56, the control unit 130 determines whether or not a food name was identified in the estimation process of S54. If a food name was identified (YES in S56), the control unit 130 proceeds to S60. On the other hand, if a food name was not identified (NO in S56), the control unit 130 proceeds to S80.

In S60, the control unit 130 causes the display unit 114 to display a selection screen including the food name identified in the estimation process in S54 and a cancel button. In S62, the control unit 130 determines whether or not a food name in the selection screen has been selected. If a food name in the selection screen has been selected (YES in S62), the control unit 130 proceeds to S70. On the other hand, if a food name in the selection screen has not been selected (NO in S62), the control unit 130 proceeds to S80.

In S70, the control unit 130 identifies the estimated information including the food name selected in S62 from the estimated information received in S52, and stores the estimated information in the estimation table 136. This causes the estimated information to be stored as information in the refrigerator 10. In S72, the control unit 130 stores the food name selected in S62 in the storage list SL in the memory 132. The control unit 130 also transmits the latest storage list SL to an external device. When S72 ends, the control unit 130 ends the processing in FIG. 2.

If the control unit 130 judges NO in S52, NO in S56, or NO in S62, it displays an input screen in S80. In S82, the control unit 130 accepts an input operation to input a food name.

In S90, the control unit 130 stores inference information including the food name input in S82 and the target feature information identified in S12 in the inference table 136. This causes the inference information to be stored as information for the refrigerator 10. In S92, the control unit 130 stores the food name input in S82 in the storage list SL in the memory 132. The control unit 130 also transmits the latest storage list SL to an external device. When S92 ends, the control unit 130 ends the processing of FIG. 2.

(Specific case)
Next, specific cases A to E executed by the process of FIG. 2 will be described with reference to FIGS.

(Case A; Figure 3)
With reference to Fig. 3, a process will be described in which a user (hereinafter, referred to as "first user") using the first terminal device 100 stores the food item "tangerines" inside the refrigerator 10. In the initial state of Fig. 3, the estimation table 136 of the first terminal device 100 and the estimation table 236 of the second terminal device 200 are empty. The interior of the refrigerator 10 is also empty.

When the first terminal device 100 accepts an application start operation at T10, it displays a top screen at T12 and accepts a registration operation at T14. Next, when the first terminal device 100 accepts an image capture operation of the food "mandarin orange" using the camera 116 at T16 (YES at S10), it captures the food "mandarin orange" at T18 and obtains image capture data ID1. Next, the first terminal device 100 performs image processing on the image capture data ID1 (S12), and at T20, it identifies feature amount information FI1 including the shape feature amount SF1 and color feature amount CF1 corresponding to the image capture data ID1. Next, the first terminal device 100 performs an estimation process using the feature amount information FI1 and the estimation table 136 in the memory 132 (S20). In this case, since no estimation information is stored in the estimation table 136, the first terminal device 100 does not identify the food name (NO at S22). In this case, the first terminal device 100 sends a data request to the second terminal device 200 at T22 (S50).

When the second terminal device 200 receives a data request from the first terminal device 100 at T22, it determines that the information in the estimation table 236 is empty. In this case, the second terminal device 200 does not transmit any estimation information.

The first terminal device 100 determines that it has not received estimated information from the second terminal device 200 even after a predetermined time has elapsed since sending the data request (NO in S52), and displays an input screen in T30 (S80). Next, the first terminal device 100 accepts an input operation to input the food name "mandarin orange" in T32 (S82), and stores estimated information EI1 including the food name "mandarin orange" and feature amount information FI1 in the estimation table 136 in T34 (S84). As a result, the estimated information EI1 is stored as being in the refrigerator 10. Next, the first terminal device 100 adds the food name "mandarin orange" to the storage list SL in T36 (S92), and transmits the latest storage list SL to the second terminal device 200 in T38 (S92).

When the second terminal device 200 receives the storage list SL from the first terminal device 100 at T38, it updates the storage list SL in the memory 232 using the received storage list SL at T40.

(Case B; Figure 4)
3, a process will be described in which the first user additionally stores the food "tangerines" in the refrigerator 10. In the initial state of case B, the estimation table 136 of the first terminal device 100 stores the estimation information EI1.

T110 to T118 are the same as T10 to T18 in case A. The first terminal device 100 performs image processing on the image data ID2 captured at T118 (S12), and at T120, identifies feature information FI2 including shape feature SF2 and color feature CF2 corresponding to the image data ID2. Next, the first terminal device 100 performs estimation processing using the feature information FI2 and the estimation table 136 in the memory 132 (S20). In case A and case B, the same user captures the food "tangerine", so the matching rate between the feature information FI2 and the feature information FI1 in the estimation table 136 is equal to or higher than a predetermined rate. Therefore, the first terminal device 100 identifies the food name "tangerine" associated with the feature information FI1 in the estimation table 136 as the food name corresponding to the feature information FI2. Next, the first terminal device 100 determines that the food name "mandarin orange" has been identified (YES in S22), and in T122, displays a selection screen including the food name "mandarin orange" (S24), and in T124, accepts the selection of the food name "mandarin orange" (YES in S26). T126 to T130 are the same as T36 to T40.

(Case C; Figure 5)
Next, a process in which the first user adds the food item "persimmon" to the interior of the refrigerator 10 will be described with reference to Fig. 5. Case C is the state after case B in Fig. 4. Therefore, the estimation table 136 of the first terminal device 100 stores the estimation information EI1.

T210 to T216 are the same as T10 to T16 in FIG. 3. T218 is the same as T18, except that the food to be imaged is a "persimmon". The first terminal device 100 performs image processing on the image data ID3 imaged in T218 (S12 in FIG. 2), and in T220, identifies feature information FI3 including shape feature SF3 and color feature CF3 corresponding to the image data ID3. Next, the first terminal device 100 performs estimation processing using the feature information FI3 and the estimation table 136 in the memory 132 (S20). The food "mandarin orange" and "persimmon" are similar in shape and color. Therefore, the first terminal device 100 determines that the matching rate between the feature information FI1 and the feature information FI3 is equal to or higher than a predetermined rate, and identifies the food name "mandarin orange" associated with the feature information FI1 as the food name corresponding to the feature information FI3. Next, the first terminal device 100 determines that the food name "mandarin orange" has been identified (YES in S22), and displays a selection screen including the food name "mandarin orange" in T222 (S24). When the selection of the cancel button is accepted in T224 (NO in S26), an input screen is displayed in T226 (S28). Next, when the first terminal device 100 accepts the input of the food name "persimmon" in T228 (S30), in T230, the estimated information EI3 including the food name "persimmon" and the feature amount information FI3 is stored in the estimation table 136 (S32). As a result, the estimated information EI3 is stored as being in the refrigerator 10. Next, the first terminal device 100 adds the food name "persimmon" to the storage list SL in T232 (S40). T234 and T236 are the same as T38 and T40.

(Effect of Case C)
When the amount of estimated information stored in the memory 132 of the first terminal device 100 is small, the accuracy of the food name estimation in the estimation process is low. As shown in case C, when the food name "mandarin orange" is estimated as the food name corresponding to the imaging data ID3, and then the food name "persimmon" different from the food name "mandarin orange" is input (T228), the first terminal device 100 stores the estimated information EI3 in which the feature amount information FI3 and the food name "persimmon" are associated as the food name in the refrigerator 10 in the memory 132. According to the above configuration, when the estimation of the food name by the first terminal device 100 is corrected, the first terminal device 100 stores the estimated information EI3 including the input food name "persimmon" as the food name in the refrigerator 10. The input food name "persimmon" is correct information. Therefore, the first terminal device 100 can use the estimated information EI3 in the subsequent estimation process, and as a result, the accuracy of the food name estimation in the estimation process can be improved.

(Case D; Figure 6)
Next, with reference to Fig. 6, a process in which a user (hereinafter, referred to as "second user") using the second terminal device 200 stores the food item "apples" in the refrigerator 10 will be described. Case D is the state after case C in Fig. 5. Therefore, the estimation table 136 of the first terminal device 100 stores the estimation information EI1 and EI3, and the estimation table 236 of the second terminal device 200 is empty. Note that the control unit 230 of the second terminal device 200 can execute the same process as the terminal process in Fig. 2 according to the management application 234.

T310 to T316 are the same as T10 to T16 in FIG. 3, except that the target device is the second terminal device 200. T318 is the same as T318, except that the imaged object is the food "apple" and the target device is the second terminal device 200. The second terminal device 200 performs image processing on the image data ID4 captured in T318, and in T320, identifies feature information FI4 including shape feature SF4 and color feature CF4 corresponding to the image data ID4. Next, the second terminal device 200 performs an estimation process using the feature information FI4 and the estimation table 236 in the memory 232. In this case, since feature information is not stored in the estimation table 236, the second terminal device 200 does not identify the food name. In this case, the second terminal device 200 sends a data request to the first terminal device 100 at T322.

When the first terminal device 100 receives a data request from the second terminal device 200 at T322, it identifies the estimated information EI1 and EI3 in the estimation table 136, and at T324, it transmits the estimated information EI1 and EI3 to the second terminal device 200.

When the second terminal device 200 receives the estimated information EI1 and EI3 from the first terminal device 100 in T324, it executes an estimation process using the feature information FI4 and the estimated information EI1 and EI3. The second terminal device 200 determines that the matching rate between the feature information FI4 and the feature information FI1 in the estimated information EI1, and the matching rate between the feature information FI4 and the feature information FI3 in the estimated information EI3 are less than a predetermined rate, and does not identify the food name. In this case, the second terminal device 200 displays an input screen in T330. Next, when the second terminal device 200 accepts input of the food name "apple" in T332, it stores the estimated information EI4 including the food name "apple" and the feature information FI4 in the estimation table 236 in T334. Next, in T336, the second terminal device 200 adds the food name "apple" to the storage list SL, and in T338, transmits the latest storage list SL to the first terminal device 100.

When the first terminal device 100 receives the storage list SL from the second terminal device 200 at T338, it updates the storage list SL in the memory 132 at T340 using the received storage list SL.

(Case E; Figure 7)
Next, a process in which the first user stores the food item "apples" in the refrigerator 10 will be described with reference to Fig. 7. This case is the state after case D in Fig. 6. That is, the estimation table 136 of the first terminal device 100 stores estimated information EI1 and EI3, and the estimation table 236 of the second terminal device 200 stores estimated information EI4.

T410 to T416 are the same as T10 to T16 in FIG. 3. T418 is the same as T18, except that the food to be imaged is an "apple." The first terminal device 100 performs image processing on the image data ID5 imaged in T418 (S12 in FIG. 2), and in T420, identifies feature information FI5 including shape feature SF5 and color feature CF5 corresponding to the image data ID5. Next, the first terminal device 100 performs estimation processing using the feature information FI5 and the estimation table 136 in the memory 132 (S20), and identifies that the matching rate between the feature information FI5 and the feature information FI1 in the estimated information EI1, and the matching rate between the feature information FI5 and the feature information FI3 in the estimated information EI3 are less than a predetermined rate, and does not identify the food name (NO in S22). In this case, the first terminal device 100 transmits a data request to the second terminal device 200 at T422 (S50), and receives estimated information EI4 from the second terminal device 200 at T424 (YES at S52).

Next, the first terminal device 100 executes an estimation process using the feature amount information FI5 and the estimated information EI4 (S54). The first terminal device 100 determines that the matching rate between the feature amount information FI5 and the feature amount information FI4 in the estimated information EI4 is equal to or higher than a predetermined rate, and identifies the food name "apple" in the estimated information EI4 as the food name corresponding to the feature amount information FI5. Next, the first terminal device 100 determines that the food name "apple" has been identified (YES in S56), and displays a selection screen including the food name "apple" in T430 (S60). Next, when the first terminal device 100 accepts the selection of the food name "apple" in T432 (YES in S62), it stores the estimated information EI4 in the estimation table 136 in T434 (S70), and adds the food name "apple" to the storage list SL in T436 (S72). As a result, the estimated information EI4 is stored as information for the refrigerator 10. T438 and T440 are similar to T38 and T40 in FIG. 3. In a modified example, the first terminal device 100 may store estimated information including the food name "apple" and feature amount information FI5 in the estimation table 136 in T434.

(Effect of Case E)
As shown in Case E, in a state in which estimated information EI4 including feature amount information FI4 based on imaging data ID4 previously acquired by the second terminal device 200 is stored, the first terminal device 100 uses the estimated information EI4 when estimating a food name corresponding to imaging data ID5 acquired by the first terminal device 100. With the above configuration, even if the food has not been imaged by the first terminal device 100 in the past, the food name of the food can be estimated.

In addition, in a state in which estimated information EI1, EI3 including feature information FI1, FI3 based on imaging data ID1, ID3 previously acquired by the first terminal device 100 and estimated information EI4 including feature information FI4 based on imaging data ID4 previously acquired by the second terminal device 200 are stored, the first terminal device 100 uses estimated information EI1, EI3 preferentially over estimated information EI4 when estimating a food name corresponding to imaging data ID5 acquired by the first terminal device 100. In this embodiment, using estimated information EI, EI3 first and then using estimated information EI4 is an example of "using estimated information EI1, EI3 preferentially over estimated information EI4". In addition, in a modified example, for example, using the estimated information EI1, EI3, and EI4 so that the influence of the estimated information EI1 and EI3 is greater than that of the estimated information EI4 may be an example of "using the estimated information EI1 and EI3 preferentially over the estimated information EI4". The food that the first user is about to store in the refrigerator 10 is likely to be food that the user has stored in the refrigerator 10 in the past. Therefore, it is likely that the name of the food corresponding to the imaging data ID5 acquired by the first terminal device 100 can be estimated by using the estimated information EI1 and EI3 including the feature amount information FI1 and FI3 based on the imaging data ID1 and ID3 acquired by the first terminal device 100 in the past. According to the above configuration, when estimating the name of a food item corresponding to imaging data ID5 acquired by the first terminal device 100, the processing load on the first terminal device 100 can be reduced compared to a configuration in which estimated information including feature information based on imaging data previously acquired by the second terminal device 200 is used preferentially over estimated information including feature information based on imaging data previously acquired by the first terminal device 100.

In addition, in the estimation of the food name corresponding to the imaging data ID5 acquired by the first terminal device 100, when the estimation using the estimated information EI1, EI3 including the feature amount information FI1, FI3 based on the imaging data ID1, ID3 acquired by the first terminal device 100 in the past fails, and the estimation using the estimated information EI4 including the feature amount information FI4 based on the imaging data ID4 acquired by the second terminal device 200 in the past is successful, the first terminal device 100 increases the influence of the estimated information EI4 in the estimation of the food name corresponding to the imaging data acquired by the first terminal device 100. In this embodiment, the first terminal device 100 increases the influence of the estimated information EI4 by storing the estimated information EI4 (T434 in FIG. 7). With this configuration, the first terminal device 100 can use the estimated information EI4 without receiving the estimated information EI4 from the second terminal device 200, so that the influence of the estimated information EI4 is increased. According to the above configuration, when the food "apple" is captured by the first terminal device 100 after the influence of the estimated information EI4 becomes large, the first terminal device 100 can estimate "apple" as the food name corresponding to the captured image data, even without receiving the estimated information EI4 from the second terminal device 200. Therefore, the estimation accuracy of the captured image data acquired by the first terminal device 100 can be improved.

(Effects of this embodiment)
As described above, when the first terminal device 100 acquires imaging data (YES in S10 of FIG. 2) and a food name corresponding to the imaging data is input (S30, S82), the first terminal device 100 stores in the memory 132 estimated information in which feature information based on the imaging data and the input food name are associated as the food name in the refrigerator 10 (S32, 90). Then, when the first terminal device 100 acquires imaging data in a state in which one or more pieces of estimated information are stored, the first terminal device 10 analyzes the imaging data using the estimated information stored in the memory 132 and estimates the food name corresponding to the imaging data (S20). According to the above configuration, the estimated information is accumulated in the memory 132 as the food name in the refrigerator 10. There is a high possibility that the food stored in the refrigerator 10 by the user who uses the refrigerator 10 is the food that the user stored in the refrigerator 10 in the past. Therefore, the estimated information is accumulated in the memory 132, and the accuracy of estimation of the food stored in the refrigerator 10 by the user who uses the refrigerator 10 can be improved. Therefore, the accuracy of estimating the food stored in the refrigerator 10 used in the house 4 can be improved, and the convenience of the user living in the house 4 can be improved.

(Correspondence)
The imaging data is an example of "target data in a predetermined format". The camera 116 of the first terminal device 100 and the camera 216 of the second terminal device 200 are an example of an "acquisition unit". The display unit 114 of the first terminal device 100 and the display unit 214 of the second terminal device 200 are an example of an "input unit". The memory 132 of the first terminal device 100 and the memory 232 of the second terminal device 200 are an example of a "storage unit". The food name is an example of "food information". The estimated information is an example of "related information". The imaging data ID1 to ID5 are an example of "specific target data". The device ID "RF1" is an example of "identification information". S20 and S54 in FIG. 2 are an example of processing executed by the "estimation unit".

The imaging data ID4 and estimated information EI4 are examples of "first target data" and "first related information". The imaging data ID1, ID3, and estimated information EI1, EI3 are examples of "second target data" and "second related information", respectively. The imaging data ID4 and estimated information EI4 are examples of "third target data" and "third related information". The imaging data ID1, ID3, and estimated information EI1, EI3 are examples of "fourth target data" and "fourth related information", respectively. The imaging data ID4 and estimated information EI4 are examples of "fifth target data" and "fifth related information".

Specific examples of the present invention have been described in detail above, but these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples given above.

(First modified example) The "target data in a specified format" is not limited to imaging data, and may be audio data. In this modified example, the memory 132 of the first terminal device 100 stores estimated information including the food name and audio features (an example of "data-related information") obtained by analyzing the audio data. In this modified example, the microphone 118 of the first terminal device 100 and the microphone 218 of the second terminal device 200 are examples of an "acquisition unit."

(Second variant) In the estimation table 136, food names may be associated with imaging data.

(Third Variation) The refrigerator 10 may be equipped with a camera. In this variation, the first terminal device 100 executes the process from S12 onward in FIG. 2 when acquiring image data captured by the camera of the refrigerator 10 from the refrigerator 10.

(Fourth variant) S32 in FIG. 2 can be omitted.

(Fifth Modification) The estimated information in the memory 132 of the first terminal device 100 and the estimated information in the memory 232 of the second terminal device 200 do not have to be shared between the first terminal device 100 and the second terminal device 200. In this modification, steps S50 to S72 in FIG. 2 can be omitted.

(Sixth Modification) The first terminal device 100 may execute S50 and S52 before S20. In this modification, the first terminal device 100 preferentially uses the estimation information in the memory 232 of the second terminal device 200 over the estimation information in the memory 132 of the first terminal device 100 in the estimation process of S20.

(Seventh variant) S70 in FIG. 2 can be omitted.

(Eighth Variation) The food management system 2 may include a server. In this variation, an estimation table is stored in the memory of the server. The server is configured to execute an estimation process when imaging data is received from the first terminal device 100 or the second terminal device 200. In this variation, the server may store information identifying the first terminal device 100 and the second terminal device 200 in order to identify the information received from the first terminal device 100 and the second terminal device 200. In this variation, the memory of the server is an example of a "storage unit". In this variation, the estimation process executed by the server is an example of a process executed by an "estimation unit".

The technical elements described in this specification or drawings have technical utility either alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. In addition, the technologies illustrated in this specification or drawings can achieve multiple objectives simultaneously, and achieving one of those objectives is itself technically useful.

2: Food management system, 4: House, 10: Refrigerator, 100: First terminal device, 112: Operation unit, 114: Display unit, 116: Camera, 118: Microphone, 120: Communication I/F, 130: Control unit, 132: Memory, 134: Management application, 136: Estimation table, 200: Second terminal device, 212: Operation unit, 214: Display unit, 216: Camera, 218: Microphone, 220: Communication I/F, 230: Control unit, 232: Memory, 234: Management application, 236: Estimation table

Claims (4)

A food management system for managing food stored in a refrigerator, comprising:
An acquisition unit that acquires target data in a predetermined format;
an input unit to which food information indicating a type of food corresponding to the acquired target data is input;
A storage unit that stores associated information in which the acquired target data or information based on the acquired target data is associated with the input food information as information for a specific refrigerator;
an estimation unit that, when the acquisition unit acquires specific target data in a state in which the storage unit stores one or more pieces of related information, analyzes the specific target data by using the one or more pieces of related information stored in the storage unit, and estimates specific food information corresponding to the specific target data;
Equipped with
The food management system includes a plurality of terminal devices,
Each of the plurality of terminal devices shares identification information for identifying the specific refrigerator,
the estimation unit uses the first related information when estimating the specific food information corresponding to the specific target data acquired by the first terminal device in a state in which first target data previously acquired by the second terminal device or first related information including information based on the first target data is stored in the storage unit;
Food control system. The food management system of claim 1, which determines that estimation of food information corresponding to the specific target data has failed when food information different from the specific food information is input to the input unit after the specific food information corresponding to the specific target data has been estimated, and stores in the storage unit specific related information in which the specific target data or information based on the specific target data is associated with the different food information as information for the specific refrigerator. 3. The food management system according to claim 1 or 2, wherein when estimating the specific food information corresponding to the specific target data acquired by the first terminal device in a state in which second related information including second target data previously acquired by the first terminal device or information based on the second target data, and third related information including third target data previously acquired by the second terminal device or information based on the third target data are stored, the estimation unit preferentially uses the second related information over the third related information when estimating the specific food information corresponding to the specific target data acquired by the first terminal device. 4. The food management system according to claim 1, wherein when an estimation using a fourth target data previously acquired by the first terminal device or a fourth related information including the fourth target data fails in estimating the specific food information corresponding to the specific target data acquired by the first terminal device, and an estimation using a fifth target data previously acquired by the second terminal device or a fifth related information based on the fifth target data is successful, the estimation unit increases the influence of the fifth related information in estimating the food information corresponding to the target data acquired by the first terminal device.

Images