KR102557092B1 - Automatic interpretation and translation and dialogue assistance system using transparent display - Google Patents

Abstract

The present invention relates to an automatic interpretation and translation and dialogue assistance system, and specifically, to an automatic interpretation and translation and dialogue assistance system using a transparent display which interprets and translates dialogues bilaterally recognized to provide a dialogue text constituting a preset pair of languages. To this end, the automatic interpretation and translation and dialogue assistance system using a transparent display comprises: a recognition unit which classifies a voice in a first language spoken in one direction and a voice in a second language spoken in the other direction for each speaker; a text generation unit which generates a sentence text for each time when analyzing the voices in the first and second languages through deep learning-based artificial intelligence and creates a dialogue window list based on the same; a translation derivation unit which derives first and second translated texts when any one among a preset rule-based, pattern-based, statistics-based, and artificial intelligence-based translators is embedded therein, and the dialogue window list is individually translated in preset languages by using the translator; and an integrated control unit which, based on time and speaker information checked from the dialogue window list, sequentially selects a pair of subtitles for output from the first and second translated texts to display the same in different output directions through the transparent display.

Description

Automatic translation and conversation assistance system using a transparent display {AUTOMATIC INTERPRETATION AND TRANSLATION AND DIALOGUE ASSISTANCE SYSTEM USING TRANSPARENT DISPLAY}

The present invention relates to an automatic interpretation/translation and dialogue assistance system, and more particularly to an automatic interpretation/translation and dialogue assistance system using a transparent display capable of interpreting and translating a conversation recognized in both directions to provide dialogue text composed of a pair of predetermined languages. .

As the opportunity for communication has expanded globally, the need for translation is becoming more and more important day by day.

In particular, as foreign tourism and migration, which have been stagnant since the pandemic, have increased responses to foreigners in government offices, banks, hotels, and other institutions in each country, many complain of difficulties in communication.

In the currently released automatic interpretation and translation devices, when you have a conversation with a foreigner, the device that translates the native language into the foreign language and the device that translates the foreign language into the native language work independently, so that the foreigner's conversation is translated into the native language and your own conversation into the foreign language. By performing independent translation, the translation target sentence is independently translated regardless of the conversation content of the other party.

Accordingly, an object of the present invention is to provide an automatic interpretation/translation and conversation assistance system using a transparent display capable of simultaneously communicating in both directions off-line.

(0001) Korean Patent Publication No. 10-2002-0076044 (2002.10.09)

The present invention has been made to solve the above problems, and an object of the present invention is to provide an automatic interpretation/translation and conversation assistance system capable of interpreting and translating a conversation recognized in both directions and outputting a conversation text composed of a pair of predetermined languages in real time. is to provide

In addition, it is to provide an automatic interpretation/translation and dialogue assistance system capable of adjusting the page switching speed of the next subtitle output after a pair of output subtitles through a speaker's transparent display.

The above and other objects and advantages of the present invention will become apparent from the following description of preferred embodiments.

In order to achieve the above object, an automatic interpretation/translation and conversation assistance system using a transparent display according to an embodiment of the present invention separates the voice of a first language spoken in one direction and the voice of a second language spoken in another direction for each speaker. A recognition unit that classifies and individually recognizes, a text generator that analyzes the voices of the first and second languages through artificial intelligence based on deep learning to generate time-specific sentence text and based on this, a text generator that creates a dialog list, Any one of rule-based, pattern-based, statistical-based, and artificial intelligence-based translators are embedded, and translation derivation for deriving first and second translated texts by individually translating the chat window list into preset languages using the embedded translator And an integrated control unit that sequentially selects a pair of subtitles for output from the first and second translated texts and outputs them in different output directions through a transparent display based on the time and speaker information identified from the window list and the dialog window list. , The pair of subtitles for output includes a first subtitle for output displayed in the first language and a second subtitle for output displayed in the second language and translated to have the same meaning.

In an embodiment, the translator includes a plurality of translators customized for each field, and the plurality of translators customized for each field are translators optimized by removing unnecessary words according to occupation, age, region, field of interest, and religion.

In an embodiment, the recognizing unit includes at least one directional microphone for selectively recognizing spoken voice uttered in a specific direction, at least one input device for receiving input text to replace the utterance voice, and toward the specific direction. At least one camera that captures and acquires a pair of speaker images, a detection module that detects a preset facial expression and gesture object from any one of the pair of speaker images, and a sign language text corresponding to the gesture object and the preset facial expression. and a text extraction module that extracts the emergency guidance text from a memory and an identification module that assigns an identification code indicating a speech time and speaker direction to the spoken voice, the input text, and the sign language text and stores them in the memory.

In an embodiment, the text generation unit learns the vocabulary word dictionary previously collected in the memory to model a deep learning-based speech recognition model, and the first modeling unit learns the pronunciation rules previously collected in the memory to obtain rules and statistics. A second modeling unit that models the based acoustic model, a signal processing unit that converts the voices of the first and second languages into input signals according to voice recognition feature vectors, and converts the input signals into the voice recognition model and the acoustic model. and a decoder for outputting character information as text for each speaker and merging the input text and the sign language text with the text for each speaker according to the identification code.

In an embodiment, the integrated control unit compares voice characteristics analyzed from each of the voices of the first and second languages recognized through the recognition unit with a preset reference sample, and a comparison result confirmed through the comparison unit. and a voice corrector for correcting each voice of the first and second languages based on the voice characteristics of the reference sample.

In an embodiment, the voice correction unit generates a noise canceling signal according to a noise signal identified from the silence signal recognized through the recognition unit, and filters noise for the first and second language voice signals based on the noise canceling signal. and a content providing unit for temporarily pausing the output of the pair of output subtitles through the transparent display and replacing and outputting a healing content image for changing the atmosphere when the mute signal and the mute signal last for a certain period of time, and the mute The signal includes a repetitive noise signal of a specific frequency band recognized through the recognition unit while the volume of the frequency band corresponding to the voice of the first and second languages is not recognized for a predetermined time.

In an embodiment, the integrated control unit adjusts the output area of the pair of subtitles for output on the transparent display to different area ratios based on distance information for each speaker measured through a distance measuring sensor disposed on the transparent display. do.

In an embodiment, the integrated control unit estimates the visual acuity of a speaker based on a feedback response signal as a simple visual acuity test table is output prior to outputting the pair of subtitles for output, and each font for the pair of subtitles for output is based on this estimate Automatically adjusts the zoom ratio.

In an embodiment, the integrated control unit adjusts a page switching speed of the next subtitle output after the pair of output subtitles based on eye movement data measured through an eye tracker module disposed on the transparent display.

According to an embodiment of the present invention, by interpreting and translating a conversation recognized in both directions and outputting a dialogue text composed of a pair of predetermined languages in real time, communication according to interpretation and translation can proceed more quickly.

In addition, communication can be more smoothly supported by adjusting the page conversion speed of the next subtitle output after a pair of output subtitles through the speaker's transparent display.

1A is a diagram schematically illustrating an automatic translation and conversation assistance system 1000 according to an embodiment of the present invention.
FIG. 1B is pictures showing an embodiment of the automatic translation and conversation assistance system 1000 of FIG. 1A.
FIG. 2 is a block diagram showing the recognition unit 100 of FIG. 1 in detail.
FIG. 3 is a block diagram of the text generator 200 of FIG. 1 according to an embodiment.
FIG. 4 is a block diagram of the integrated control unit 400 of FIG. 1 according to an embodiment.

Hereinafter, the present invention will be described in detail with reference to embodiments and drawings of the present invention. These examples are only presented as examples to explain the present invention in more detail, and it will be apparent to those skilled in the art that the scope of the present invention is not limited by these examples. .

In addition, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of skill in the art to which this invention belongs, and in case of conflict, this specification including definitions of will take precedence.

In order to clearly explain the proposed invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification. And, when a certain component is said to "include", this means that it may further include other components without excluding other components unless otherwise stated. Also, a “unit” described in the specification means one unit or block that performs a specific function.

In each step, the identification code (first, second, etc.) is used for convenience of description, and the identification code does not describe the order of each step, and each step does not clearly describe a specific order in context. It may be performed differently from the order specified above. That is, each step may be performed in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order. 1A is a diagram schematically illustrating an automatic translation and conversation assistance system 1000 according to an embodiment of the present invention, and FIG. 1B is pictures showing an embodiment of the automatic translation and conversation assistance system 1000 of FIG. 1A.

Referring to FIGS. 1A and 1B , the automatic translation and conversation assistance system 1000 may include a recognition unit 100, a text generator 200, a translation derivation unit 300, and an integrated control unit 400. there is.

First, the recognition unit 100 may separately recognize the voice of a first language spoken in one direction and the voice of a second language spoken in another direction by speaker.

Next, the text generation unit 200 analyzes the voices of the first and second languages sequentially recognized through the recognition unit 100 through a deep learning-based artificial intelligence model, thereby generating time-specific sentence text and converting it to Based on this, you can create a dialog list.

Here, the chat window list may be a chat window in which sentence texts by time between a speaker located in one direction and a speaker located in another direction are distinguished from each other.

In this case, the deep learning-based artificial intelligence model may be any one algorithm of an artificial neural network, a support vector machine (SVM), a decision tree, and a random forest. For example, an artificial neural network is a statistical learning algorithm that is mainly used in deep learning and is inspired by neural networks in machine learning and biology, and may be a convolutional neural network including a feature extraction neural network and a classification neural network.

Next, the translation derivation unit 300 uses any one of rule-based machine translation (RBMT), pattern-based, statistical machine translation (SMT), and artificial intelligence-based (neuronal machine translation: NMT). One translator is embedded, and the first and second translated texts may be derived by individually translating the sentence text for each speaker detected in the chat window list into a preset language by using the embedded translator.

Here, the translator uses preset rule-based machine translation (RBMT), pattern-based, statistical machine translation (SMT) and artificial intelligence-based (neuronal Morphological dictionary data, grammar dictionary data, sentence pattern dictionary data, phase dictionary data, and parallel corpus data required for machine translation (NMT) translation technology may be registered in a memory (not shown) in advance.

Such a translator may include a plurality of field-specific customized translators. Specifically, the translators customized for each field may be translators from which unnecessary words are removed according to occupation, age, region, field of interest, or religion. For example, if the user's job is a doctor, the translation derivation unit 300 may use a translator optimized for medical terms, and if the user's age is an infant, a translator optimized for neologism terms may be used.

According to an embodiment, the first and second translated texts may be obtained by individually translating sentence texts for each speaker detected in the chat window list into a preset language through a domain-specialized translation technology.

Here, the domain-specialized translation technology can machine-translate each speaker's sentence text into a preset language through a transformer-based encoding and decoding algorithm.

Next, the integrated control unit 400 selects a pair of subtitles for output from the first and second translation texts based on the time and speaker information checked from the dialog window list and outputs them in different output directions through the transparent display 10. can make it

Here, a pair of subtitles for output are sentence texts translated so that their meanings correspond to each other. A first subtitle for output displayed in a first language to be provided to a speaker located on one side and a second subtitle displayed in a second language to be provided to a speaker located on the other side 2 Subtitles for output may be included.

According to an embodiment, the integrated control unit 400 outputs a pair of subtitles for the transparent display 10 based on distance information for each speaker measured through a distance measuring sensor (not shown) disposed on the transparent display 10. Each output area of can be adjusted to different area ratio.

For example, when the distance information of a speaker located on one side is 80 cm and the distance information of a speaker located on the other side has a certain distance difference of 40 cm, the integrated control unit 400 determines the output area of the first subtitle for output among a pair of subtitles for output. and the output area of the second output caption may be increased.

According to another embodiment, the integrated control unit 400 is based on the psychological state of each speaker analyzed according to the respiratory and heart rate signals of each speaker measured through a laser sensor (not shown) disposed on the transparent display 10, Captions for output can be corrected with color therapy colors.

For example, if the psychological state is lethargic, the color therapy color is red, if the psychological state is sadness and loss, the color therapy color is orange, and if the psychological state is insomnia, fatigue, or headache, the color therapy color is blue, and the psychological state is blue. If the state is unstable, the color therapy color may be green, and if the psychological state is boring or boring, the color therapy color may be yellow.

According to another embodiment, the integrated control unit 400 outputs a simple visual acuity test table before outputting a pair of captions for output, so as to estimate the visual acuity of a speaker based on a feedback response signal, and based on this estimate the visual acuity of a speaker, to output a pair of captions for output. The magnification ratio of each font can be automatically adjusted.

According to another embodiment, the integrated control unit 400 outputs a pair of subtitles through the transparent display 10 based on eye movement data measured through an eye tracker module (not shown) disposed on the transparent display 10. The page switching speed of the next subtitle that is output later can be adjusted.

For example, when eye movement data is equal to or greater than a predetermined speed, the integrated controller 400 may increase the page switching speed of the next subtitle output after a pair of output subtitles through the transparent display 10 . In addition, when the eye movement data is less than a predetermined speed, the integrated controller 400 may decrease the page switching speed of the next subtitle output after a pair of output subtitles through the transparent display 10 .

According to another embodiment, the integrated control unit 400 registers SNS account information received through at least one input device 120 to be described in FIG. By using the chat window list, it is possible to transmit the chat window list as an image file to the speaker terminal 20 corresponding to the SNS account information.

According to another embodiment, the integrated control unit 400 outputs the first translated text through the transparent display 10 based on the first and second translated texts collected in a memory (not shown) so as to perform the second translation. It is possible to provide a learning service mode that requests text in any one of voice, sign language, and text.

According to another embodiment, the integrated control unit 400 issues a disaster alert through the transparent display 10 based on a disaster environmental state detected through environmental sensors (not shown) disposed on the transparent display 10. can be printed out. Here, the environmental sensors (not shown) may include a smoke sensor, a carbon monoxide sensor, and a temperature sensor.

Hereinafter, the configuration of the present invention and its effects will be described in more detail through specific examples and comparative examples. However, these examples are for explaining the present invention in more detail, and the scope of the present invention is not limited to these examples.

FIG. 2 is a block diagram showing the recognition unit 100 of FIG. 1 in detail.

1A to 2, the recognition unit 100 includes at least one directional microphone 110, at least one input device 120, at least one camera 130, a detection module 140, and a text extraction module. (150) and an identification module (160).

First, at least one directional microphone 110 is disposed on the transparent display 10 and can selectively recognize an utterance spoken in a specific direction spaced apart from the transparent display 10 by a certain distance.

Next, at least one input device 120 is arranged to be spaced apart from the transparent display 10 by a predetermined distance, and can receive input text to replace the spoken voice.

Next, at least one camera 130 may acquire a pair of speaker images by photographing in a specific direction.

For example, at least one directional microphone 110 and at least one camera 130 are individually disposed on one side and the other side of the transparent display 10, and at least one input device 120 covers both sides. It may be arranged at a location spaced apart from a predetermined distance as a reference.

Next, the detection module 140 may detect a preset facial expression and gesture object from any one of a pair of speaker images captured through at least one camera 130 .

Next, the text extraction module 150 may extract and recognize the preset emergency guidance text according to the sign language text corresponding to the gesture object and the preset facial expression from a memory (not shown).

Next, the identification module 160 uses the speech voice recognized through at least one directional microphone 110, the input text received through at least one input device 120, and the sign language extracted through the text extraction module 150. An identification code indicating a conversation time and a speaker direction may be given to the text for storage and classification in a memory (not shown).

The identification module 160 may classify voices of the first and second languages according to identification codes, and sequentially input the sentences to the deep learning-based artificial intelligence model in the text generator 200 in units of sentences.

FIG. 3 is a block diagram of the text generator 200 of FIG. 1 according to an embodiment.

Referring to FIGS. 1A to 3 , the text generator 200 may include first and second modeling units 210 and 220 , a noise processing unit 230 and a decoder 240 .

First, the first modeling unit 210 may model a deep learning-based speech recognition model by learning vocabulary words previously collected in a memory (not shown). Here, the deep learning-based speech recognition model may be an algorithm derived through either Google's Deep Mind or IBM's TrueNorth, which learns pre-collected vocabulary words.

Next, the second modeling unit 220 may model a rule- and statistical-based acoustic model by learning pronunciation rules previously collected in a memory (not shown). Here, the rule- and statistics-based acoustic model may be an algorithm derived through IBM's Watson program that learns pre-collected pronunciation rules.

Next, the signal processing unit 230 may convert the speech of the first and second languages received for each sentence through the identification module 160 into an input signal according to the speech recognition feature vector.

Here, the speech recognition feature vector may include any one of various feature vectors such as mel frequency cepstral coefficient (MFCC), linear predictive coefficients (LPC), and perceptual linear prediction (PLP).

In this case, the input signal may mean a factor input to a deep learning-based speech recognition model and a rule- and statistics-based acoustic model.

Next, the decoder 240 applies the input signal converted and processed through the signal processing unit 330 to a deep learning-based speech recognition model and a rule- and statistics-based acoustic model, thereby outputting character information as text for each speaker. there is.

In this case, the decoder 240 may sequentially list and merge the input text and the sign language text into the text for each speaker according to the identification code checked in the memory (not shown).

FIG. 4 is a block diagram of the integrated control unit 400 of FIG. 1 according to an embodiment.

Referring to FIGS. 1A to 4 , the integrated control unit 400 may include a comparison unit 410 and a voice correction unit 420 .

First, the comparator 410 may compare voice characteristics analyzed from each voice of the first and second languages recognized through the recognizer 100 with a preset reference sample.

Here, the reference sample means a sample voice enabling accurate voice recognition, and the voice characteristics may include any one of a voice rate per syllable, a silence time during voice, and a formant frequency of voice.

At this time, the voice correction unit 420 corrects each voice of the first and second languages according to the voice characteristics of the reference sample based on the comparison result confirmed through the comparison unit 410, and feeds back to the recognition unit 100. can provide

For example, when the rate of speech per syllable is out of the critical range corresponding to the reference sample, the voice correction unit 220 may correct the rate of speech of the first and second languages to the rate of speech according to the reference sample. .

According to an embodiment, the voice compensator 420 integrates the audio visual information obtained by frequency-analyzing the voice signals of the first and second languages recognized through the recognition unit 100 into a pair of subtitles for output so as to be transparent. By outputting through the display 10, it is possible to induce speech at an optimal volume level of the speaker's voice.

According to another embodiment, the voice compensator 420 is based on the volume level of a band other than the audible frequency of the voice recognized through the recognition unit 100, for example, ultrasonic waves of 1000,000 hz or more and infrasonic waves of 7 hz or less, It may be controlled to amplify a predetermined audible frequency band for the voice signals of the first and second languages.

The voice correction unit 420 may further include a filtering unit 421 and a content providing unit 422 . Specifically, the filtering unit 421 generates a noise canceling signal according to the noise signal identified from the silence signal recognized through the recognition unit 100, and filters the noise of the first and second language voice signals based on the generated noise signal. can do. Here, the silence signal may include a repetitive noise signal of a specific frequency band recognized through the recognition unit 100 while the volume of the frequency band corresponding to the voice of the first and second languages is not recognized for a predetermined time. there is. At this time, the content providing unit 422 may suspend the output of a pair of subtitles for output through the transparent display 10 when the mute signal continues for a certain period of time, and replace and output healing content for changing the atmosphere. .

In this specification, only a few examples of various embodiments performed by the present inventors are described, but the technical spirit of the present invention is not limited or limited thereto, and can be modified and implemented in various ways by those skilled in the art, of course.

10: transparent display
100: recognition unit
200: text generator
300: translation derivation department
400: integrated control unit
1000: automatic translation and conversation assistance system

Claims (9)

a recognition unit for separately recognizing a voice of a first language spoken in one direction and a voice of a second language spoken in another direction by speaker;
a text generation unit that analyzes the voices of the first and second languages through artificial intelligence based on deep learning to generate time-specific sentence text and based on this, creates a dialog list;
Any one of preset rule-based, pattern-based, statistical-based, and artificial intelligence-based translators is embedded, and by using this, the dialog list is individually translated into preset languages to derive first and second translated texts. translation derivation department; and
An integrated control unit that sequentially selects a pair of subtitles for output from the first and second translation texts based on the time and speaker information identified from the dialog window list and outputs them in different output directions through a transparent display;
The pair of subtitles for output includes a first subtitle for output displayed in the first language and a second subtitle for output displayed in the second language and translated to have the same meaning as each other;
The recognizing unit includes at least one directional microphone for selectively recognizing a spoken voice uttered in a specific direction;
at least one input device for receiving an input text to replace the spoken voice;
at least one camera for acquiring a pair of speaker images by photographing in the specific direction;
a detection module for detecting a predetermined facial expression and gesture object from one of the pair of speaker images;
a text extraction module extracting from a memory a text for sign language corresponding to the gesture object and an emergency guidance text according to the preset facial expression; and
an identification module for assigning an identification code indicating a speech time and a speaker direction to the spoken voice, the input text, and the sign language text and storing them in the memory;
The integrated control unit includes a comparison unit that compares voice characteristics analyzed from each voice of the first and second languages recognized through the recognition unit with a preset reference sample; and
Based on the comparison result confirmed through the comparison unit, a voice correction unit for correcting each voice of the first and second languages according to the voice characteristics of the reference sample,
The voice correction unit integrates audio and visual information obtained by frequency-analyzing the voice signals of the first and second languages into the pair of output subtitles and outputs them through the transparent display;
The audio visual information is an audio spectrum visualized by frequency analysis of components and strengths of sound in order to induce utterance at an optimal volume level of the speaker's voice;
The voice correction unit controls to amplify a predetermined audible frequency band for the voice signals of the first and second languages based on the volume level of a band other than the audible frequency of voice,
The integrated control unit adjusts the output area of the pair of subtitles for output to the transparent display at different area ratios based on distance information for each speaker measured through a distance measuring sensor disposed on the transparent display,
As the integrated control unit outputs a simple visual acuity checklist before outputting the pair of subtitles for output, the integrated controller estimates the visual acuity of the speaker based on the response signal received as feedback, and automatically adjusts the magnification ratio of each font for the pair of subtitles for output based on this estimate. to regulate,
The integrated control unit adjusts a page switching speed of the next subtitle output after the pair of output subtitles based on eye movement data measured through an eye tracker module disposed on the transparent display;
As the integrated control unit registers SNS account information received through the at least one input device, it transmits the dialog list as an image file to a speaker terminal corresponding to the SNS account information using an integrated management server connected through a network. ,
The integrated control unit outputs a disaster alert through the transparent display based on a disaster environmental state detected through environment sensors disposed on the transparent display,
The environment detection sensors include a smoke detection sensor, a carbon monoxide detection sensor, and a temperature detection sensor, and an automatic interpretation/translation and conversation assistance system using a transparent display. According to claim 1,
The translator includes a plurality of field-specific customized translators,
The automatic translation and conversation assistance system using a transparent display, wherein the plurality of field-specific customized translators are translators from which unnecessary words are removed according to occupation, age, region, interest, and religion. delete According to claim 1,
a first modeling unit for modeling a deep learning-based voice recognition model by learning the vocabulary word dictionary previously collected in the memory;
a second modeling unit that learns the pronunciation rules previously collected in the memory and models a rule- and statistics-based acoustic model;
a signal processing unit which converts the voices of the first and second languages into input signals according to voice recognition feature vectors; and
A decoder for outputting the input signal as text for each speaker, which is character information, by applying the input signal to the speech recognition model and the acoustic model, and merging the input text and the sign language text with the text for each speaker according to the identification code , Automatic translation and conversation assistance system using a transparent display. delete According to claim 1,
The voice correction unit may include a filtering unit generating a noise canceling signal according to the noise signal identified from the silence signal recognized through the recognition unit, and filtering noise of the first and second language voice signals based thereon; and
When the silence signal lasts for a certain period of time, a content providing unit for temporarily pausing the output of the pair of output subtitles through the transparent display and replacing and outputting a healing content image for changing the atmosphere,
The mute signal includes a repetitive noise signal of a specific frequency band recognized through the recognition unit while the volume of the frequency band corresponding to the voice of the first and second languages is not recognized for a predetermined time. Automatic translation and conversation assistance system used.


delete delete delete

Images