Document processing: Difference between revisions

'''Document processing''' is a field of research and a set of [[production process]]es aimed at making an analog [[document]] digital. Document processing does not simply aim to photograph or [[Image scanning|scan]] a document to obtain a [[digital image]], but also to make it digitally intelligible. This includes extracting the structure of the document or the [[Document layout analysis|layout]] and then the content, which can take the form of text or images. The process can involve traditional [[computer vision]] algorithms, [[convolutional neural networks]] or manual labor. The problems addressed are related to [[semantic segmentation]], [[object detection]], [[optical character recognition|optical character recognition (OCR)]], [[Handwritten text recognition|handwritten text recognition (HTR)]] and, more broadly, [[Transcription (linguistics)|transcription]], whether [[Automation|automatic]] or not.<ref>{{Cite book

|date=2003 |publisher=Idea Group Inc (IGI) |isbn=9781591400554}}</ref> The term can also include the phase of digitizing the document using a scanner and the phase of interpreting the document, for example using [[Natural language processing|natural language processing]] (NLP)]] or [[image classification]] technologies. It is applied in many industrial and scientific fields for the optimization of administrative processes, mail processing and the digitization of analog [[Archiving|archives]] and historical documents.

|date=2005-12-05 |publisher=Springer Science & Business Media |isbn=9783540247944}}</ref> Document processing can indeed involve some kind of externalized manual labor, such as [[Amazon Mechanical Turk|mechanical turkTurk]].

As an example of manual document processing, as relatively recent as 2007,<ref name="VisaDox">{{cite news |newspaper=[[The New York Times]]

|author=Julia Preston |date=December 2, 2007}}</ref> document processing for "millions of visa and citizenship applications" was about use of "approximately 1,000 contract workers" working to "manage mailroommail room and [[data entry clerk|data entry]]."

While document processing involved data entry via keyboard well before use of a [[computer mouse]] or a [[Image scanner|computer scanner]], a 1990 article in ''[[The New York Times]]'' regarding what it called the "[[paperless office]]" stated that "document processing begins with the scanner.".<ref name="Paper.NYT">{{cite news|newspaper=[[The New York Times]]

|author=Lawrence M. Fisher |date=July 7, 1990}}</ref> In this context, a former [[Xerox]] Vicevice-president, Paul Strassman, expressed a critical opinion, saying that computers add rather than reduce the volume of paper in an office.<ref name="Paper.NYT"/> It was said that the engineering and maintenance documents for an airplane weigh "more than the airplane itself"{{citation needed|date=April 2019}}.

A technology called automatic document processing or sometimes intelligent document processing (IDP) emerged as a specific form of [[Process Automation|Intelligent Process Automation]] (IPA), combining [[artificial intelligence]] such as [[Machine Learning]] (ML), [[Natural Language Processing]] (NLP) or [[Intelligent Character Recognition]] (ICRICE) to extract data from several types documents.<ref>{{Cite web|url=http://www.di.uniba.it/~ndm/pubs/esposito05icdar.pdf|title=Intelligent Document processing by Floriana Esposito , Stefano Ferilli , Teresa M. A. Basile , Nicola Di Mauro|date=2005-04-07|website=Department of Computer Science – University of Bari|access-date=2018-09-08}}</ref><ref>{{Cite book

|author=[[Floriana Esposito ]], Stefano Ferilli , Teresa M. A. Basile , Nicola Di Mauro |date=2005-04-01

Automatic document processing applies to a whole range of documents, whether structured or not. For instance, in the world of business and finance, technologies may be used to process paper-based invoices, forms, purchase orders, contracts, and currency bills.<ref>{{cite patent |country=US|number=US7873576B2|status=active|title= Financial document processing system |pubdate=2011-01-18|gdate=2011-01-18|invent1=John E. Jones|invent2=William J. Jones|invent3=Frank M. Csultis|url=https://patents.google.com/patent/US7873576B2/en}}</ref> Financial institutions use intelligent document processing to process high volumes of forms such as regulatory forms or loan documents. IDPID uses AI to extract and classify data from documents, replacing manual data entry.<ref>{{Cite web|last=Bridgwater|first=Adrian|title=Appian Adds Google Cloud Intelligence To Low-Code Automation Mix|url=https://www.forbes.com/sites/adrianbridgwater/2020/03/09/appian-adds-google-cloud-intelligence-to-low-code-automation-mix/|access-date=2021-04-21|website=Forbes|language=en}}</ref>

In medicine, document processing methods have been developed to facilitate patient follow-up and streamline administrative procedures, in particular by digitizing medical or laboratory analysis reports. The goal is also to standardize medical databases.<ref>{{cite journal |last1=Adamo|first1=Francesco|last2=Attivissimo|first2=Filippo|first3=Attilio|last3=Di Nisio|first4=Maurizio|last4=Spadavecchia|date=February 2015|title=An automatic document processing system for medical data extraction|url=https://www.sciencedirect.com/science/article/pii/S0263224114005016?casa_token=heNt0-b7OZQAAAAA:uvH1ZRZOGIw3XbLF245ePomFOwVl8svsdARmurepRv7G7vcISA9Gc8d5ZhmcG6Y0RJTJeJ2Sn98|journal=Measurement|volume=61|pages=88–99 |doi=10.1016/j.measurement.2014.10.032|bibcode=2015Meas...61...88A |access-date=31 January 2021}}</ref> Algorithms are also directly used to assist the physicians in medical diagnosis, e.g. by analyzing [[Magnetic resonance imaging|magnetic resonance images]],<ref>{{cite journal |last1=Changwan|first1=Kim|last2=Seong-Il|first2=Lee|last3=Won Joon|first3=Cho|date=September 2020|title=Volumetric assessment of extrusion in medial meniscus posterior root tears through semi-automatic segmentation on 3-tesla magnetic resonance images|url=https://www.sciencedirect.com/science/article/abs/pii/S1877051720301994|journal=Orthopaedics & Traumatology: Surgery & Research|volume=101|issue=5|pages=963–968|doi=10.1016/j.rcot.2020.06.003|s2cid=225215597 |access-date=31 January 2021}}</ref><ref>{{cite journal |last1=Despotović|first1=Ivana|last2=Bart|first2=Goossens|last3=Wilfried|first3=Philips|date=1 March 2015|title=MRI Segmentation of the Human Brain: Challenges, Methods, and Applications|journal=Computational Intelligence Techniques in Medicine|volume=2015|pages=963–968|doi=10.1155/2015/450341|pmid=25945121|pmc=4402572|doi-access=free}}</ref> or [[Microscope|microscopic]] images.<ref>{{cite journal |last1=Putzua|first1=Lorenzo|last2=Caocci|first2=Giovanni|last3=Di Rubertoa|first3=Cecilia|title=Leucocyte classification for leukaemia detection using image processing techniques|journal=Artificial Intelligence in Medicine|date=November 2014|url=https://www.sciencedirect.com/science/article/pii/S0933365714001031?casa_token=kKYi78VDsLgAAAAA:VN2gsae1xi3mH6_g1LpGPbx-kS-3VwfmiFlHtAYLydID7GYJAlVLIlyB-p-LFiucergfTdzZOwM|volume=63|issue=3|pages=179–191|doi=10.1016/j.artmed.2014.09.002|pmid=25241903|hdl=11584/94592|hdl-access=free}}</ref>

If, from the 1980s onwardsonward, traditional computer vision algorithms were widely used to solve document processing problems,<ref>{{cite journal |last1=Fujisawa|first1=H.|last2=Nakano|first2=Y.|last3=Kurino|first3=K.|date= July 1992

|bibcode=1996PatRe..29.1931T |access-date=3 February 2021}}</ref> these have been gradually replaced by neural network technologies in the 2010s.<ref>{{cite conference |url=https://ieeexplore.ieee.org/document/8563218|title= dhSegment: A Generic Deep-Learning Approach for Document Segmentation|last1=Ares Oliveira|first1=Sofia|last2=Seguin|first2=Benoit|last3=Kaplan|first3=Frederic|date=5-85–8 August 2018

At the other end of the chain are various image completion, extrapolation or data cleanup algorithms. For textual documents, the interpretation can use [[natural language processing]] (NLP) techologiestechnologies.