research-article

Automatic evaluation of summary on fidelity, conciseness and coherence for text summarization based on semantic link network

Authors:

Hai ZhugeAuthors Info & Claims

Volume 206, Issue C

https://doi.org/10.1016/j.eswa.2022.117777

Published: 15 November 2022 Publication History

Highlights

•

Automatic and accurate evaluation of summary is a key problem of text summarization research.

•

A framework for evaluating summary from fidelity, conciseness, and coherence.

•

An evaluation method based on the semantic link network representation of text.

•

The semantic link network plays an important role in representing and analyzing texts.

Abstract

Automatic evaluation of summary is one of the basic problems of automatic text summarization research. The commonly used approaches evaluate the informativeness, conciseness and coherence of summary separately based on different representations with weak semantics such as n-gram and vector of words. By transforming text into semantic link network, this paper proposes a general framework for automatically evaluating summary from the following aspects: (1) fidelity, the extent of a summary that conveys the core of its source text; (2) conciseness, the extent of non-redundancy within summary and, (3) coherence, the extent of relatedness between representations within summary, which are based on a uniform semantic representation of text. Experiments on the evaluation of the summarization of scientific papers and news show that the proposed framework achieves comparable or higher correlations with human judgments than the popular evaluation models. This research also verifies the role of the semantic link network in representing and analysing texts.

References

[1]

S. Banerjee A. Lavie METEOR: An automatic metric for MT evaluation with improved correlation with human judgments 2005 Prague, Czech Republic 65 72.

[2]

R. Barzilay, N. Elhadad, Inferring strategies for sentence ordering in multidocument news summarization, Journal of artificial intelligence research 17 (2002) 35–55,.

[3]

Cao, M., Sun, X., & Zhuge, H. (2018). The contribution of cause-effect link to representing the core of scientific paper -- The role of semantic link network. PloS one, 13(6), Article e0199303. https://doi.org/10.1371/journal.pone.0199303.

[4]

Cao, M., & Zhuge, H. (2019). Automatic evaluation of text summarization based on semantic link network. Paper presented at the 15^th International Conference on Semantics, Knowledge and Grids (pp. 107-114), Guangzhou, China. https://doi.org/10.1109/SKG49510.2019.00026.

[5]

M. Cao, H. Zhuge, Grouping sentences as better language unit for extractive text summarization, Future Generation Computer Systems 109 (2020) 331–359,.

[6]

Chen, J., & Zhuge, H. (2019). Automatic generation of related work through summarizing citations. Concurrency and Computation: Practice and Experience, 31(3), Article e4261. https://doi.org/10.1002/cpe.4261.

[7]

Clark, E., Celikyilmaz, A., & Smith, N. A. (2019). Sentence mover’s similarity: Automatic evaluation for multi-sentence texts. Paper presented at the 57^th Annual Meeting of the Association for Computational Linguistics (pp. 2748-2760), Florence, Italy. http://dx.doi.org/10.18653/v1/P19-1264.

[8]

Dang, H. T. (2005). Overview of DUC 2005. Paper presented at the Document Understanding Conference (pp. 1-12). https://www-nlpir.nist.gov/projects/duc/pubs/2005papers/OVERVIEW05.pdf.

[9]

Dang, H. T., & Owczarzak, K. (2008). Overview of the TAC 2008 update summarization task. Paper presented at the 1^st Text Analysis Conference, Gaithersburg, Maryland, USA,. https://tac.nist.gov//publications/2008/additional.papers/update_summ_overview08.proceedings.pdf.

[10]

G. Doddington Automatic evaluation of machine translation quality using n-gram co-occurrence statistics 2002 San Diego, CA 128 132 10.3115/1289189.1289273.

[11]

L. Ermakova, J.V. Cossu, J. Mothe, A survey on evaluation of summarization methods, Information Processing & Management 56 (5) (2019) 1794–1814,.

Digital Library

[12]

Gao, Y., Zhao, W., & Eger, S. (2020). SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization. Paper presented at the 58^th Annual Meeting of the Association for Computational Linguistics (pp. 1347-1354), Online. 10.18653/v1/2020.acl-main.124.

[13]

S. Gholamrezazadeh, M.A. Salehi, B. Gholamzadeh, A comprehensive survey on text summarization systems, in: Paper presented at the 2^nd International Conference on Computer Science and Its Applications, 2009, pp. 1–6,.

[14]

B.J. Grosz, A.K. Joshi, S. Weinstein, Centering: A framework for modelling the local coherence of discourse, Computation Lingustics 21 (2) (1995) 203–225,.

[15]

Guinaudeau, C., & Strube, M. (2013). Graph-based local coherence modeling. Paper presented at the 51^st Annual Meeting of the Association for Computational Linguistics (pp. 93-103), Sofia, Bulgaria. https://www.aclweb.org/anthology/P13-1010/.

[16]

Halteren, H. V., & Teufel, S. (2003). Examining the consensus between human summaries: Initial experiments with factoid analysis. Paper presented at the Human Language Technology Conference of the North American Chapter of the ACL on Text Summarization Workshop (pp. 57-64). https://doi.org/10.3115/1119467.1119475.

[17]

W. Li, H. Zhuge, Abstractive multi-document summarization based on semantic link network, IEEE Transactions on Knowledge and Data Engineering 33 (1) (2019) 43–54,.

Digital Library

[18]

Lin, C.-Y. (2004). Rouge: A package for automatic evaluation of summaries. Paper presented at the ACL Workshop on Text Summarization Branches Out (pp. 74-81), Barcelona, Spain. https://www.aclweb.org/anthology/W04-1013.

[19]

Melnik, S., Garcia-Molina, H., & Rahm, E. (2002). Similarity flooding: A versatile graph matching algorithm and its application to schema matching. Paper presented at the 18^th International Conference on Data Engineering (pp. 117-128), San Jose, CA, USA. https://doi.org/10.1109/ICDE.2002.994702.

[20]

Mesgar, M., & Strube, M. (2018). A neural local coherence model for text quality assessment. Paper presented at the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 4328-4339), Brussels, Belgium. http://dx.doi.org/10.18653/v1/D18-1464.

[21]

Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Paper presented at the 26^th International Conference on Neural Information Processing Systems (pp. 3111-3119), Lake Tahoe, Nevada, USA. https://dl.acm.org/doi/abs/10.5555/2999792.2999959.

[22]

Mrabet, Y., & Demner-Fushman, D. (2020). HOLMS: Alternative summary evaluation with large language models. Paper presented at the 28^th International Conference on Computational Linguistics (pp. 5679-5688), Barcelona, Spain (Online). http://dx.doi.org/10.18653/v1/2020.coling-main.498.

[23]

Nguyen, D. T., & Joty, S. (2017). A neural local coherence model. Paper presented at the 55^th Annual Meeting of the Association for Computational Linguistics (pp. 1320-1330), Vancouver, Canada. https://www.aclweb.org/anthology/P17-1121.pdf.

[24]

Oxford. (2010). Oxford dictionary of English (A. Stevenson Ed. 3 ed.). Oxford University Press, USA.

[25]

Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). BLEU: A method for automatic evaluation of machine translation. Paper presented at the 40^th Annual Meeting of the Association for Computational Linguistics (pp. 311-318), Philadelphia, Pennsylvania, USA. https://doi.org/10.3115/1073083.1073135.

[26]

Parveen, D., & Strube, M. (2015). Integrating importance, non-redundancy and coherence in graph-based extractive summarization. Paper presented at the 24^th International Joint Conference on Artificial Intelligence (pp. 1298-1304), Buenos Aires, Argentina. https://dl.acm.org/doi/10.5555/2832415.2832430.

[27]

Peyrard, M. (2019). A simple theoretical model of importance for summarization. Paper presented at the 57^th Annual Meeting of the Association for Computational Linguistics (pp. 1059-1073), Florence, Italy. https://doi.org/10.18653/v1/P19-1101.

[28]

Peyrard, M., & Gurevych, I. (2018). Objective function learning to match human judgements for optimization-based summarization. Paper presented at the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 654-660), New Orleans, Louisiana. http://dx.doi.org/10.18653/v1/N18-2103.

[29]

ShafieiBavani, E., Ebrahimi, M., Wong, R., & Chen, F. (2018). A graph-theoretic summary evaluation for rouge. Paper presented at the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 762-767), Brussels, Belgium. http://dx.doi.org/10.18653/v1/D18-1085.

[30]

X. Sun, H. Zhuge, Summarization of scientific paper through reinforcement ranking on semantic link network, IEEE Access 6 (2018) 40611–40625,.

[31]

Tratz, S., & Hovy, E. H. (2008). Summarization evaluation using transformed basic elements. Paper presented at the 1^st Text Analysis Conference, Gaithersburg, Maryland, USA,. https://tac.nist.gov/publications/2008/additional.papers/ISI.proceedings.pdf.

[32]

Vasilyev, O., Dharnidharka, V., & Bohannon, J. (2020, nov). Fill in the BLANC: Human-free quality estimation of document summaries. Paper presented at the 2020 Conference on Empirical Methods in Natural Language Processing (pp. 11-20), Online. http://dx.doi.org/10.18653/v1/2020.eval4nlp-1.2.

[33]

R. Vedantam, C. Lawrence Zitnick, D. Parikh, CIDEr: Consensus-based image description evaluation, in: Paper presented at the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 4566–4575,.

[34]

Wan, X., Yang, J., & Xiao, J. (2007). Manifold-ranking based topic-focused multi-document summarization. Paper presented at the 20^th International Joint Conference on Artificial Intelligence (pp. 2903-2908), Hyderabad, India. https://www.aaai.org/Papers/IJCAI/2007/IJCAI07-467.pdf.

[35]

Xenouleas, S., Malakasiotis, P., Apidianaki, M., & Androutsopoulos, I. (2019). SUM-QE: a BERT-based summary quality estimation model. Paper presented at the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 6005-6011), Hong Kong, China. http://dx.doi.org/10.18653/v1/D19-1618.

[36]

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020, April 26-30). BERTScore: Evaluating text generation with BERT. Paper presented at the International Conference on Learning Representations, Addis Ababa, Ethiopia. https://arxiv.org/abs/1904.09675.

[37]

Y. Zhong, A theory of semantic information, China communications 14 (1) (2017) 1–17,.

[38]

Zhou, L., Lin, C.-Y., Munteanu, D. S., & Hovy, E. (2006). ParaEval: Using paraphrases to evaluate summaries automatically. Paper presented at the Human Language Technology Conference of the North American Chapter of the ACL (pp. 447-454), New York. https://dl.acm.org/doi/10.3115/1220835.1220892.

[39]

W. Zhou, K. Xu, Learning to compare for better training and evaluation of open domain natural language generation models, in: Paper presented at the AAAI Conference on Artificial Intelligence, 2020, pp. 9717–9724,.

[40]

H. Zhuge, Active e-document framework ADF: Model and tool, Information & Management 41 (1) (2003) 87–97,.

Digital Library

[41]

H. Zhuge, Discovery of knowledge flow in science, Communications of the Acm 49 (5) (2006) 101–107,.

Digital Library

[42]

H. Zhuge, The Web Resource Space Model, Springer, Boston, 2008.

[43]

H. Zhuge, Communities and emerging semantics in semantic link network: Discovery and learning, IEEE Transactions on Knowledge and Data Engineering 21 (6) (2009) 785–799,.

Digital Library

[44]

H. Zhuge, Interactive semantics, Artificial Intelligence 174 (2) (2010) 190–204,.

Digital Library

[45]

Zhuge, H. (2010b). Socio-natural thought semantic link network: A method of semantic networking in the cyber physical society. Paper presented at the 24^th IEEE International Conference on Advanced Information Networking and Applications (pp. 19-26), Perth, WA, Australia. https://doi.org/10.1109/AINA.2010.186.

[46]

H. Zhuge, Semantic linking through spaces for cyber-physical-socio intelligence: A methodology, Artificial Intelligence 175 (5) (2011) 988–1019,.

Digital Library

[47]

H. Zhuge, The Knowledge Grid: Toward Cyber-Physical Society, (2 ed.)., World Scientific Publishing Co., 2012.

[48]

H. Zhuge, Multi-Dimensional Summarization in Cyber-Physical Society, Elsevier, 2016.

[49]

H. Zhuge, Cyber-Physical-Social Intelligence on Human-Machine-Nature Symbiosis, Springer, Singapore, 2020.

[50]

H. Zhuge, X. Li, Peer-to-peer in metric space and semantic space, IEEE Transactions on Knowledge and Data Engineering 19 (6) (2007) 759–771,.

Digital Library

[51]

H. Zhuge, J. Liu, L. Feng, X. Sun, C. He, Query routing in a P2P semantic link network, Computational Intelligence 21 (2) (2005) 197–216,.

[52]

H. Zhuge, Y. Xing, Probabilistic resource space model for managing resources in cyber-physical society, IEEE Transactions on Service Computing 5 (3) (2012) 404–421,.

Digital Library

[53]

Zhuge, H., Zheng, L., Zhang, N., & Li, X. (2004). An automatic semantic relationships discovery approach. Paper presented at the 13^th International World Wide Web Conference on Alternate Track Papers & Posters (pp. 278-279), New York, USA. https://doi.org/10.1145/1013367.1013434.

Cited By

Zhang MLi CWan MZhang XZhao Q(2024)ROUGE-SEMExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121364237:PAOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121364

Index Terms

Automatic evaluation of summary on fidelity, conciseness and coherence for text summarization based on semantic link network

Index terms have been assigned to the content through auto-classification.

Recommendations

Recent automatic text summarization techniques: a survey

As information is available in abundance for every topic on internet, condensing the important information in the form of summary would benefit a number of users. Hence, there is growing interest among the research community for developing new ...
Fuzzy Genetic Semantic Based Text Summarization
DASC '11: Proceedings of the 2011 IEEE Ninth International Conference on Dependable, Autonomic and Secure Computing

Automatic text summarization is a data reduction process to exclude unnecessary details and present important information in a shorter version. One way to summarize document is by extracting important sentences in the document. To select suitable ...
Sentiment diversification for short review summarization
WI '17: Proceedings of the International Conference on Web Intelligence

With the abundance of reviews published on the Web about a given product, consumers are looking for ways to view major opinions that can be presented in a quick and succinct way. Reviews contain many different opinions, making the ability to show a ...

Comments

Information & Contributors

Information

Published In

cover image Expert Systems with Applications: An International Journal

Expert Systems with Applications: An International Journal Volume 206, Issue C

Nov 2022

1603 pages

ISSN:0957-4174

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 15 November 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang MLi CWan MZhang XZhao Q(2024)ROUGE-SEMExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121364237:PAOnline publication date: 27-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121364

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents