Wikidata:ProVe

ProVe: Automated PROvenance VErification of Knowledge Graphs against Textual Sources

Introduction

ProVe is a tool for helping editors improve the references of Wikidata items. In general, Wikidata item statements should be verifiable and referenced by a source (e.g. a book, scientific paper, etc.). However, a large number of Wikidata item statements do not provide any reference and, if they do, such references are of varying quality which tends to decay over time. ProVe helps with this by providing information about the quality of the references of Wikidata items, based on techniques such as large language models, triple verbalisation, and semantic similairty. In its UI, ProVe introduces a reference quality indicator (located at the top right of the page) with a Reference score beteween 0 and 1 (the higher the score, the better the references) and three expandable tables that give an overview of the quality of references for the item's statements. These are categorized based on whether their references support, refute, or are inconclusive with respect to the claim. See, for example, Q44 (Beer), with the tool installed (instructions below) for a demonstration.

The reference quality score represents the overall level of support (from 0 to 1) for a Wikidata entity's references, with 1 indicating strongest support. Five different battery status icons visually represent this supportive level. Behind the scenes, ProVe uses an automated pipeline to analyse the level of support of a triple by its corresponding reference source. This involves several steps, including retrieval of the source document, extraction of text within it, verbalisation of the triple, selection of the sentences within the reference document that are most relevant for the claim, and finally the actual claim verification against those sentences.

ProVe aims to assist authors in identifying areas where reference quality can be improved and to inform data consumers about the level of support provided by the references on a specific Wikidata page.

ProVe is designed to access external web pages via reference URLs to verify the appropriateness of their content. It uses a Chrome web crawler driver to access non-Wikimedia sites without archiving the accessed pages. Only relevant parts (sentences) from the web page are used for text entailment, and the web page contents are discarded afterward.

IMPORTANT NOTE: To compute scores the ProVe widget communicates with a non-wikimedia site/server for processing claims and references. This cannot be avoided as ProVe needs the support of LLMs/heavy processing. We are working towards having a wikimedia-site only version.

User Guide

ProVe Installation Guide

ProVe adds visual components to each Wikidata entity page, helping you assess the quality of references. Follow these steps to enable ProVe on your Wikidata account.

1. Log into your Wikidata account: Make sure you're logged in to your Wikidata account.

2. Create or access your common.js file: Visit your common.js to open or create your common.js file. This will take you directly to your file, allowing you to edit it.

3. Add the ProVe script to your common.js: In the common.js file, paste the following line of code:

importScript('User:1hangzhao/ProVe.js');

// IMPORTANT NOTE: To compute scores, the ProVe widget communicates with a non-Wikimedia site/server for processing claims and references. This cannot be avoided as ProVe needs the support of LLMs/heavy processing. We are working towards having a Wikimedia-site-only version.

4. Save your changes: After pasting the script, click the "Publish changes" button to save the file.

5. Refresh the page: Once you’ve saved the changes, refresh the Wikidata page you're viewing. You should now see ProVe components activated and ready for use.

How to Use ProVe

1. Visit an item of interest, for example, Q44 (Beer)
2. At the top right corner, ProVe will show a Reference score summarising the quality of references for the item (closer to 0 means lower quality, and closer to 1 means higher quality). You can click on "Recompute" if you want to recompute this score after making changes to references (it may take some time).
3. At the top left corner, ProVe will show a "Check the quality of references" button. By clicking on it, ProVe will display a summary table with the following:
   1. The number of statements with and without references. Statements without references are a priority, so you'll be encouraged (by a "Start adding references to unreferenced statements" button) to add references to the first unsupported statement.
   2. "Red" statements with references to sources whose authoritativeness could not be checked
   3. "Yellow" statements with references that are possibly not relevant to backing the statement
   4. "Green" statements with references that most likely support the statement

Users can click on the "Link" anchor to any statement to scroll automatically to the reference in question and start editing it if needed.

(Re)Computing reference quality

Computing ProVe scores and assessing possible issues with references is an expensive process. Some items, instead of showing a score and the reference quality table directly, might show a "Prioritise" button, meaning that ProVe does not have a pre-computed score/reference table yet for that item. If you are interested in that item, you can click that button and the item will be given priority in the queue. Usually it only takes a couple of minutes to show updated scores and the table of references (simply refresh the page after a few minutes).

How ProVe Works

Reference Score Calculation

The reference score (RS) for a wikidata item with $n$ claims is calculated using the following formula:

$RS={\frac {1}{n}}\sum _{i=1}^{n}label_{i}$

where $label_{i}\in \{-1,0,1\}$ represents the reference verification result for the $i$-th claim of the wikidata item. $label_{i}$ is 1, if the result is 'SUPPORTS'; $label_{i}$ is -1, if the result is 'REFUTES'; otherwise, $label_{i}$ is 0.

Steps:

1. Retrieve Results: Collect the results for the target item.

2. Classify Results: Each result is classified into one of three categories:

1. SUPPORTS: The item supports the target claim.
2. REFUTES: The item refutes the target claim.
3. NOT ENOUGH INFO: The item provides insufficient information to make a conclusion.

3. Calculate Proportions:

1. SUPPORTS: It increases the reference score to 1. If all claims are determined as 'SUPPORTS', then the reference score should be 1.
2. REFUTES: It decreases the reference score to -1. If all claims are determined as 'REFUTES', then the reference score should be -1.
3. NOT ENOUGH INFO: It doesn't affect the score directly, but it increases the denominator. Therefore, if a Wikidata item has many 'NOT ENOUGH INFO' values compared to other labels, the absolute value of the reference score will be minimized, approaching zero.

4. Normalize the Score: The total reference score is normalized by dividing the sum of weighted negative results by the total number of results.

Frontend Components

ProVe extends the Wikidata entity pages by adding two visual components:

• A ProVe Score indicator, ranging from 0 (poor level of supportive references) to 1 (satisfactory level of references);
• Three expandable tables of external references categorized by different levels of supportiveness (supports, refutes, and not enough information)

ProVe Reasoning

Main algorithm

• The ProVe reasoning process is conducted sequentially for each Wikidata claim belonging to an item. ProVe processes an item by inputting its Q-id and returns 'No external URLs' if the item doesn't have any available external URLs. ProVe is designed to verify only external URLs, not internal URLs like connections among other Wikimedia and Wikidata resources.

ProVe Workflow

• Wikidata Parser: First, ProVe collects and filters Wikidata claims for an item with the requested Q-id. We have a great interest in checking the quality of external references, so ProVe filters out unnecessary Wikidata claims such as those having no URL or only internal URLs. For URLs encoded in a format different from common URLs, it transforms them into commonly accessible URLs for browsers. Then, it collects HTML documents according to these accessible URLs.

• Verbalization: In this function, ProVe transforms a set of Wikidata claims with URLs from RDF format to natural language sentences by adopting a verbalization model based on pre-trained T5. This function is designed to transform Wikidata claims into natural sentences for comparison with sentences from the HTML documents of URLs attached to the Wikidata claims. It runs for all Wikidata claims obtained in the previous Wikidata parsing process.

• Sentence Segmentation: This function cleans HTML documents obtained in the Wikidata parsing process to have only clean text without HTML tags and scripts. Then, it splits the cleaned HTML documents into a set of sentences using a Python library (pysbd). Segmented sentences will be compared to the Wikidata claims, respectively.

• Top-N Sentences Selection: ProVe creates multiple pairs for the Wikidata claims of the Wikidata Item. Each pair consists of one verbalized sentence from the Wikidata claim and a set of segmented sentences from the HTML document obtained by accessing the URL of the Wikidata claim. This function selects the top-N semantically similar sentences from the set of segmented sentences compared to the verbalized sentence, based on a fine-tuned BERT model.

• Text Entailment: This function determines if the Wikidata claim is supported or not by the N selected sentences, based on a fine-tuned BERT model with the FEVER dataset. The result for each selected sentence will be 'SUPPORTS', 'REFUTES', or 'NOT ENOUGH INFORMATION'. Finally, it returns a final result for the Wikidata claim by aggregating these text entailment results for the N selected sentences.

System architecture

• ProVe is currently running on a VM hosted at King's College London to utilize GPU resources. The system architecture of ProVe's backend consists of three main parts:

• ProVe Process: The first part is a set of Python codes for the ProVe process. You can find the source code of ProVe in the GitHub repository: https://github.com/King-s-Knowledge-Graph-Lab/RQV

• DBs and Resources: The second part relates to databases and resources used by ProVe or used to store data produced by ProVe. ProVe uses the Wikidata API to collect Wikidata claims and uses an SQLite database to store these collected claims. The ML model DB contains pre-trained machine learning models with datasets and configuration values. The HTML DB (SQLite) is designed to store raw HTML documents obtained during the ProVe process, serving as a kind of cache for HTML scraping. The Reasoning Data DB (SQLite) stores the entire intermediary and final result data during the ProVe process.

• API Server: The third part is an API host server using Flask. It returns the requested data from external sources by interacting with the databases and functions of ProVe. Details of the APIs are written at the bottom of this page.

ProVe Tools

ProVe Gadget

• This is a widget that represents the results of ProVe at the top of the Wikidata Item view page.

• The ProVe Gadget displays a button to request Wikidata item processing from the ProVe backend server on the same line as the Wikidata item title. It shows a 'Reference score' and its icon if the item has already been processed in the backend.
• The ProVe Gadget displays three boxes:
  • A red box includes potential bad references
  • A yellow box shows the list of references without enough information to be determined
  • A green box represents potential good references
• You can find more additional information and a tutorial in the Google Slides presentation: Empowering Wikidata editors and content with the Wikidata Quality Toolkit, Wikimania 2024 Hackathon
• To implement and activate the ProVe gadget, please visit https://www.wikidata.org/wiki/Wikidata:ProVe
• We are working to publish the ProVe Gadget on the official Wikidata gadget list.

ProVe Dashboard (Under Development)

• It will display statistics related to ProVe's work, such as (tentative):

 - The number of processed items
 - The number of processed triples
 - The number of processed references
 - User requests for Wikidata items
 - A reference score line graph over time

ProVe API

• The formal API documentation for the ProVe project is accessible on SwaggerHub. Click the link to view the complete interactive documentation, where you can explore the API endpoints, parameters, and responses.

WikiProject Reference Verification

We would like to encourage editors to use ProVe to improve the quality of references of some pre-selected items that seem to need priority in checking the quality of their references. We have a WikiProject on Reference Verification that we encourage you to visit and use to try ProVe. Your feedback for improving the tool, and improving the overall quality of Wikidata references, will be really appreciated! :-)

Additional Resources

Presentation: Empowering Wikidata editors and content with the Wikidata Quality Toolkit, Wikimania 2024 Hackathon