[epic] SDoC: Determine baseline for metrics
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	debt
	Aug 30 2017, 12:35 AM

Description

As the SDC General project ramps up, we'll need to figure out a baseline for metrics on Commons in order to measure future successes. This ticket will be used as a base for the needs/wants for this endeavor.

This is scheduled to take place in Q2, as per goals: https://www.mediawiki.org/wiki/Wikimedia_Technology/Goals/2017-18_Q2#Segment_2:_Search_integration_and_exposure

Let's do this by:

Investigating what and if sufficient eventlogging is already setup for Commons search
- if not, investigate what are the appropriate metrics to track
- is eventlogging setup only on Commons or is it on other projects
  - Multi-projects at once?
  - Search satisfaction schema
How do users generally use Commons
- Are there behaviors that we can easily identify to help make search better
- Is there a way to tell what the current zero results rate is for searches on Commons
How many search "hits" are based on a match in the file name vs. description vs. category
Better analysis of how many files may be currently "unfindable" because of lack of categorization, unhelpful file name, no description (or poor description)
Analysis of how many contributions are made by individuals vs. mass-tools/institutions

Things to think about / keep in mind:

Zero results rate (ZRR)
More relevant results
- What is ‘relevant’
Clickthroughs from cross-project searches
API usage
User engagement
- ‘People were able to learn more’
User satisfaction
Effort users spend finding something
Time users spend finding something
Tracking downloads of media
Steps to unique queries
How many embeds on other projects
How many times a specific file/image has shown up in searches
How many files/images never show up in searches
How many searches are by exclusions
- ‘Pictures of cats but not calicos’

Related Objects
Search...

Status	Assigned	Task
Resolved	None	T174072 [Q1 2017-18 Objective] Extend Commons search to support media via structured data
Resolved	• chelsyx	T174519 [epic] SDoC: Determine baseline for metrics
Resolved	• chelsyx	T177353 Metrics for SDoC: look at search hits based on which element the search is hitting
Open	None	T182849 Identify unhelpful file names on commons
Resolved	• chelsyx	T177354 Metrics for SDoC: look at contributions
Resolved	mpopov	T177356 Metrics for SDoC: look at querying databases
Declined	None	T177357 Metrics for SDoC: future work of interest (templates and licensing)
Resolved	mpopov	T177358 Metrics for SDoC: translations
Invalid	None	T182352 UDF for language detection
Resolved	• chelsyx	T177534 Search Metrics for SDoC: eventlogging
Resolved	• chelsyx	T179450 Documentation of SDoC findings

Event Timeline

debt created this task.Aug 30 2017, 12:35 AM

Restricted Application added a project: Wikidata. · View Herald TranscriptAug 30 2017, 12:35 AM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• SandraF_WMF awarded a token.Aug 30 2017, 7:48 AM

• SandraF_WMF moved this task from Backlog to CURRENT - Work in progress on the SDC General board.

• SandraF_WMF subscribed.

• EBjune added a parent task: T174072: [Q1 2017-18 Objective] Extend Commons search to support media via structured data.Sep 1 2017, 12:06 AM

• Ramsey-WMF subscribed.Sep 6 2017, 3:58 PM

Since this will involve some exploratory analyst work on Commons, can that include some general Commons metrics? @Capt_Swing can set up the general metric work, if the analysts can run the numbers.

Yup, we've got an upcoming meeting with @Capt_Swing to discuss just that!

debt updated the task description. (Show Details)Sep 13 2017, 10:14 PM

Some specific desired metrics from multimedia (some are simple/basic, others...maybe not so much):

how many search "hits" are based on a match in the file name vs. description vs. category
better analysis of how many files may be currently "unfindable" because of lack of categorization, unhelpful file name, no description (or poor description)
analysis of how many contributions are made by individuals vs. mass-tools/institutions

debt moved this task from Needs triage to Up Next on the Discovery-Analysis board.Sep 21 2017, 8:42 PM

debt renamed this task from Determine baseline for metrics on Commons to [epic] Determine baseline for metrics on Commons.Oct 3 2017, 11:25 PM

debt renamed this task from [epic] Determine baseline for metrics on Commons to [epic] SDoC: Determine baseline for metrics.Oct 3 2017, 11:33 PM

debt raised the priority of this task from Medium to High.

debt edited projects, added Discovery-Analysis (Current work); removed Discovery-Analysis.

debt updated the task description. (Show Details)

debt created subtask T177353: Metrics for SDoC: look at search hits based on which element the search is hitting.Oct 3 2017, 11:36 PM

debt created subtask T177354: Metrics for SDoC: look at contributions.

debt created subtask T177356: Metrics for SDoC: look at querying databases.Oct 3 2017, 11:41 PM

debt created subtask T177357: Metrics for SDoC: future work of interest (templates and licensing).Oct 3 2017, 11:46 PM

debt created subtask T177358: Metrics for SDoC: translations.

debt created subtask T177534: Search Metrics for SDoC: eventlogging.Oct 5 2017, 8:12 PM

Liuxinyu970226 awarded a token.Oct 16 2017, 10:42 AM

Liuxinyu970226 subscribed.

debt closed subtask T177356: Metrics for SDoC: look at querying databases as Resolved.Oct 19 2017, 8:00 PM

Please have in mind that metrics for commons exist is https://stats.wikimedia.org/wikispecial/EN/TablesWikipediaCOMMONS.htm , let's make sure those are looked at when this work is taking place.

debt created subtask T179450: Documentation of SDoC findings.Oct 31 2017, 11:42 PM

debt changed the status of subtask T177353: Metrics for SDoC: look at search hits based on which element the search is hitting from Open to Stalled.Nov 7 2017, 9:31 PM

debt closed subtask T177354: Metrics for SDoC: look at contributions as Resolved.Nov 21 2017, 9:10 PM

• chelsyx changed the status of subtask T177353: Metrics for SDoC: look at search hits based on which element the search is hitting from Stalled to Open.Dec 12 2017, 7:10 PM

• chelsyx mentioned this in T182849: Identify unhelpful file names on commons.Dec 14 2017, 7:17 PM

Lydia_Pintscher moved this task from incoming to monitoring on the Wikidata board.Dec 18 2017, 3:08 PM

Thanks for the work so far, @chelsyx and @mpopov! 🎉

Read the documentation here: https://meta.wikimedia.org/wiki/Research:Baseline_Metrics_for_Structured_Data_on_Wikimedia_Commons

• chelsyx moved this task from In progress to Needs review on the Discovery-Analysis (Current work) board.Dec 22 2017, 1:51 AM

Nice! Thank you for documenting.

@Ramsey-WMF Is there any feedback about the baseline metrics from the team? Could we resolve this ticket and other child tickets?

Hi @chelsyx . The main feedback I'd give is: AWESOME! Thanks to both you and @mpopov for fantastic work. Commons is a little wild, but you bravely navigated the jungle :)

We've shared the findings with a number of team members over the past few weeks and I haven't heard anything further, so I think we can close this baseline round.