@kostajh -- as I think about this, I think there is another metric that we will be interested in knowing (cc @RHo @SCherukuwada):

Account creations: how does the deployment of temp accounts affect the number of accounts being created?

thanks @MMiller_WMF, yes, account creation changes would be really helpful to know for sure.
And @kostajh apologies if this is covered within the "Edit rate" measurement, but does temp account editing encompass IP editing as well in the comparison? That is, is there some baseline measure prior to temp accounts to determine what the IP edit/Account edit ratio is versus Temp accounts?

Another potentially useful metric would be temp account edit abandonment vs IP editing abandonment if that is something we have info on? This may help us determine if the new messages and dialogs about temp editing deters/spurs temp account edits.

In T357763#9759685, @RHo wrote:

thanks @MMiller_WMF, yes, account creation changes would be really helpful to know for sure.
And @kostajh apologies if this is covered within the "Edit rate" measurement, but does temp account editing encompass IP editing as well in the comparison? That is, is there some baseline measure prior to temp accounts to determine what the IP edit/Account edit ratio is versus Temp accounts?

In theory, we would switch on the instrumentation some time in advance of launching temp accounts, so we'd have a bit of a baseline. That said, it will be difficult to draw direct comparisons between IP edit/temp account editing, because in the IP editing paradigm, there might be multiple distinct devices from a particular IP (see T357771#9648033 for more info). We'll have to see how that will work out in practice.

Another potentially useful metric would be temp account edit abandonment vs IP editing abandonment if that is something we have info on? This may help us determine if the new messages and dialogs about temp editing deters/spurs temp account edits.

Thanks for the suggestion, I updated the description.

Re: Rate of Blocks - we should also include IP Blocks in the comparison because they will still be an option. It would be nice if we see fewer IP blocks and more temp account blocks instead as it could help with the collateral damage problem.

Some more metrics that are potentially useful to see:

• Rate of reverts for Temporary Accounts compared to those for IP editors
  • Why: It would be interesting to see if temporary accounts receive a different treatment due to the lack of widespread of IP address accessibility and the possibility of being able to communicate better with them and help them become constructive editors.

Metrics to understand the patroller behavior towards Temp accounts and IP addresses -

• Number of IP Reveal opt-ins
  • Why: Capturing this metric can help us get an estimate of how many patrollers are needing to see IP addresses to patrol effectively. We had some projections for this early on but it would be good to get actual numbers to understand this further. Would be even better to get a % of how many users qualify to opt-in versus how many choose to do so.

• Active IP Reveal users
  • Why: A step further from the above metric: Users who have opted-in to view IPs and actually use the feature frequently. This can tell us how many editors feel the need to use the feature in practice over time.

• Number of IPs Revealed as a percentage of total IPs that edited
  • Why: This can be a useful impact metric. It tells us how many users actually had their information exposed in practice. If this is a quite small number, we know that fewer people's PII is being looked at and that's a win. If this is a very large number, it may point to issues and we may need to investigate other tools we could build to assist in the patrolling process.

Adding to @Niharika's list, I think it would be useful to measure the following as well:

• Number of CheckUser investigations involving a Temporary account
  • Why: Temporary accounts should be workable without having CheckUser-level permissions. A reasonable number of CU investigations for Temporary accounts is likely okay. However, if the number of CU investigations of temporary accounts is too high, it hints at Reveal IP perhaps not being sufficient, requiring CheckUsers to step in.
  • I'm unsure how to define "involving" exactly. It is expected Temporary accounts will be shown when querying by an IP. Maybe we should measure the number of times a checkuser entered a Temporary account name into Special:CheckUser, and that would constitute involvement for this metric?

In T357763#10133212, @Urbanecm wrote:

Adding to @Niharika's list, I think it would be useful to measure the following as well:

• Number of CheckUser investigations involving a Temporary account
  • Why: Temporary accounts should be workable without having CheckUser-level permissions. A reasonable number of CU investigations for Temporary accounts is likely okay. However, if the number of CU investigations of temporary accounts is too high, it hints at Reveal IP perhaps not being sufficient, requiring CheckUsers to step in.
  • I'm unsure how to define "involving" exactly. It is expected Temporary accounts will be shown when querying by an IP. Maybe we should measure the number of times a checkuser entered a Temporary account name into Special:CheckUser, and that would constitute involvement for this metric?

@Urbanecm Do you think the total number of CU checks made on a wiki will be a reasonable proxy for this?

Note that I replaced Number of IPs Revealed as a percentage of total IPs that edited with Percentage of temporary accounts that had their IP revealed as the latter is easier to capture.