Page MenuHomePhabricator

Variant tests: Create hybrid of Variant C / D
Closed, ResolvedPublic

Assigned To
Authored By
MMiller_WMF
Jan 22 2021, 10:43 PM
Referenced Files
F34148516: image.png
Mar 9 2021, 10:49 PM
F34148492: image.png
Mar 9 2021, 10:49 PM
F34148503: image.png
Mar 9 2021, 10:49 PM
F34148497: image.png
Mar 9 2021, 10:49 PM
F34148482: image.png
Mar 9 2021, 10:49 PM
F34148500: image.png
Mar 9 2021, 10:49 PM
F34148511: image.png
Mar 9 2021, 10:49 PM
F34148509: image.png
Mar 9 2021, 10:49 PM

Description

We've analyzed the results of this variant test, and we'll be publishing the specifics in the coming weeks. In short, Variant D leads to more users completing suggested edits on desktop, and Variant C leads to more users completing suggested edits on mobile. Therefore, we want all users to receive the winning variants for their platforms.

Specifically:

  • This change is for all users who receive Growth features (i.e. excluding those in our control groups), so variant A, C, and D users are included in this group, along with users who preceded explicit variant assignments.
  • If the user is on desktop, they should be experiencing Variant D, regardless of the platform on which they created their account.
    • uninitiated suggested edits
  • If the user is on mobile, they should be experiencing Variant C, regardless of the platform on which they created their account.
    • pre-initiated suggested edits
  • Both desktop and mobile should include messaging in onboarding dialogs that is informed by welcome survey responses (currently this only exists for variant A/C users)
  • Both variants include onboarding steps that users only go through once. Variant D has two screens that users go through (topics and difficulty), while Variant C has a popups urging users to read onboarding dialogs. For users that open the homepage on both platforms, we want to respect their onboarding status:
    • If the user creates their account on desktop, and gets through Variant D's onboarding screens to initiate the module, then when they open the homepage on mobile, they should not receive Variant C's popup or any other onboarding.
    • If the users creates their account on mobile, then when they open the homepage on desktop, it should already be initiated.
    • If the user creates their account on desktop, and does not activate the module, then when they open the homepage on mobile the suggested edits module should already be activated but not show a task preview. But once they tap the preview, the should get the onboarding dialogs.

See also Miro workflow

Mobile (Var C) and Desktop (Var D) flow chart - MOBILE (VAR C) AND DESKTOP (VAR D) FLOW CHART.jpg (3×5 px, 536 KB)

Event Timeline

My initial thought is that we would remove all the "if user in variant C do X, if user in variant D do Y", and instead change that code to "if user is on mobile do X, if user is on desktop do Y". In other words our code would no longer care about the variant assignment a user is in, all users would be presented with a single version of the newcomer tasks experience. @Tgr does that sound like a reasonable way to go about it?

Per @MMiller_WMF there will be other variants in the future, so we should leave in place the little bits of infrastructure that exist for facilitating that (like storing variant assignments in user preferences, utility functions, etc).

Agreed, that sounds like the best way forward.

Change 666466 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Homepage: Remove start module

https://gerrit.wikimedia.org/r/666466

Change 666467 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Homepage: Remove variant A layout grouping

https://gerrit.wikimedia.org/r/666467

Change 666468 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] SiteNoticeGenerator: Remove variant A check

https://gerrit.wikimedia.org/r/666468

Change 666469 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] SuggestedEdits: remove variant A check for mobile preview

https://gerrit.wikimedia.org/r/666469

Change 666470 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] [WIP] Homepage: Show start-editing to desktop users with unactivate SE module

https://gerrit.wikimedia.org/r/666470

Change 666575 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] SuggestedEdits: Pre-initiate for accounts created on mobile

https://gerrit.wikimedia.org/r/666575

@RHo @MMiller_WMF One clarification, do you want the info icon in the suggested edits module to only appear in mobile but never in desktop?

image.png (740×860 px, 48 KB)
image.png (740×2 px, 63 KB)
IMHO it could be nice to show consistently in both.

Another question: Variants A/C would vary the message shown in start editing based on the welcome survey responses. Variant D did not do that. Should the new version of Start Editing also vary the messages based on welcome survey response?

On the topic of start editing module: it should now only ever appear on desktop, it should never be shown on mobile, correct?

Change 666592 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] StartEditing: This module is a desktop only experience now

https://gerrit.wikimedia.org/r/666592

Hi @kostajh replying to a few comments below together:

@RHo @MMiller_WMF One clarification, do you want the info icon in the suggested edits module to only appear in mobile but never in desktop?

image.png (740×860 px, 48 KB)
image.png (740×2 px, 63 KB)
IMHO it could be nice to show consistently in both.

Agree it would be good to show on both.

Another question: Variants A/C would vary the message shown in start editing based on the welcome survey responses. Variant D did not do that. Should the new version of Start Editing also vary the messages based on welcome survey response?

Hmm how much effort is it to vary the messages based on welcome survey responses on Var D? I don't think we have strong indication that different welcome survey responses lead to differences in behaviour so would be fine to not have varying messages if it is high-ish effort. But maybe @nettrom_WMF can comment on whether there is any pattern for newcomer editing based on their welcome survey responses for Q1?

On the topic of start editing module: it should now only ever appear on desktop, it should never be shown on mobile, correct?

I'm a bit confused here as to what the "Start editing module" is. Are these the two onboarding overlays (as shown in

Variant C and D flow.png (2×4 px, 653 KB)
)? Or are these the pre-initiated Suggested Edits module states? If the latter, then no it should not be on mobile.

Hi @kostajh replying to a few comments below together:

@RHo @MMiller_WMF One clarification, do you want the info icon in the suggested edits module to only appear in mobile but never in desktop?

image.png (740×860 px, 48 KB)
image.png (740×2 px, 63 KB)
IMHO it could be nice to show consistently in both.

Agree it would be good to show on both.

Cool, will do that.

Another question: Variants A/C would vary the message shown in start editing based on the welcome survey responses. Variant D did not do that. Should the new version of Start Editing also vary the messages based on welcome survey response?

Hmm how much effort is it to vary the messages based on welcome survey responses on Var D? I don't think we have strong indication that different welcome survey responses lead to differences in behaviour so would be fine to not have varying messages if it is high-ish effort. But maybe @nettrom_WMF can comment on whether there is any pattern for newcomer editing based on their welcome survey responses for Q1?

Zero effort, so I'll keep these for the start editing dialog.

On the topic of start editing module: it should now only ever appear on desktop, it should never be shown on mobile, correct?

I'm a bit confused here as to what the "Start editing module" is. Are these the two onboarding overlays (as shown in

Variant C and D flow.png (2×4 px, 653 KB)
)? Or are these the pre-initiated Suggested Edits module states? If the latter, then no it should not be on mobile.

The start editing module provides the uninitiated suggested edits experience on desktop in variant D, along with the mobile summary view that prompts the user to go through the onboarding flow. Since Suggested Edits should now always be initiated on mobile (no matter whether the account was created on desktop or mobile), then the change here is to remove the mobile summary for start editing but to ensure that the onboarding dialog appears when the user taps on suggested edits.

[..]]

On the topic of start editing module: it should now only ever appear on desktop, it should never be shown on mobile, correct?

I'm a bit confused here as to what the "Start editing module" is. Are these the two onboarding overlays (as shown in

Variant C and D flow.png (2×4 px, 653 KB)
)? Or are these the pre-initiated Suggested Edits module states? If the latter, then no it should not be on mobile.

The start editing module provides the uninitiated suggested edits experience on desktop in variant D, along with the mobile summary view that prompts the user to go through the onboarding flow. Since Suggested Edits should now always be initiated on mobile (no matter whether the account was created on desktop or mobile), then the change here is to remove the mobile summary for start editing but to ensure that the onboarding dialog appears when the user taps on suggested edits.

Perfect, thanks!

@kostajh -- I have a question about this line from the description:

If the users creates their account on mobile, then when they open the homepage on desktop, it should already be initiated.

Ideally, this would be the case if the user creates their account on mobile and visits the homepage. If they simply create their account on mobile, do nothing, and then visit the homepage on desktop, we would prefer for them to go through onboarding. Is that possible?

@kostajh -- I have a question about this line from the description:

If the users creates their account on mobile, then when they open the homepage on desktop, it should already be initiated.

Ideally, this would be the case if the user creates their account on mobile and visits the homepage. If they simply create their account on mobile, do nothing, and then visit the homepage on desktop, we would prefer for them to go through onboarding. Is that possible?

I suppose so, but that also feels edge case enough to not justify the additional work and code complexity needed to handle it. Let me know if you want to go that route though.

A related question:

  • User creates account on desktop (so suggested edits is uninitiated), then logs into mobile at some point and sees that it is initiated, and then goes back to desktop -- I assume SuggestedEdits should be marked as initiated now since they've seen the "initiated" view on Special:Homepage? And would we care if the user on mobile had gone to their homepage or would they have also needed to click on the SE module to see the detail view?

@kostajh -- I have a question about this line from the description:

If the users creates their account on mobile, then when they open the homepage on desktop, it should already be initiated.

Ideally, this would be the case if the user creates their account on mobile and visits the homepage. If they simply create their account on mobile, do nothing, and then visit the homepage on desktop, we would prefer for them to go through onboarding. Is that possible?

I suppose so, but that also feels edge case enough to not justify the additional work and code complexity needed to handle it. Let me know if you want to go that route though.

A related question:

  • User creates account on desktop (so suggested edits is uninitiated), then logs into mobile at some point and sees that it is initiated, and then goes back to desktop -- I assume SuggestedEdits should be marked as initiated now since they've seen the "initiated" view on Special:Homepage? And would we care if the user on mobile had gone to their homepage or would they have also needed to click on the SE module to see the detail view?

IMHO a person should only be seen as having 'initiated' once they see the detail view of the SE module on mobile in this scenario. And if they don't complete the onboarding screens (e.g., on mobile they leave after only seeing onboarding dialog 1) then they are still uninitiated. Is that going to be hard to include?

@kostajh -- I have a question about this line from the description:

If the users creates their account on mobile, then when they open the homepage on desktop, it should already be initiated.

Ideally, this would be the case if the user creates their account on mobile and visits the homepage. If they simply create their account on mobile, do nothing, and then visit the homepage on desktop, we would prefer for them to go through onboarding. Is that possible?

I suppose so, but that also feels edge case enough to not justify the additional work and code complexity needed to handle it. Let me know if you want to go that route though.

A related question:

  • User creates account on desktop (so suggested edits is uninitiated), then logs into mobile at some point and sees that it is initiated, and then goes back to desktop -- I assume SuggestedEdits should be marked as initiated now since they've seen the "initiated" view on Special:Homepage? And would we care if the user on mobile had gone to their homepage or would they have also needed to click on the SE module to see the detail view?

IMHO a person should only be seen as having 'initiated' once they see the detail view of the SE module on mobile in this scenario. And if they don't complete the onboarding screens (e.g., on mobile they leave after only seeing onboarding dialog 1) then they are still uninitiated. Is that going to be hard to include?

That's a change from how we currently do variant C -- currently if the user dismisses the onboarding dialog, we don't show it to them again. I guess we could make that change.

In code, we would have to pretend that the SE module is not actually initiated, and we would just show it as initiated on page load. Then we'd mark it as initiated once the user sees the detail view. I don't know if this would be confusing from an analytics perspective.


We might want to take some more time to define the task description to clarify the various scenarios in more detail, maybe with an updated workflow diagram unless that's a large amount of effort.

@kostajh -- I have a question about this line from the description:

If the users creates their account on mobile, then when they open the homepage on desktop, it should already be initiated.

Ideally, this would be the case if the user creates their account on mobile and visits the homepage. If they simply create their account on mobile, do nothing, and then visit the homepage on desktop, we would prefer for them to go through onboarding. Is that possible?

I suppose so, but that also feels edge case enough to not justify the additional work and code complexity needed to handle it. Let me know if you want to go that route though.

A related question:

  • User creates account on desktop (so suggested edits is uninitiated), then logs into mobile at some point and sees that it is initiated, and then goes back to desktop -- I assume SuggestedEdits should be marked as initiated now since they've seen the "initiated" view on Special:Homepage? And would we care if the user on mobile had gone to their homepage or would they have also needed to click on the SE module to see the detail view?

IMHO a person should only be seen as having 'initiated' once they see the detail view of the SE module on mobile in this scenario. And if they don't complete the onboarding screens (e.g., on mobile they leave after only seeing onboarding dialog 1) then they are still uninitiated. Is that going to be hard to include?

That's a change from how we currently do variant C -- currently if the user dismisses the onboarding dialog, we don't show it to them again. I guess we could make that change.

In code, we would have to pretend that the SE module is not actually initiated, and we would just show it as initiated on page load. Then we'd mark it as initiated once the user sees the detail view. I don't know if this would be confusing from an analytics perspective.


We might want to take some more time to define the task description to clarify the various scenarios in more detail, maybe with an updated workflow diagram unless that's a large amount of effort.

I gave it a try here in case that helps:

Mobile (Var C) and Desktop (Var D) flow chart - MOBILE (VAR C) AND DESKTOP (VAR D) FLOW CHART.jpg (3×5 px, 536 KB)

You can also view and make comments on the Miro board if that is preferred: https://miro.com/app/board/o9J_lSBS7E4=/

Change 666466 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Homepage: Remove start module

https://gerrit.wikimedia.org/r/666466

Change 666467 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Homepage: Remove variant A layout grouping

https://gerrit.wikimedia.org/r/666467

Change 666468 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] SiteNoticeGenerator: Remove variant A check

https://gerrit.wikimedia.org/r/666468

Change 666469 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] SuggestedEdits: remove variant A check for mobile preview

https://gerrit.wikimedia.org/r/666469

Change 650558 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Remove variant A code

https://gerrit.wikimedia.org/r/650558

Change 650558 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Remove variant A code

https://gerrit.wikimedia.org/r/650558

I gave it a try here in case that helps:

Mobile (Var C) and Desktop (Var D) flow chart - MOBILE (VAR C) AND DESKTOP (VAR D) FLOW CHART.jpg (3×5 px, 536 KB)

You can also view and make comments on the Miro board if that is preferred: https://miro.com/app/board/o9J_lSBS7E4=/

Thanks, that is very helpful!

@RHo for the user who arrives at Special:Homepage on mobile and hasn't activated the module, which of these three states (mobile summary contents as well as the drawer text) should they see?

  1. the existing variant D version

image.png (1×790 px, 143 KB)

  1. The existing variant C version

image.png (1×790 px, 212 KB)

  1. Variant C version but when no articles are found, so there's a more generic button there rather than a specific article

image.png (1×790 px, 166 KB)

If we go with option 3, if/when we do T264460: Enable users to open the first suggested article directly from the mobile task preview, would tapping an article without going through the onboarding dialogs be considered an activation of the module?

kostajh updated the task description. (Show Details)

@RHo for the user who arrives at Special:Homepage on mobile and hasn't activated the module, which of these three states (mobile summary contents as well as the drawer text) should they see?

  1. the existing variant D version

image.png (1×790 px, 143 KB)

  1. The existing variant C version

image.png (1×790 px, 212 KB)

  1. Variant C version but when no articles are found, so there's a more generic button there rather than a specific article

image.png (1×790 px, 166 KB)

If we go with option 3, if/when we do T264460: Enable users to open the first suggested article directly from the mobile task preview, would tapping an article without going through the onboarding dialogs be considered an activation of the module?

Hi @kostajh - I think we should go with option 3. And if/when we do T264460: Enable users to open the first suggested article directly from the mobile task preview, this card should still be shown even when there are articles found. I have added a note on the T264460: Enable users to open the first suggested article directly from the mobile task preview that when there are no SE articles, this same generic card is still shown.

kostajh renamed this task from Variant tests: all mobile users get Variant C, all desktop users get Variant D to Variant tests: Create hybrid of Variant C / D.Mar 8 2021, 10:58 AM
kostajh triaged this task as High priority.Mar 8 2021, 11:07 AM

@RHo what should the welcome tour look like for desktop users who haven't activated the module yet?

Should it be like this (launch dialog with info about SE module):

image.png (916×1 px, 195 KB)

image.png (916×1 px, 177 KB)

Or like this (no launch of info dialog after clicking done):

image.png (916×1 px, 196 KB)

As a reminder, per an earlier comment in this thread, we will have the info icon on both mobile and desktop versions of the module which can also launch the info dialog about suggested edits.

Change 669872 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] [WIP] Variant C/D hybrid

https://gerrit.wikimedia.org/r/669872

@RHo for the user who arrives at Special:Homepage on mobile and hasn't activated the module, which of these three states (mobile summary contents as well as the drawer text) should they see?

  1. the existing variant D version

image.png (1×790 px, 143 KB)

  1. The existing variant C version

image.png (1×790 px, 212 KB)

  1. Variant C version but when no articles are found, so there's a more generic button there rather than a specific article

image.png (1×790 px, 166 KB)

If we go with option 3, if/when we do T264460: Enable users to open the first suggested article directly from the mobile task preview, would tapping an article without going through the onboarding dialogs be considered an activation of the module?

Hi @kostajh - I think we should go with option 3. And if/when we do T264460: Enable users to open the first suggested article directly from the mobile task preview, this card should still be shown even when there are articles found. I have added a note on the T264460: Enable users to open the first suggested article directly from the mobile task preview that when there are no SE articles, this same generic card is still shown.

I was thinking about this some more. If we go with option 3, aren't we implementing a key part of variant D experience for mobile users? Like in variant D, they won't see an actual task to work on in the mobile summary, and then they'll have to go through some onboarding dialogs (they are not optional in the task description AIUI) for task/topic selection.

Test wiki created on Patch Demo by KHarlan (WMF) using patch(es) linked to this task:

https://patchdemo.wmflabs.org/wikis/3b0fdd546dc772f8571f46462d2bc238/w/

@RHo / @MMiller_WMF could you please have a look at the patchdemo site here and let me know how far off I am from what you envision?

@RHo what should the welcome tour look like for desktop users who haven't activated the module yet?
[...]
Or like this (no launch of info dialog after clicking done):

image.png (916×1 px, 196 KB)

Definitely this one without info dialog (since it is just repeating the content in the pre-activated SE module. Thanks!

As a reminder, per an earlier comment in this thread, we will have the info icon on both mobile and desktop versions of the module which can also launch the info dialog about suggested edits.

  • Yes, however, I expect the info icon only appear on the activated SE module, not on the pre-activated versions of the module shown above - is that right?

@RHo for the user who arrives at Special:Homepage on mobile and hasn't activated the module, which of these three states (mobile summary contents as well as the drawer text) should they see?

  1. the existing variant D version

image.png (1×790 px, 143 KB)

  1. The existing variant C version

image.png (1×790 px, 212 KB)

  1. Variant C version but when no articles are found, so there's a more generic button there rather than a specific article

image.png (1×790 px, 166 KB)

If we go with option 3, if/when we do T264460: Enable users to open the first suggested article directly from the mobile task preview, would tapping an article without going through the onboarding dialogs be considered an activation of the module?

Hi @kostajh - I think we should go with option 3. And if/when we do T264460: Enable users to open the first suggested article directly from the mobile task preview, this card should still be shown even when there are articles found. I have added a note on the T264460: Enable users to open the first suggested article directly from the mobile task preview that when there are no SE articles, this same generic card is still shown.

I was thinking about this some more. If we go with option 3, aren't we implementing a key part of variant D experience for mobile users? Like in variant D, they won't see an actual task to work on in the mobile summary, and then they'll have to go through some onboarding dialogs (they are not optional in the task description AIUI) for task/topic selection.

Derp, you're right @kostajh. I got caught up in the part where they *do* still have to go through the onboarding dialogs as was the case in var C mobile, but newcomers did see the first task in the preview. Sorry about that and please do go with your option 2 above.

As a reminder, per an earlier comment in this thread, we will have the info icon on both mobile and desktop versions of the module which can also launch the info dialog about suggested edits.

  • Yes, however, I expect the info icon only appear on the activated SE module, not on the pre-activated versions of the module shown above - is that right?

✔ that's how it's set up, yes.

Derp, you're right @kostajh. I got caught up in the part where they *do* still have to go through the onboarding dialogs as was the case in var C mobile, but newcomers did see the first task in the preview. Sorry about that and please do go with your option 2 above.

OK cool, so some follow-up questions per option 2:

  • user clicks "Continue", they should see the onboarding / start editing dialog *without* the topic and task selection checkboxes, and then be taken into Suggested Edits detail view at which point we consider the module "activated", right?
  • user dismisses the drawer then task on the task card or the "see all" suggestion button -- for both the user should go directly to the article, or to the detail view of suggested edits, but they should not see the onboarding / start editing dialog, is that correct?

Change 670096 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Homepage: Check welcome notice seen preference

https://gerrit.wikimedia.org/r/670096

Change 670096 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@master] Homepage: Check welcome notice seen preference

https://gerrit.wikimedia.org/r/670096

@nettrom_WMF @MMiller_WMF as a heads-up, a patch for removing variant A code unintentionally removed code that checked if the user had already seen the welcome drawer on mobile visits to Special:Homepage. So if you see higher impressions and interactions with that drawer from March 4 to March 9, that's why. We'll be backporting a fix today. The patch was merged just after branch cut for wmf.33 😅 so this didn't make it to production wikis. We are merging the backported fix to wmf.34 which goes to group2 on Thursday.

Change 670107 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.34] Homepage: Check welcome notice seen preference

https://gerrit.wikimedia.org/r/670107

Change 670096 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Homepage: Check welcome notice seen preference

https://gerrit.wikimedia.org/r/670096

Change 670107 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@wmf/1.36.0-wmf.34] Homepage: Check welcome notice seen preference

https://gerrit.wikimedia.org/r/670107

As a reminder, per an earlier comment in this thread, we will have the info icon on both mobile and desktop versions of the module which can also launch the info dialog about suggested edits.

  • Yes, however, I expect the info icon only appear on the activated SE module, not on the pre-activated versions of the module shown above - is that right?

✔ that's how it's set up, yes.

Great, thanks for confirming.

Derp, you're right @kostajh. I got caught up in the part where they *do* still have to go through the onboarding dialogs as was the case in var C mobile, but newcomers did see the first task in the preview. Sorry about that and please do go with your option 2 above.

OK cool, so some follow-up questions per option 2:

  • user clicks "Continue", they should see the onboarding / start editing dialog *without* the topic and task selection checkboxes, and then be taken into Suggested Edits detail view at which point we consider the module "activated", right?
  • user dismisses the drawer then task on the task card or the "see all" suggestion button -- for both the user should go directly to the article, or to the detail view of suggested edits, but they should not see the onboarding / start editing dialog, is that correct?

Yes and yes. 👍🏻

Test wiki created on Patch Demo by KHarlan (WMF) using patch(es) linked to this task:

https://patchdemo.wmflabs.org/wikis/8607949f1969be036095dfa696b63f61/w/

@RHo would you mind giving the patch demo above a try to see if I've implemented this task correctly?

Test wiki created on Patch Demo by KHarlan (WMF) using patch(es) linked to this task:

https://patchdemo.wmflabs.org/wikis/8607949f1969be036095dfa696b63f61/w/

@RHo would you mind giving the patch demo above a try to see if I've implemented this task correctly?

Thanks @kostajh - Is there a console command to reset the welcome banners/drawers? I seem to already have activated the module and can't bring the welcome back

Test wiki created on Patch Demo by KHarlan (WMF) using patch(es) linked to this task:

https://patchdemo.wmflabs.org/wikis/8607949f1969be036095dfa696b63f61/w/

@RHo would you mind giving the patch demo above a try to see if I've implemented this task correctly?

Thanks @kostajh - Is there a console command to reset the welcome banners/drawers? I seem to already have activated the module and can't bring the welcome back

@RHo Yes indeed! Try:

new mw.Api().saveOptions( {
  "growthexperiments-tour-homepage-welcome": "0",
  "growthexperiments-homepage-suggestededits-activated": "0"
} ).done( () => location.reload() )

Test wiki created on Patch Demo by KHarlan (WMF) using patch(es) linked to this task:

https://patchdemo.wmflabs.org/wikis/8607949f1969be036095dfa696b63f61/w/

@RHo would you mind giving the patch demo above a try to see if I've implemented this task correctly?

Thanks @Kosta - I have tested the following sequence to check through the 'main scenarios' for Desktop --> Mobile, and vice versa and the only thing that I noticed was amiss is that the mobile welcome drawer is not shown when going from Desktop to Mobile (without clicking on the welcome banner on Desktop or activating the module) in scenarios A & B below:

A. Desktop --> Mobile without initiating on Desktop workflowResult
1. Go to Desktop welcome pop-up show, page has pre-initiated SE module
image.png (1×2 px, 456 KB)
2. Switch to Mobile welcome drawer on mobile is NOT shown
image.png (1×736 px, 84 KB)
B. Mobile --> Desktop without doing mobile onboarding flowResult
1. Go to Mobile welcome drawer on mobile over a SE preview with a task
image.png (1×764 px, 104 KB)
2. Switch to Desktop welcome pop-up on desktop is shown
image.png (1×2 px, 444 KB)
3. Switch back to Mobile welcome drawer on mobile is NOT shown on this second time (even though the welcome drawer/pop-up is still not selected, nor SE module activated on Desktop)
C. Desktop + initiated --> Mobile workflowResult
1. Go to Desktop welcome banner over pre-initiated module
image.png (1×2 px, 456 KB)
2. Close welcome and initiate SE module on desktop SE module initiated with info icon
image.png (1×1 px, 189 KB)
3. Switch to Mobile SE module initiated (no welcome drawer)
image.png (1×756 px, 460 KB)
D. Mobile welcome completed --> Desktop workflowResult
1. Go to Mobile welcome drawer on mobile over a SE preview with a task
image.png (1×764 px, 104 KB)
2. Tap "Continue" on welcome drawer Read-only onboarding
image.png (1×758 px, 122 KB)
image.png (1×752 px, 117 KB)
3. Complete onboarding SE module initiated
image.png (1×746 px, 94 KB)
4. Switch to Desktop SE module is initiated, with the same task
image.png (1×2 px, 373 KB)

Test wiki created on Patch Demo by KHarlan (WMF) using patch(es) linked to this task:

https://patchdemo.wmflabs.org/wikis/8607949f1969be036095dfa696b63f61/w/

@RHo would you mind giving the patch demo above a try to see if I've implemented this task correctly?

Thanks @Kosta - I have tested the following sequence to check through the 'main scenarios' for Desktop --> Mobile, and vice versa and the only thing that I noticed was amiss is that the mobile welcome drawer is not shown when going from Desktop to Mobile (without clicking on the welcome banner on Desktop or activating the module) in scenarios A & B below:

Thanks for the thorough review @RHo! I can adjust that so that the welcome drawer is shown in those scenarios, but wanted to clarify the rules first.

When we first rolled out the guided tour on desktop (T222852) and welcome drawer on mobile (T258010) the rules were that after a user sees the drawer on mobile, they should not see the drawer on mobile again, and they should not see the guided tour on desktop, and same for desktop: after a user sees the guided tour on desktop they shouldn't see it again on desktop nor should they see the drawer on mobile.

I would need to lightly adjust the patch to implement to those rules, but that would be fairly easy to do.

It sounds like you might be proposing something else, though, where the visibility of the welcome drawer on mobile and guided tour on desktop might depend on whether the user has interacted with the drawer/tour and/or whether the user has activated the module?

[...]
Thanks for the thorough review @RHo! I can adjust that so that the welcome drawer is shown in those scenarios, but wanted to clarify the rules first.

When we first rolled out the guided tour on desktop (T222852) and welcome drawer on mobile (T258010) the rules were that after a user sees the drawer on mobile, they should not see the drawer on mobile again, and they should not see the guided tour on desktop, and same for desktop: after a user sees the guided tour on desktop they shouldn't see it again on desktop nor should they see the drawer on mobile.
I would need to lightly adjust the patch to implement to those rules, but that would be fairly easy to do.
It sounds like you might be proposing something else, though, where the visibility of the welcome drawer on mobile and guided tour on desktop might depend on whether the user has interacted with the drawer/tour and/or whether the user has activated the module?

Thanks @kostajh, I wanted to clarify if your small adjustment is so that the welcome drawer/pop-up shows up on the relevant platform if a person did not interact at all with the welcome drawer/pop-up on the other platform? That's the only thing that was unexpected for me. I agree that the visibility of the welcome message should not be dependent on the activation-state of the SE module.

[...]
Thanks for the thorough review @RHo! I can adjust that so that the welcome drawer is shown in those scenarios, but wanted to clarify the rules first.

When we first rolled out the guided tour on desktop (T222852) and welcome drawer on mobile (T258010) the rules were that after a user sees the drawer on mobile, they should not see the drawer on mobile again, and they should not see the guided tour on desktop, and same for desktop: after a user sees the guided tour on desktop they shouldn't see it again on desktop nor should they see the drawer on mobile.
I would need to lightly adjust the patch to implement to those rules, but that would be fairly easy to do.
It sounds like you might be proposing something else, though, where the visibility of the welcome drawer on mobile and guided tour on desktop might depend on whether the user has interacted with the drawer/tour and/or whether the user has activated the module?

Thanks @kostajh, I wanted to clarify if your small adjustment is so that the welcome drawer/pop-up shows up on the relevant platform if a person did not interact at all with the welcome drawer/pop-up on the other platform? That's the only thing that was unexpected for me. I agree that the visibility of the welcome message should not be dependent on the activation-state of the SE module.

I guess it's unclear to me what counts as an interaction. I'll try to list out possible permutations:

  1. User is on mobile and sees welcome drawer. User taps outside the welcome drawer but does nothing else. Then the user reloads Special:Homepage. Should the user see the welcome drawer again? If they go to Homepage on desktop, should they see the guided tour?
  2. User is on mobile and sees welcome drawer. User taps on the welcome drawer and the onboarding dialog launches, then the user cancels and reloads the homepage. Should they see the welcome drawer again? What if they go to homepage on desktop, should they see the guided tour?
  3. User is on mobile and sees welcome drawer but neither taps outside nor taps on the welcome drawer, let's assume they just press back on the browser. Next time they are on homepage on mobile or desktop, should they see the drawer again?

When we first rolled out the guided tour on desktop (T222852) and welcome drawer on mobile (T258010) the rules were that after a user sees the drawer on mobile, they should not see the drawer on mobile again, and they should not see the guided tour on desktop, and same for desktop: after a user sees the guided tour on desktop they shouldn't see it again on desktop nor should they see the drawer on mobile.

Overall what I'm trying to clarify is that it looks like we are switching from the current rules that say "only show the welcome drawer/tour a single time, no matter what" to the opposite: keep showing the welcome drawer and tour until the user interacts with it (clicks the call to action) on mobile or desktop, and then don't show it to them again.

[...]
Thanks for the thorough review @RHo! I can adjust that so that the welcome drawer is shown in those scenarios, but wanted to clarify the rules first.

When we first rolled out the guided tour on desktop (T222852) and welcome drawer on mobile (T258010) the rules were that after a user sees the drawer on mobile, they should not see the drawer on mobile again, and they should not see the guided tour on desktop, and same for desktop: after a user sees the guided tour on desktop they shouldn't see it again on desktop nor should they see the drawer on mobile.
I would need to lightly adjust the patch to implement to those rules, but that would be fairly easy to do.
It sounds like you might be proposing something else, though, where the visibility of the welcome drawer on mobile and guided tour on desktop might depend on whether the user has interacted with the drawer/tour and/or whether the user has activated the module?

Thanks @kostajh, I wanted to clarify if your small adjustment is so that the welcome drawer/pop-up shows up on the relevant platform if a person did not interact at all with the welcome drawer/pop-up on the other platform? That's the only thing that was unexpected for me. I agree that the visibility of the welcome message should not be dependent on the activation-state of the SE module.

I guess it's unclear to me what counts as an interaction. I'll try to list out possible permutations:

  1. User is on mobile and sees welcome drawer. User taps outside the welcome drawer but does nothing else. Then the user reloads Special:Homepage. Should the user see the welcome drawer again? If they go to Homepage on desktop, should they see the guided tour?
  2. User is on mobile and sees welcome drawer. User taps on the welcome drawer and the onboarding dialog launches, then the user cancels and reloads the homepage. Should they see the welcome drawer again? What if they go to homepage on desktop, should they see the guided tour?
  3. User is on mobile and sees welcome drawer but neither taps outside nor taps on the welcome drawer, let's assume they just press back on the browser. Next time they are on homepage on mobile or desktop, should they see the drawer again?

Ah yes, the scenarios are helpful. I am really only talking about the 3rd scenario as expecting the welcome drawer to come up when there has been no interaction with the welcome drawer/pop-up, where interactions include both tapping to view more and any dismiss action. Since tapping outside of the drawer on mobile is the only way to dismiss instead of the CTA to "continue" seeing onboarding, that IMO is counted as interacting so therefore the reload on mobile shouldn't bring it up again.

When we first rolled out the guided tour on desktop (T222852) and welcome drawer on mobile (T258010) the rules were that after a user sees the drawer on mobile, they should not see the drawer on mobile again, and they should not see the guided tour on desktop, and same for desktop: after a user sees the guided tour on desktop they shouldn't see it again on desktop nor should they see the drawer on mobile.

Overall what I'm trying to clarify is that it looks like we are switching from the current rules that say "only show the welcome drawer/tour a single time, no matter what" to the opposite: keep showing the welcome drawer and tour until the user interacts with it (clicks the call to action) on mobile or desktop, and then don't show it to them again.

Does what I said above makes sense and help clarify? I guess I am asking for the current rule to be amended to show the welcome drawer until it is dismissed or a call to action is selected; because I don't consider navigating away from the page to 'count' as seeing the welcome. Having said that, I trust your call if this is a level of change and effort which substantially increases the scope of the work (ie. feel free to push back and I will adjust the flow chart and my expectations accordingly).

Does what I said above makes sense and help clarify? I guess I am asking for the current rule to be amended to show the welcome drawer until it is dismissed or a call to action is selected; because I don't consider navigating away from the page to 'count' as seeing the welcome. Having said that, I trust your call if this is a level of change and effort which substantially increases the scope of the work (ie. feel free to push back and I will adjust the flow chart and my expectations accordingly).

Got it, yes. So I will rework such that the growthexperiments-tour-homepage-welcome preference will be set to 1 if the user taps outside the welcome drawer or on the drawer.

Change 669872 merged by jenkins-bot:
[mediawiki/extensions/GrowthExperiments@master] Variant C/D hybrid

https://gerrit.wikimedia.org/r/669872

Change 666592 abandoned by Kosta Harlan:
[mediawiki/extensions/GrowthExperiments@master] StartEditing: This module is a desktop only experience now

Reason:

https://gerrit.wikimedia.org/r/666592

Change 666470 abandoned by Kosta Harlan:
[mediawiki/extensions/GrowthExperiments@master] Homepage: Show start-editing to desktop users with unactivated SE module

Reason:

https://gerrit.wikimedia.org/r/666470

Change 666575 abandoned by Kosta Harlan:
[mediawiki/extensions/GrowthExperiments@master] SuggestedEdits: Pre-initiate for accounts created on mobile

Reason:

https://gerrit.wikimedia.org/r/666575

Etonkovidova updated the task description. (Show Details)

Checked after T277727 fix - all scenarios listed seem to be in place.