Page MenuHomePhabricator
Create Task
Maniphest T281356

track number of edits per namespace over time
Closed, ResolvedPublic5 Estimated Story Points
Actions

Description

Problem:
We are tracking the number of editors who make at least 1/5/100 edits in the past 30 days split by namespace at https://grafana.wikimedia.org/d/000000162/wikidata-site-stats?orgId=1.
We now also want a graph showing the number of edits per namespace over time to better understand how many edits are happening across the namespaces. This graph can be added to https://grafana.wikimedia.org/d/000000162/wikidata-site-stats?orgId=1 as well.

Acceptance criteria:

Notes:

  • See the total edits panel on the same board for similar tracking. The code is linked in the little info i in the corner of that panel.

Details

SubjectRepoBranchLines +/-
Make recent_changes_by_namespace track all namespacesanalytics/wmde/scriptsproduction+4 -15
Make recent_changes_by_namespace track all namespacesanalytics/wmde/scriptsmaster+4 -15
Customize query in gerrit

Related Objects

Event Timeline

• amy_rc created this task.Apr 28 2021, 11:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2021, 11:41 AM
• amy_rc moved this task from Incoming to Needs Wikidata PM Work on the Wikidata-Campsite board.Apr 28 2021, 11:41 AM
Lydia_Pintscher renamed this task from track editor numbers split by 'edits' to track edits per namespace over time.Apr 28 2021, 7:08 PM
Lydia_Pintscher renamed this task from track edits per namespace over time to track number of edits per namespace over time.
Lydia_Pintscher triaged this task as Medium priority.
Lydia_Pintscher updated the task description. (Show Details)
Lydia_Pintscher moved this task from Needs Wikidata PM Work to Unconnected Stories on the Wikidata-Campsite board.
DVrandecic subscribed.Apr 29 2021, 12:54 AM
Lydia_Pintscher updated the task description. (Show Details)May 4 2021, 8:29 AM
Lydia_Pintscher updated the task description. (Show Details)May 4 2021, 8:31 AM
darthmon_wmde updated the task description. (Show Details)May 5 2021, 9:24 AM
darthmon_wmde updated the task description. (Show Details)
darthmon_wmde set the point value for this task to 5.May 5 2021, 9:29 AM
darthmon_wmde moved this task from Unconnected Stories to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.
darthmon_wmde edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.May 5 2021, 10:15 AM
Ladsgroup claimed this task.May 17 2021, 8:57 AM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptMay 17 2021, 8:57 AM
gerritbot added a comment.May 17 2021, 9:07 AM

Change 692259 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[analytics/wmde/scripts@master] Introduce edit_count_by_namespace metric

https://gerrit.wikimedia.org/r/692259

gerritbot added a project: Patch-For-Review.May 17 2021, 9:07 AM
Ladsgroup moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 17 2021, 9:08 AM
Ladsgroup added a comment.May 17 2021, 9:11 AM

https://grafana.wikimedia.org/d/000000162/wikidata-site-stats?viewPanel=23&orgId=1

Maintenance_bot moved this task from Incoming to In progress on the User-Ladsgroup board.May 17 2021, 9:15 AM
Michael subscribed.May 17 2021, 4:41 PM

Now that I'm trying to review the code, I got confused: How is this any different from what was added in T218901: Track number of Wikidata edits by namespace?

Ladsgroup added a subscriber: Lydia_Pintscher.May 18 2021, 6:07 AM

I didn't know that existed...

Here's a comparison of values:

image.png (542×356 px, 52 KB)

Notable differences I noticed:

  • The values of content namespaces don't differ that much, it can be just due to different timing of the query
  • The values of talk namespaces differ a lot. The old script says only 20 edits happened on talk pages for that day, while the new script says 1K edits happened.
  • The old code is limited to just a set of namespaces and it's pretty small. Maybe we can just remove that?

These questions need answering with @Lydia_Pintscher I assume.

Michael added a comment.EditedMay 18 2021, 1:08 PM
In T281356#7094642, @Ladsgroup wrote:

[...]

Notable differences I noticed:

  • [...]
  • The values of talk namespaces differ a lot. The old script says only 20 edits happened on talk pages for that day, while the new script says 1K edits happened.
  • [...]

The existing SQL of the old script includes this line in its WHERE clause:

AND rc_source = 'mw.edit'

I'm not perfectly sure what it does, but maybe that explains the lower numbers?

Edit: rc_source records the type of the change. So it would seem to make sense to, for example, exclude log actions. Though that documentation is somewhat vague on the set of possible values and whether we maybe want to record more than only mw.edit?

Jakob_WMDE moved this task from Peer Review to Stalled/Waiting on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 20 2021, 9:17 AM
Lydia_Pintscher added a comment.May 23 2021, 5:08 PM
In T281356#7094642, @Ladsgroup wrote:

I didn't know that existed...

Here's a comparison of values:

image.png (542×356 px, 52 KB)

Notable differences I noticed:

  • The values of content namespaces don't differ that much, it can be just due to different timing of the query
  • The values of talk namespaces differ a lot. The old script says only 20 edits happened on talk pages for that day, while the new script says 1K edits happened.
  • The old code is limited to just a set of namespaces and it's pretty small. Maybe we can just remove that?

These questions need answering with @Lydia_Pintscher I assume.

Sorry. I thought we had added all of this to this ticket during story time... We discussed this in story time because I wasn't aware of that board initially. We said the existing one doesn't quite do what we want, mainly due to it not covering all namespaces. We said we'd remove the existing one from https://grafana.wikimedia.org/d/000000170/wikidata-edits?orgId=1&refresh=1m and add a new one covering all namespaces to https://grafana.wikimedia.org/d/000000162/wikidata-site-stats?orgId=1.

Lydia_Pintscher added a comment.May 23 2021, 5:09 PM
In T281356#7095576, @Michael wrote:
In T281356#7094642, @Ladsgroup wrote:

[...]

Notable differences I noticed:

  • [...]
  • The values of talk namespaces differ a lot. The old script says only 20 edits happened on talk pages for that day, while the new script says 1K edits happened.
  • [...]

The existing SQL of the old script includes this line in its WHERE clause:

AND rc_source = 'mw.edit'

I'm not perfectly sure what it does, but maybe that explains the lower numbers?

Edit: rc_source records the type of the change. So it would seem to make sense to, for example, exclude log actions. Though that documentation is somewhat vague on the set of possible values and whether we maybe want to record more than only mw.edit?

Hmmm I'd say we don't want to count log actions as edits for the purpose of the stats required here.

Lydia_Pintscher moved this task from Stalled/Waiting to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 23 2021, 5:09 PM
Michael added a comment.May 25 2021, 8:36 AM

Thank you for your explanation, I didn't get that from the acceptance criteria above.

In T281356#7106166, @Lydia_Pintscher wrote:

[...]
We said the existing one doesn't quite do what we want, mainly due to it not covering all namespaces.
[...]

Out of curiosity: which namespace do you want to see that you are missing in the existing one?

/me goes back to the patch and gives proper feedback.

Lydia_Pintscher added a comment.May 25 2021, 9:49 AM
In T281356#7110359, @Michael wrote:

Out of curiosity: which namespace do you want to see that you are missing in the existing one?

So the existing one only tracks the entity namespaces. Things like the project/Wikidata and Help namespaces are also pretty useful to understand.

Michael added a comment.May 25 2021, 9:50 AM

Ah, I see. Thanks!

gerritbot added a comment.May 26 2021, 9:59 AM

Change 692259 merged by jenkins-bot:

[analytics/wmde/scripts@master] Make recent_changes_by_namespace track all namespaces

https://gerrit.wikimedia.org/r/692259

Ladsgroup mentioned this in rADESdf9b6ad64923: Make recent_changes_by_namespace track all namespaces.May 26 2021, 10:03 AM
Silvan_WMDE subscribed.May 26 2021, 10:08 AM

The patch looks good - merged. Two minor issues are remaining:

  • the third AC "[ ] the graph is deleted from the board it comes from" isn't met yet
  • sortByMaxima() is inconsistent with the three panels above, that also list all namespaces but do sortByName(true)
Maintenance_bot removed a project: Patch-For-Review.May 26 2021, 10:11 AM
gerritbot added a comment.May 26 2021, 10:21 AM

Change 695040 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[analytics/wmde/scripts@production] Make recent_changes_by_namespace track all namespaces

https://gerrit.wikimedia.org/r/695040

gerritbot added a project: Patch-For-Review.May 26 2021, 10:21 AM

Change 695040 merged by jenkins-bot:

[analytics/wmde/scripts@production] Make recent_changes_by_namespace track all namespaces

https://gerrit.wikimedia.org/r/695040

Ladsgroup mentioned this in rADESd6b74745abc8: Make recent_changes_by_namespace track all namespaces.May 26 2021, 10:26 AM
Maintenance_bot removed a project: Patch-For-Review.May 26 2021, 11:10 AM
Ladsgroup added a comment.May 26 2021, 1:45 PM
In T281356#7114706, @Silvan_WMDE wrote:

The patch looks good - merged. Two minor issues are remaining:

  • the third AC "[ ] the graph is deleted from the board it comes from" isn't met yet
  • sortByMaxima() is inconsistent with the three panels above, that also list all namespaces but do sortByName(true)

both are done. Please take a look

Silvan_WMDE added a comment.May 26 2021, 1:52 PM
          #
  #      ##
  #     # #
#####     #
  #       #
  #       #
        #####
Silvan_WMDE moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.May 26 2021, 1:52 PM
Addshore subscribed.May 28 2021, 9:45 AM

I will leave this for @Lydia_Pintscher to verify!

• amy_rc closed this task as Resolved.May 31 2021, 7:56 AM
• amy_rc moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

🎉 works \o/

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits