For example, from google:
My guess is this might be somehow related to framing.
However, it uses:
webHost : window.location.hostname
which in theory should work even in a framing scenario, unless the context of the JavaScript was somehow lost. On http://jsfiddle.net/596mX/ the top web host is htp://jsfiddle.net, the frame host is http://fiddle.jshell.net, and it alerts the latter's hostname.
Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=55449
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Dbrant | T117631 MobileWikiAppDailyStats should not count Googlebot | |||
| Resolved | • fdans | T67508 Don't accept data from automated bots in Event Logging |
Thanks for bringing this up and for linking the other ticket, this is an issue that we should take very seriously as people crunching the data will rarely remember to filter by webHost (assuming filtering by wiki is sufficient) which will cause the inclusion of a lot of bogus events caused by test instances on labs or legitimate but spurious events caused by users with proxies or other factors.
I think we should try and enforce stricter validation for the webHost field to only accept events from a list of known hostnames and set up monitoring of events that fail validation on the webHost field.
Steven also suggested it could be related to Chrome's automatic translation feature.
My tests indicate that still uses the original hostname in Chromium 34.0.1847.132 Debian 7.5 (265804) , but this behavior might vary by version or something.
We can do this work now that we are parsing the user agent for incoming EventLogging data.
We need to: add a "self-identified" bot filter for all incoming data. We just need to use the same regex than the pageview code uses: https://github.com/wikimedia/analytics-refinery-source/blob/master/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/Webrequest.java#L59
It is unfortunate that this regex would need to be duplicated on both EL and pageview data.
This would take care of events that asent by , say, a google bot crawling the android application but it will not take care of other issues that have to do with javascript. Premise of ticket is not super clear on this regard.
How do we keep track of bot-identified traffic?
Sending data to graphite
We can publish bot-dentified events to a bot schema that is similar to EventError that is only pushed to hadoop (not to db): https://meta.wikimedia.org/wiki/Schema:EventError
Change 350234 had a related patch set uploaded (by Fdans):
[eventlogging@master] Flag requests sent by spiders/bots using AutomatedRequest schema
Change 350235 had a related patch set uploaded (by Fdans):
[operations/puppet@production] Add AutomatedRequest to schema black list
Change 350234 merged by Ottomata:
[eventlogging@master] Mark events as bots if they self-identify
Change 352579 had a related patch set uploaded (by Fdans; owner: Fdans):
[eventlogging@master] Add handler for event filters
Change 350235 abandoned by Fdans:
Add AutomatedRequest to schema black list
Reason:
This is no longer needed since we've changed the approach for the task
Change 352582 had a related patch set uploaded (by Fdans; owner: Fdans):
[operations/puppet@production] Add bot filter to mysql consumer
Change 352579 merged by Ottomata:
[eventlogging@master] Add handler for event filters
Change 352582 merged by Ottomata:
[operations/puppet@production] Add bot filter to mysql consumer
Change 355238 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Remove use of is_not_bot filter in eventlogging mysql until code is fixed and change is cleared (announced)
Change 355238 merged by Ottomata:
[operations/puppet@production] Remove use of is_not_bot filter in eventlogging mysql until code is fixed and change is cleared (announced)
@fdans: Yes, since this is going to affect the results of various queries (even though it's by improving their accuracy), people working with them should be notified. I think a quick note to Analytics-l would be justified.
Change 355482 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Use is_not_bot filter function for eventlogging mysql consumer
Change 355482 merged by Ottomata:
[operations/puppet@production] Use is_not_bot filter function for eventlogging mysql consumer
How will this affect EventLogging calls made from PHP (which might need to be recorded whether the user used some bot framework or not)?
@Tgr: All calls go through varnish, there are no direct posts from php anymore (it is been a while), thus they are all process equally.
Except that requests initiated from the backed will be rejected because the useragent is MediaWiki (or something similar). I feel this hasn't really been thought through.
Note that even if EventLogging::logEvent would forward the user agent (which it currently doesn't), filtering on that would still make no sense for schemas such as Pingback or CommandInvocation.
mysql:research@analytics-store.eqiad.wmnet [log]> select timestamp, count(*) from MediaWikiPingback_15781718 group by substr(timestamp, 1, 8) order by timestamp desc limit 50; +----------------+----------+ | timestamp | count(*) | +----------------+----------+ | 20170524001855 | 125 | | 20170523003847 | 148 | | 20170522000909 | 140 | | 20170521002601 | 131 | | 20170520000349 | 118 | | 20170519001419 | 179 |
Change 356423 had a related patch set uploaded (by Gergő Tisza; owner: Gergő Tisza):
[operations/puppet@production] Revert "Use is_not_bot filter function for eventlogging mysql consumer"
Note that even if EventLogging::logEvent would forward the user agent (which it currently doesn't)
To recap from IRC. Server side events DO forward UA: https://github.com/wikimedia/mediawiki-extensions-EventLogging/blob/master/includes/EventLogging.php#L67
However there are 2 schemas that do not log using this method: MWPingback and CommandLine, of these we think Commandline is not used.
Change 356423 abandoned by Gergő Tisza:
Revert "Use is_not_bot filter function for eventlogging mysql consumer"
Reason:
I misdiagnosed the problem, it only affects a few schemas. There are more useful ways to handle it.
What about logging from the job queue (which could in theory happen for PageDeletion etc when some job creates/moves/deletes pages)? That will probably have a bot UA too. (It will probably be rare though and not sure whether the people owning the schemas would want to log it in the first place.)
Re IRC question: for the MediaWikiPingback schema, if the UA is not recorded, it won't be missed. All the information that could be possibly learned from it (MW version, or PHP version) is already included in the payload.
Change 356624 had a related patch set uploaded (by Fdans; owner: Fdans):
[eventlogging@master] Add is_mediawiki property to UA map
Change 356626 had a related patch set uploaded (by Fdans; owner: Fdans):
[operations/puppet@production] Add exception for events tagged as coming from MW
Change 356624 merged by Nuria:
[eventlogging@master] Add is_mediawiki property to UA map
Change 357243 had a related patch set uploaded (by Ottomata; owner: Nuria):
[eventlogging@master] Simpler parsing of user_agent to asses whether 'mediawiki' is present
Change 357243 merged by Ottomata:
[eventlogging@master] Simpler parsing of user_agent to asses whether 'mediawiki' is present
Change 356626 merged by Ottomata:
[operations/puppet@production] Add exception for events tagged as coming from MW