maintenance server

**maintenance server**
	maintenance server
Location:	eqiad
Cluster:	Miscellaneous eqiad
Node name:	mwmaint1002.eqiad.wmnet (fingerprint)
Usage:	Miscellaneous
Status
Overall:	Active
Icinga?:	host status services status
Hardware
Software

The Maintenance server is where we run scheduled and ad-hoc MediaWiki maintenance scripts from the command-line.

Server

As of June 2024, the servers are:

mwmaint1002, current active maintenance server in eqiad
mwmaint2002, this one is passive, to be used if we switch activity to codfw (switchover).

We manage the scheduling in Puppet (puppet: profile/mediawiki/maintenance.pp), where we declare which scripts to run, when and how often, and what wikis to run them on. As of 2020, the scripts are run via systemd timers. Previously, we used cron to run the maintenance jobs.

Connecting

Connect to maintenance server over ssh by proxying through a bastion (e.g. bast1003.wikimedia.org). As with other hosts, ensure you have ProxyJump configured in .ssh/config (see Production shell access). Then connect as follows:

user@laptop:~$ ssh mwmaint1002.eqiad.wmnet

To run maintenance scripts on the Beta Cluster you need to connect to the equivalent servers in the "deployment-prep" Cloud VPS project:

deployment-mwmaint02.deployment-prep.eqiad1.wikimedia.cloud

Runbook

Access recent runs

List scheduled scripts (what is running, which will soon run, when did they last run).

mwmaint$ systemctl list-timers 'mediawiki*'
NEXT                  LEFT        LAST                  UNIT
2024-07-09 20:47:00Z  5s left     2024-07-09 20:46:00Z  mediawiki_job_db_lag_stats_reporter.timer
2024-07-09 21:00:00Z  13min left  2024-07-08 21:00:00Z  mediawiki_job_initsitestats.timer
2024-07-09 21:15:00Z  28min left  2024-07-09 18:15:00Z  mediawiki_job_growthexperiments-updateMenteeData-s1.timer
…
2024-07-09 21:39:00Z  52min left  2024-07-09 20:39:00Z  mediawiki_job_wikidata_resubmit_changes_for_dispatch.timer
2024-07-10 01:00:00Z  4h left     2024-07-09 01:00:00Z  mediawiki_job_purge_parsercache_pc1.timer
…

Display recent script output. You can use -f to follow it in real-time. Omit ".timer" from the name here (based on the output above)

mwmaint$ sudo journalctl -u mediawiki_job_startupregistrystats -n100

Run a maintenance script on a wiki

As of September 2024, maintenance scripts should be run on Kubernetes. For now, mwscript on the maintenance server is available as a fallback, but it will be going away.

When running a long-running maintenance script, consider using screen.

To run a maintenance script, log into the active maintenance server.

Use the mwscript command-line utility:

mwscript scriptname.php --wiki dbname

The above command will run the script scriptname.php located in the /maintenance directory of the specified wiki's php directory.

You can also use a path relative to root of the MediaWiki installation, for example to run a script from an extension rather than core. It is recommended that in this case you first browser to one of the MediaWiki directories to allow for tab completion:

$ cd /srv/mediawiki/php-1.xx-wmf.xx/    # optional; enables tab completion
$ mwscript extensions/MyExtension/maintenance/scriptname.php --wiki dbname

If you need to pass parameters to your maintenance script, you can add them to the end. You may also want to explicitly declare the database parameter to avoid confusion:

mwscript scriptname.php --wiki=enwiki --days=1

In this case, the wiki parameter is handled by mwscript and the 'days' parameter is handled by scriptname.php. The "wiki" must be in wikiversions.json so the maintenance script machinery can figure out what MediaWiki version it is running. The wiki parameter must be exactly after the script name – otherwise, it is not recognised.

Many maintenance scripts are used to run heavy database queries; remember to check the replication lag and other possible byproducts.

Run a custom maintenance script

You can also run an arbitrary PHP file outside the MediaWiki directory as a maintenance script. Typically this is done to test changes to maintenance scripts that cannot be done locally, or to avoid having to wait for a train deployment / doing a backport when running the fixed version of a maintenance script.

mwscript /home/foo/scriptname.php --wiki=enwiki

will run /home/foo/scriptname.php with the MediaWiki root directory set to whichever directory enwiki is served from (so all other files and classes referenced in the script will be loaded from their standard location).

Use this responsibly. The privilege policy applies here as well: you should not run non-trivial code in production that wasn't reviewed by at least one other person. Make sure that, if something goes wrong, others can reconstruct exactly what code was executed (e.g. by saying !log T12345 running mwscript /home/foo/scriptname.php --wiki=enwiki on IRC).

History

Former services

noc.wikimedia.org: As of mid-2023, the mediawiki-noc web app is now served from the "mw-misc" cluster in MediaWiki On Kubernetes.

Hosts

Eqiad data center: terbium (2013), mwmaint1001 (2018), mwmaint1002 (2018).
Codfw data center: wasat (2017), mwmaint2001 (2018), mwmaint2002 (2021).