Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RhinosF1
	Oct 7 2019, 4:20 PM

Description

The cloud-services-team planned, announced, and executed an upgrade of OpenStack components from the "Mitaka" version to the "Newton" version on 2019-10-07. After upgrading the software and related database schemas, various issues were found with instances in Cloud VPS projects. This ticket is tracking some of those issues, but is not the primary location that the Cloud Services SRE team is using to coordinate investigation and correction of the issues.

Original report:

500 internal server error on tools.wmflabs.org and all CI is dead

All tools.wmflabs.org tools are currently down with web services returning HTTP error 500

Not seen any alerts on wikitech to say it's planned so filing

Related Objects
Search...

Status	Assigned	Task
Resolved	Andrew	T237749 Upgrade wmcs OpenStack version to Ocata
Resolved	Andrew	T210715 cloudvps: PDNS 3.x vs 4.x
Open	None	T132225 Add SSHFP dns records to bastions
Resolved	MoritzMuehlenhoff	T224549 Track remaining jessie systems in production
Open	None	T224708 Drop most of mwopenstackclients.DnsManager in favour of designateclient
Resolved	aborrero	T212302 CloudVPS: upgrade: jessie -> stretch & mitaka -> newton
Resolved	Andrew	T234834 Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07
Duplicate	None	T234830 CloudVPS: m5-master databases for openstack may require re-encoding
Resolved	Andrew	T234876 nova-conductor running out of mysql connections
Resolved	Andrew	T234836 CloudVPS: update DNS record for eqiad1 egress (routing_source_ip) & ingress
Invalid	• JHedden	T235070 Resolve keystone schema errors

Event Timeline

RhinosF1 created this task.Oct 7 2019, 4:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 7 2019, 4:20 PM

Boldly setting to UBN!

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptOct 7 2019, 4:21 PM

Fallout from T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton, right?

RhinosF1 added a subscriber: aborrero.Oct 7 2019, 4:29 PM

Marostegui subscribed.Oct 7 2019, 4:31 PM

We are actively working on this. The OpenStack upgrade we planned for this morning has resulted in some networking issues that we are still attempting to work through.

bd808 added a parent task: T212302: CloudVPS: upgrade: jessie -> stretch & mitaka -> newton.Oct 7 2019, 4:34 PM

Jdforrester-WMF renamed this task from 500 internal server error on tools.wmflabs.org to 500 internal server error on tools.wmflabs.org and all CI is dead.Oct 7 2019, 4:36 PM

Jdforrester-WMF added a project: Release-Engineering-Team (CI & Testing services).

RhinosF1 updated the task description. (Show Details)Oct 7 2019, 4:38 PM

It appears that some Tools (most?) tools were actually working, but the 'admin' tool that serves up both https://tools.wmflabs.org/ and https://tools.wmflabs.org/admin/ was not working.

aborrero lowered the priority of this task from Unbreak Now! to High.Oct 7 2019, 4:46 PM

Reedy merged a task: T234833: tools.wmflabs.org/pb does not work.Oct 7 2019, 4:52 PM

Reedy added a subscriber: Boshomi_Phabricator.

In T234834#5552684, @bd808 wrote:

It appears that some Tools (most?) tools were actually working, but the 'admin' tool that serves up both https://tools.wmflabs.org/ and https://tools.wmflabs.org/admin/ was not working.

https://tools.wmflabs.org/versions/ wm-bot and Zppix-Bot were also down when I filed this.

Only wm-bot is back as of now.

tools.wmflabs.org/ is up though for me so that doesn't make sense

Confirming that CI is now fixed. Thanks!

Various issues we have seen/are working on:

Neutron (software defined network for OpenStack instances) was routing traffic between Cloud VPS instances and bare meta hosts (internal Wikimedia Foundation network) using a different source ip than expected after software updates
- The different source IP stopped a lot of traffic from being accepted due to local firewall settings for various services not allowing the new IP
- Restarting the Neutron services seems to have corrected this and started routing traffic via the expected IP
Communications with DNS recursors, LDAP directory servers, and NFS servers were all affected by the Neutron IP issue
Rolling restarts of instances in Toolforge are happening to correct the NFS mounting issues.

Aklapper added a project: Wikimedia-Incident.Oct 7 2019, 5:14 PM

Zppix subscribed.Oct 7 2019, 5:27 PM

Noting that all tools that I've seen as down are now back (one needed it's processes killing so they can restart by a maintainer)

bd808 renamed this task from 500 internal server error on tools.wmflabs.org and all CI is dead to Various user visible errors in Cloud VPS projects following OpenStack upgrade on 2019-10-07.Oct 7 2019, 5:30 PM

bd808 updated the task description. (Show Details)

RhinosF1 updated the task description. (Show Details)Oct 7 2019, 5:32 PM

SQL subscribed.Oct 7 2019, 5:37 PM

Tgr mentioned this in T234694: Cloud-vagrant: shared directory (/vagrant/settings.d/) seen by CLI PHP only.Oct 7 2019, 6:30 PM

Krenair subscribed.Oct 7 2019, 6:49 PM

Core infrastructure appears to be working as intended. Known instances with NFS mount failures have been rebooted. Toolforge seems to be working normally. Some tools may still need individual restarts, but we tried to force a restart of Kubernetes pods and other long running jobs that should have fixed many of them.

This incident will need some documentation follow up and analysis for ways that we could try to avoid unintended breakage in the future. I am going to remove my self as assigned for now, but also place this ticket in a queue for discussion at the next WMCS team meeting.

RhinosF1 awarded a token.Oct 7 2019, 8:44 PM