Page MenuHomePhabricator
Create Task
Maniphest T242766

upgrade cloud-vps openstack to Openstack version 'Queens'
Closed, ResolvedPublic
Actions

Description

As far as I know there's nothing preventing us from speeding ahead to Queens.

Horizon is already running version 'train' so this task does not apply to Horizon.

Details

SubjectRepoBranchLines +/-
wmf_sink: update conf settingsoperations/puppetproduction+14 -6
openstack: queens: drop python2 packagesoperations/puppetproduction+1 -44
Openstack: move eqiad1 to version 'queens'operations/puppetproduction+2 -2
Horizon: put in maintenance mode for the pike=>queens upgradeoperations/puppetproduction+1 -1
openstack: queens: don't install libpam-systemd from bpooperations/puppetproduction+1 -1
nova-compute: change virt_type to qemuoperations/puppetproduction+1 -1
nova-compute: install version-specific configoperations/puppetproduction+3 -3
wmfnovamiddleware: adjust encoding, againoperations/puppetproduction+3 -3
Neutron metadata_agent.ini: use nova_metadata_host instead of nova_metadata_ipoperations/puppetproduction+1 -1
Openstack queens packages: absent some python2 packagesoperations/puppetproduction+16 -0
wmfnovamiddleware: adjust string encoding for python3operations/puppetproduction+2 -3
nova-placement haproxy: update the health check routeoperations/puppetproduction+3 -3
openstack haproxy: change glance-api health checkoperations/puppetproduction+10 -8
nova-placement: update the uwsgi init script for Queensoperations/puppetproduction+112 -42
Openstack glance queens: override package-installed init scriptoperations/puppetproduction+235 -0
keystone: use python2 mod_wsgi for pike but python3 for queensoperations/puppetproduction+2 -1
cloudservices: update designate to openstack Queensoperations/puppetproduction+4 -0
wmf_sink: remove utf8 encodingoperations/puppetproduction+2 -2
designate: include python3-git -- we need this for wmfsinkoperations/puppetproduction+1 -0
designate: install python3 versions of sink handlersoperations/puppetproduction+32 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Andrew created this task.Jan 14 2020, 4:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 14 2020, 4:43 PM
Andrew added a comment.Jan 14 2020, 4:47 PM

Here are the steps from the O -> P upgrade, rearranged based on what we've learned.

  • downtime everything
  • on cloudcontrol1003 and cloudcontrol1004:
    • apply puppet
    • DEBIAN_FRONTEND=noninteractive apt-get install nova-common glance keystone nova-placement-api nova-api nova-conductor neutron-server -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • apt-get upgrade
    • apply puppet again
    • disable puppet
    • stop nova-conductor, nova-api, neutron-server, nova-scheduler sudo systemctl stop nova-conductor nova-api neutron-server nova-scheduler apache2
  • on cloudcontrol1003:
    • keystone-manage db_sync
    • nova-manage db sync

      root@cloudcontrol1003:~# nova-manage db sync

Option "notification_driver" from group "DEFAULT" is deprecated. Use option "driver" from group "oslo_messaging_notifications".
Option "notification_topics" from group "DEFAULT" is deprecated. Use option "topics" from group "oslo_messaging_notifications".

  • glance-manage db sync
  • nova-manage api_db sync

Neutron rolling upgrade order:

  1. stop and disable neutron-server
  2. upgrade neutron packages on cloudcontrol100[34]
  3. ensure neutron-server is stopped and run neutron-db-manage upgrade head
  4. restart neutron-server
  5. upgrade and restart agents on cloudnet and cloudvirt
  • on cloudcontrol1003 and cloudcontrol1004:
    • re-enable puppet
    • re-run puppet agent
  • on cloudnet1003 and cloudnet1004: (Move to after neutron-db-manage)
    • DEBIAN_FRONTEND=noninteractive apt-get install python-openstackclient python-openstacksdk neutron-common neutron-dhcp-agent neutron-l3-agent neutron-linuxbridge-agent neutron-metadata-agent python-neutron python-neutron-lib python-neutronclient -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • DEBIAN_FRONTEND=noninteractive apt-get upgrade (needed to resolve neutron linuxbridge dependencies)
    • puppet agent -tv
  • on cloudcontrol1003:
    • neutron-db-manage upgrade head (ensure neutron-server is stopped before running)
  • on cloudnet1003 and cloudnet1004: (only after neutron-db-manage)
    • puppet agent -tv
    • restart all neutron services sudo systemctl restart neutron*
  • on cloudvirts (via cumin): (only after neutron-db-manage)
    • puppet agent -tv
    • DEBIAN_FRONTEND=noninteractive apt-get install nova-compute nova-common nova-compute-kvm python-nova python-novaclient neutron-common neutron-linuxbridge-agent python-neutron python-neutron-lib python-neutronclient -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
    • DEBIAN_FRONTEND=noninteractive apt-get upgrade (needed to resolve neutron linuxbridge dependencies)
    • puppet agent -tv
    • service neutron-linuxbridge-agent restart
    • service nova-compute restart
  • merge https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/564658/, apply on labweb hosts
Andrew added a subtask: T234830: CloudVPS: m5-master databases for openstack may require re-encoding.Jan 14 2020, 4:54 PM
Paladox subscribed.Jan 14 2020, 4:56 PM
Krenair subscribed.Jan 24 2020, 10:18 PM
• Bstorm triaged this task as Medium priority.Feb 11 2020, 4:13 PM
aborrero closed subtask T246287: Import packages for Openstack Queens and Rocky as Resolved.Mar 3 2020, 10:34 AM
Andrew claimed this task.Mar 3 2020, 5:38 PM
gerritbot added a comment.Mar 3 2020, 5:39 PM

Change 576399 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: install python3 versions of sink handlers

https://gerrit.wikimedia.org/r/576399

gerritbot added a project: Patch-For-Review.Mar 3 2020, 5:39 PM

Change 576399 merged by Andrew Bogott:
[operations/puppet@production] designate: install python3 versions of sink handlers

https://gerrit.wikimedia.org/r/576399

gerritbot added a comment.Mar 3 2020, 5:55 PM

Change 576403 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] designate: include python3-git -- we need this for wmfsink

https://gerrit.wikimedia.org/r/576403

gerritbot added a comment.Mar 3 2020, 5:58 PM

Change 576403 merged by Andrew Bogott:
[operations/puppet@production] designate: include python3-git -- we need this for wmfsink

https://gerrit.wikimedia.org/r/576403

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2020, 6:11 PM
gerritbot added a comment.Mar 3 2020, 6:37 PM

Change 576410 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmf_sink: remove utf8 encoding

https://gerrit.wikimedia.org/r/576410

gerritbot added a project: Patch-For-Review.Mar 3 2020, 6:37 PM

Change 576410 merged by Andrew Bogott:
[operations/puppet@production] wmf_sink: remove utf8 encoding

https://gerrit.wikimedia.org/r/576410

Maintenance_bot removed a project: Patch-For-Review.Mar 3 2020, 7:11 PM
gerritbot added a comment.Mar 4 2020, 9:57 PM

Change 576972 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] cloudservices: update designate to openstack Queens

https://gerrit.wikimedia.org/r/576972

gerritbot added a project: Patch-For-Review.Mar 4 2020, 9:57 PM
gerritbot added a comment.Mar 4 2020, 10:19 PM

Change 576972 merged by Andrew Bogott:
[operations/puppet@production] cloudservices: update designate to openstack Queens

https://gerrit.wikimedia.org/r/576972

Maintenance_bot removed a project: Patch-For-Review.Mar 4 2020, 11:10 PM
gerritbot added a comment.Mar 5 2020, 9:28 PM

Change 577347 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] keystone: use python2 mod_wsgi for pike but python3 for queens

https://gerrit.wikimedia.org/r/577347

gerritbot added a project: Patch-For-Review.Mar 5 2020, 9:28 PM
gerritbot added a comment.Mar 5 2020, 9:31 PM

Change 577347 merged by Andrew Bogott:
[operations/puppet@production] keystone: use python2 mod_wsgi for pike but python3 for queens

https://gerrit.wikimedia.org/r/577347

Maintenance_bot removed a project: Patch-For-Review.Mar 5 2020, 10:11 PM
gerritbot added a comment.Mar 5 2020, 10:14 PM

Change 577358 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack glance queens: override package-installed init script

https://gerrit.wikimedia.org/r/577358

gerritbot added a project: Patch-For-Review.Mar 5 2020, 10:14 PM
gerritbot added a comment.Mar 5 2020, 10:20 PM

Change 577358 merged by Andrew Bogott:
[operations/puppet@production] Openstack glance queens: override package-installed init script

https://gerrit.wikimedia.org/r/577358

Maintenance_bot removed a project: Patch-For-Review.Mar 5 2020, 11:10 PM
Andrew added a subtask: T247050: ldap integration broken in python3-keystone 2:13.0.2-1~bpo10+1.Mar 6 2020, 5:15 AM
aborrero closed subtask T247050: ldap integration broken in python3-keystone 2:13.0.2-1~bpo10+1 as Resolved.Mar 6 2020, 10:43 AM
aborrero reopened subtask T247050: ldap integration broken in python3-keystone 2:13.0.2-1~bpo10+1 as Open.Mar 6 2020, 10:52 AM
gerritbot added a comment.Mar 6 2020, 5:04 PM

Change 577620 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] openstack haproxy: change glance-api health check

https://gerrit.wikimedia.org/r/577620

gerritbot added a project: Patch-For-Review.Mar 6 2020, 5:04 PM
gerritbot added a comment.Mar 6 2020, 5:16 PM

Change 577625 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-placement: update the usgi init script for Queens

https://gerrit.wikimedia.org/r/577625

gerritbot added a comment.Mar 6 2020, 5:19 PM

Change 577625 merged by Andrew Bogott:
[operations/puppet@production] nova-placement: update the uwsgi init script for Queens

https://gerrit.wikimedia.org/r/577625

gerritbot added a comment.Mar 6 2020, 5:31 PM

Change 577630 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-placement haproxy: update the health check route

https://gerrit.wikimedia.org/r/577630

gerritbot added a comment.Mar 6 2020, 5:51 PM

Change 577620 merged by Andrew Bogott:
[operations/puppet@production] openstack haproxy: change glance-api health check

https://gerrit.wikimedia.org/r/577620

gerritbot added a comment.Mar 6 2020, 5:51 PM

Change 577630 merged by Andrew Bogott:
[operations/puppet@production] nova-placement haproxy: update the health check route

https://gerrit.wikimedia.org/r/577630

Maintenance_bot removed a project: Patch-For-Review.Mar 6 2020, 6:11 PM
gerritbot added a comment.Mar 9 2020, 4:50 PM

Change 578359 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfnovamiddleware: adjust string encoding for python3

https://gerrit.wikimedia.org/r/578359

gerritbot added a project: Patch-For-Review.Mar 9 2020, 4:50 PM
gerritbot added a comment.Mar 9 2020, 4:55 PM

Change 578359 merged by Andrew Bogott:
[operations/puppet@production] wmfnovamiddleware: adjust string encoding for python3

https://gerrit.wikimedia.org/r/578359

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2020, 5:10 PM
gerritbot added a comment.Mar 9 2020, 7:41 PM

Change 578378 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack queens packages: absent some python2 packages

https://gerrit.wikimedia.org/r/578378

gerritbot added a project: Patch-For-Review.Mar 9 2020, 7:41 PM
gerritbot added a comment.Mar 9 2020, 8:09 PM

Change 578378 merged by Andrew Bogott:
[operations/puppet@production] Openstack queens packages: absent some python2 packages

https://gerrit.wikimedia.org/r/578378

Maintenance_bot removed a project: Patch-For-Review.Mar 9 2020, 8:10 PM
Andrew closed subtask T247050: ldap integration broken in python3-keystone 2:13.0.2-1~bpo10+1 as Resolved.Mar 9 2020, 9:29 PM
Andrew added a subtask: T247135: codfw1dev unavailable?.
aborrero closed subtask T247135: codfw1dev unavailable? as Resolved.Mar 18 2020, 11:29 AM
gerritbot added a comment.Mar 18 2020, 9:40 PM

Change 581104 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Neutron metadata_agent.ini: use nova_metadata_host instead of nova_metadata_ip

https://gerrit.wikimedia.org/r/581104

gerritbot added a project: Patch-For-Review.Mar 18 2020, 9:40 PM
gerritbot added a comment.Mar 18 2020, 9:43 PM

Change 581104 merged by Andrew Bogott:
[operations/puppet@production] Neutron metadata_agent.ini: use nova_metadata_host instead of nova_metadata_ip

https://gerrit.wikimedia.org/r/581104

Maintenance_bot removed a project: Patch-For-Review.Mar 18 2020, 10:11 PM
gerritbot added a comment.Mar 19 2020, 2:16 AM

Change 581210 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmfnovamiddleware: adjust encoding, again

https://gerrit.wikimedia.org/r/581210

gerritbot added a project: Patch-For-Review.Mar 19 2020, 2:16 AM
gerritbot added a comment.Mar 19 2020, 2:20 AM

Change 581210 merged by Andrew Bogott:
[operations/puppet@production] wmfnovamiddleware: adjust encoding, again

https://gerrit.wikimedia.org/r/581210

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2020, 3:10 AM
gerritbot added a comment.Mar 19 2020, 5:34 PM

Change 581666 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-compute: change virt_type to qemu

https://gerrit.wikimedia.org/r/581666

gerritbot added a project: Patch-For-Review.Mar 19 2020, 5:34 PM
gerritbot added a comment.Mar 19 2020, 5:36 PM

Change 581666 merged by Andrew Bogott:
[operations/puppet@production] nova-compute: change virt_type to qemu

https://gerrit.wikimedia.org/r/581666

gerritbot added a comment.Mar 19 2020, 6:06 PM

Change 581682 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] nova-compute: install version-specific config

https://gerrit.wikimedia.org/r/581682

gerritbot added a comment.Mar 19 2020, 6:12 PM

Change 581682 merged by Andrew Bogott:
[operations/puppet@production] nova-compute: install version-specific config

https://gerrit.wikimedia.org/r/581682

Maintenance_bot removed a project: Patch-For-Review.Mar 19 2020, 7:12 PM
Andrew added a comment.Mar 26 2020, 4:23 AM

https://etherpad.wikimedia.org/p/queensupgrade

gerritbot added a comment.Mar 26 2020, 2:11 PM

Change 583599 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: don't install libpam-systemd from bpo

https://gerrit.wikimedia.org/r/583599

gerritbot added a project: Patch-For-Review.Mar 26 2020, 2:11 PM

Change 583599 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: queens: don't install libpam-systemd from bpo

https://gerrit.wikimedia.org/r/583599

gerritbot added a comment.Mar 26 2020, 2:15 PM

Change 583600 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Horizon: put in maintenance mode for the pike=>queens upgrade

https://gerrit.wikimedia.org/r/583600

gerritbot added a comment.Mar 26 2020, 2:15 PM

Change 583601 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] Openstack: move eqiad1 to version 'queens'

https://gerrit.wikimedia.org/r/583601

gerritbot added a comment.Mar 26 2020, 2:54 PM

Change 583600 merged by Andrew Bogott:
[operations/puppet@production] Horizon: put in maintenance mode for the pike=>queens upgrade

https://gerrit.wikimedia.org/r/583600

Stashbot added a comment.Mar 26 2020, 3:01 PM

Mentioned in SAL (#wikimedia-cloud) [2020-03-26T15:01:09Z] <andrewbogott> beginning openstack upgrade window for T242766

gerritbot added a comment.Mar 26 2020, 3:02 PM

Change 583601 merged by Andrew Bogott:
[operations/puppet@production] Openstack: move eqiad1 to version 'queens'

https://gerrit.wikimedia.org/r/583601

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2020, 3:10 PM
gerritbot added a comment.Mar 26 2020, 4:42 PM

Change 583680 had a related patch set uploaded (by Arturo Borrero Gonzalez; owner: Arturo Borrero Gonzalez):
[operations/puppet@production] openstack: queens: drop python2 packages

https://gerrit.wikimedia.org/r/583680

gerritbot added a project: Patch-For-Review.Mar 26 2020, 4:42 PM
Andrew closed this task as Resolved.EditedMar 26 2020, 9:12 PM

The final checklist (with amendments) is:

Cloudcontrols:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install glance glance-api glance-registry glance-common keystone nova-api nova-conductor nova-placement-api nova-scheduler nova-common glance neutron-server python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • dump databases: nova, nova-api, neutron, glance, keystone
  • puppet agent -tv
  • nova-manage db sync
  • nova-manage api_db sync
  • glance-manage db_sync
  • keystone-manage db_sync
  • neutron-db-manage upgrade heads (( neutron-db-manage --database-connection 'mysql+pymysql://neutron:<redacted>@m5-master.eqiad.wmnet/neutron?charset=latin1' upgrade head `))
  • Check to see if ceph user exists ('groups ceph'). If not, 'dpkg-reconfigure ceph-common'

Cloudvirts:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • touch /usr/share/nova-common/policy.json
  • DEBIAN_FRONTEND=noninteractive apt-get install -y python3-libvirt python3-os-vif nova-compute neutron-common neutron-linuxbridge-agent python3-neutron python3-keystoneauth1 python3-positional python3-requests python3-urllib3 -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • service virtlogd restart
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --allow-downgrades -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • service virtlogd restart
  • puppet agent -tv
  • service neutron-linuxbridge-agent restart
  • service nova-compute restart
  • service libvirtd restart
  • dpkg-reconfigure ceph-common

cloudnets:

  • merge version patch https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/583601/
  • puppet agent -tv
  • apt-get update
  • DEBIAN_FRONTEND=noninteractive apt-get install -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold" neutron-l3-agent python3-neutronclient python3-glanceclient
  • DEBIAN_FRONTEND=noninteractive apt-get upgrade -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"
  • puppet agent -tv
  • service neutron-l3-agent restart

There was also a lot of encoding nonsense -- I changed all of the glance tables to have a default encoding of utf8, and we also had to change the specific alembic version column to utf8. For Neutron we didn't re-encode anything but I did hack the upgrade code to skip table creation and then created a table by hand.

Soon we'll rebuild and re-import these databases and then hopefully they'll have more modern encodings.

Andrew added a comment.Mar 27 2020, 4:57 PM

For cumin runs on cloudvirts:

cumin 'P{R:Class = role::wmcs::openstack::eqiad1::virt or R:Class = role::wmcs::openstack::eqiad1::virt_ceph}'

gerritbot added a comment.Apr 7 2020, 11:48 AM

Change 583680 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] openstack: queens: drop python2 packages

https://gerrit.wikimedia.org/r/583680

Maintenance_bot removed a project: Patch-For-Review.Apr 7 2020, 12:10 PM
gerritbot added a comment.Apr 9 2020, 1:57 PM

Change 587741 had a related patch set uploaded (by Andrew Bogott; owner: Andrew Bogott):
[operations/puppet@production] wmf_sink: update conf settings

https://gerrit.wikimedia.org/r/587741

gerritbot added a project: Patch-For-Review.Apr 9 2020, 1:57 PM

Change 587741 merged by Andrew Bogott:
[operations/puppet@production] wmf_sink: update conf settings

https://gerrit.wikimedia.org/r/587741

Maintenance_bot removed a project: Patch-For-Review.Apr 9 2020, 2:10 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits