Hi Mohammed,

regarding CentOS CI slowness - you are not alone, kola is experiencing it too.

Some time ago I asked around and discovered that various projects experience noticeable CI slowness using CentOS compared to Ubuntu.

The hit is more noticeable on image building than deploying and I guess since OSA does "building/installing" while deploying, it experiences our worse hit.

We deploy in containers so I even checked the host:Ubuntu, container:CentOS scenario but it was noticeably slower anyway.

Hence it seems related to some CentOS libraries or common stuff (like Python maybe?), not kernel.

We even ran into broken CI situation by being hit by horizon changing translation-related internals and our once hacky way of compiling translations which resulted in n^2 compilations instead of n - Ubuntu was a bit slower than usual but CentOS timed out, possibly not even halfway through horizon image building process...

Kind regards,

Radek

pt., 26 lip 2019 o 15:45 Mohammed Naser <mnaser@vexxhost.com> napisał(a):

Hi everyone,

I'm going to try doing these more often to keep everyone updated on
the current state of openstack-ansible (mostly around what is
discussed during our office hours / meetings).

- Dmitriy has been working on removing unused code in playbooks,
mostly around logs as we move to 100% systemd-journald.
- We're discussing the proposal to remove os-log-dir-setup.yml, as
well as rsyslog_client and rsyslog_server roles as they aren’t used
anymore with the move to systemd-journald
- We're also discussing removing all the complex bind mounts, which
would clean it and simplify the code base.
- Our continuous integration has been stable but needs upgrade jobs
and increased coverage. Our coverage would go up significantly by
running aio_lxc for all supported systems.
- Jonathan's work on binding to management IPs is making progress (and
seems to be at the Galera stage).
- CentOS CI has been very slow and nobody knows why. It seems to be
something we've regressed at some point.
- A unicast flood issue affecting deployments using linuxbridge that
happened inside os-vif needs to be patched. It's still pending os-vif
to patch.
- Namrata mentioned an issue during the upgrade from Rocky to Stein
(when upgrading WSREP SST method from xtrabackup-V2 to mariabackup)
that will have to be fixed is Stein. That's something that's in
progress now.

Thanks everyone for reading!

Regards,
Mohammed

--
Mohammed Naser — vexxhost
-----------------------------------------------------
D. 514-316-8872
D. 800-910-1726 ext. 200
E. mnaser@vexxhost.com
W. http://vexxhost.com