Open Stack

On Thu, Mar 02, 2017 at 10:13:07AM +0100, Marcin Juszkiewicz wrote:
> I am working on some improvements for Kolla. Part of that work is
> sending patches for review.
> 
> Once patch is set for review.openstack.org there is a set of Jenkins
> jobs started to make sure that patch does not break already working
> code. And this is good thing.
> 
> How it is done is not good ;(
> 
> 1. Kolla output is nightmare to debug.
> 
> There is --logs-dir option to provide separate logs for each image build
> but it is not used. IMHO it should be as digging through such logs is
> easier.
> 
> Several images feels like they try to do install again and again to pass
> through what distribution consider a bug - like "user XY already exists"
> bugs while Debian/Ubuntu are used as base distro. Which adds several
> error messages to check/ignore.
> 
> 
> 2. Some jobs fail because "sometimes (not always) the gate can't access
> some source on the internet".
> 
> I spent most of my career on building software. Having builds fail for
> such reasons was hardly acceptable when building could take hours. So we
> mirrored all sources and used own mirror(s) as fallback. On other
> systems I used 10-20GB w3cache to handle that for me (because there was
> no way to provide local mirror of used sources).
> 
> OpenStack infrastructure lacks any of it. Using "recheck" comment in
> review to restart Jenkins jobs is not a solution - how many times it has
> to fail to make sure that it is patch's fault not infrastructure one?
> 
Lets not use absolutes please. In fact, the openstack-infra team does mirror a
lot of things today[1] but it sounds like not everything you need for jobs.

For the most part, the openstack-infra team is pretty accommodation to mirroring
requests. I admit, sometimes we are not the fastest to implement them, but we
try.  At the PTG, the openstack-ansible team reminded me about setting up a
mirror for mariada, which I plan on working on today / tomorrow.
> 

> As a contributor I started to ignore Jenkins tests. Instead I do builds
> on several machines to check does everything works with my patches. If
> something does not then I update my patchset.
> 
Now, this is my personal opinion not openstack-infra, if you have to rely on
local testing over testing in the gate, I think there is some serious issues
with your project.

Testing is hard, I get that. Nobody wants to recheck their patch multiple times
to get a green check mark.  At the same time, there are very few people in
general that gravitate towards testing infrastructure (this is a general
comment, not specific to kolla).

As a suggestion, I would have a serious talk about stopping the addition of
additional features to your code base and maybe work towards improving your
testing coverage. Having a strong test base to me makes a world of difference
for your developers and moral.

I can tell you from personal experience, if a project (inside openstack or
outside) doesn't have good test coverage, and I am always fighting with flapping
tests, I find it difficult to keep contributing like you mentioned above.

[1] http://mirror.dfw.rax.openstack.org/