Open Stack

On 02/06/2014 09:52 AM, Luke Gorrie wrote:
> Howdy!
> 
> My name is Luke and I'm helping my friends at Tail-f Systems to
> support Neutron with their NCS [1] product. This went really smoothly
> for us on the Havana cycle, but lately we're having a harder time with
> Icehouse. In particular, our attempt to fulfill the 3rd party testing
> requirements has caused a lot of frustration for the #openstack-infra
> team around New Year. So I'm writing now to look for a good solution.
> 
> Our goal for Icehouse is to keep our mechanism driver officially
> supported. The code already works, and has unit tests to keep it
> working. The risk we want to avoid is going on the dreaded
> "deprecated" list for some other reason, which would confuse our
> customers.
> 
> For background, our mechanism driver is about 150 lines of code. It
> translates each network/port/subnet API call into a REST/JSON
> notification to an external system. That external system returns HTTP
> "200 OK". That's about all. It's a pretty trivial program.
> 
> In December I sat down with Tobbe Tornqvist and we tried to setup
> Jenkins 3rd party testing. We created a Vagrantfile that spins up an
> Ubuntu VM, installs Neutron and NCS, and performs integration tests
> with them connected together. We hooked this into Jenkins with a
> service account.
> 
> This went fine to start with, but after Christmas our tests started
> failing due to random setup issues with 'tox', and the script started
> making -1 votes. Those -1's created major headaches for the
> infrastructure team and others upstream, I am sorry to say, and ended
> up requiring a messy process of manual cleanup, and a public flogging
> for us on IRC. Bad feeling all around ...

The part to keep in mind is that random issues crop up all the time. For
us in -infra they are a weekly event, and some weeks they are a daily
event. The part that was the most difficult was the lack of
responsiveness to our efforts to communicate, to get an acknowledgement
from the maintainers that this account was experiencing some issues.
Then to get the issues addressed. In the end we failed and had to resort
to revoking verify voting permissions for this account.

It has become evident through an IRC conversation with Tobbe Tornqvist
[0] that the human resources required to stay available to maintain this
testing system is an issue.

While understanding of the need of different sized companies to make
decisions to provide value for their customers, in -infra, we can not
lose sight of the fact that our customers are our developers and we need
to be responsive to their requirements.

We are moving to be able to merge 200 patches a day with our system, by
making a commitment to that rate of development as our target, we have
to ensure any dependencies for testing also are able to move at a
similar pace or at least be responsive to ours.

Third-party CI [1] service accounts can still place comments on open
reviews even if they lack the ability to set a verify vote (-1/+1), and
this can be used to show whether the backend is reliably representing
the changes it tests in an effort to build community trust in its
results. Your account, Tail-f, can vote on our sandbox repo [2] as a way
of testing voting for the system.

Thank you,
Anita.

[0]
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-01-10.log
timestamp ~ 2014-01-10T14:48:01
[1] https://review.openstack.org/#/admin/groups/270,members
[2] http://git.openstack.org/cgit/openstack-dev/sandbox/


> 
> And that's where we are today.
> 
> Now, reviewing the situation, I would love to find a solution that
> doesn't make us deprecated _and_ doesn't require us to be so deeply
> hooked into the OpenStack Jenkins infrastructure.
> 
> Could we, for example, write an accurate emulator for the external
> system so that the MD code can be tested on OpenStack's own servers?
> That would suit us very well. It seems like a reasonable request to
> make given the simplicity of our driver code.
> 
> Hoping for a simple solution...
> 
> Cheers,
> -Luke & friends at Tail-f
> 
> [1] http://blog.ipspace.net/2013/05/tail-f-network-control-system-first.html
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>