Open Stack

I think testing with sandbox repo is really useful during setting
up third party CI. Not so many people know sandbox repo.
When neutron folks were setting up third party CI, we created test
reviews in openstack/neutron, but if we knew it it would be great.

It is worth documented in "Third Party Testing" web page [1].
I will add it if nobody adds it.

[1] http://ci.openstack.org/third_party.html

Akihiro

(2014/02/10 12:13), Anita Kuno wrote:
> On 02/06/2014 09:52 AM, Luke Gorrie wrote:
>> Howdy!
>>
>> My name is Luke and I'm helping my friends at Tail-f Systems to
>> support Neutron with their NCS [1] product. This went really smoothly
>> for us on the Havana cycle, but lately we're having a harder time with
>> Icehouse. In particular, our attempt to fulfill the 3rd party testing
>> requirements has caused a lot of frustration for the #openstack-infra
>> team around New Year. So I'm writing now to look for a good solution.
>>
>> Our goal for Icehouse is to keep our mechanism driver officially
>> supported. The code already works, and has unit tests to keep it
>> working. The risk we want to avoid is going on the dreaded
>> "deprecated" list for some other reason, which would confuse our
>> customers.
>>
>> For background, our mechanism driver is about 150 lines of code. It
>> translates each network/port/subnet API call into a REST/JSON
>> notification to an external system. That external system returns HTTP
>> "200 OK". That's about all. It's a pretty trivial program.
>>
>> In December I sat down with Tobbe Tornqvist and we tried to setup
>> Jenkins 3rd party testing. We created a Vagrantfile that spins up an
>> Ubuntu VM, installs Neutron and NCS, and performs integration tests
>> with them connected together. We hooked this into Jenkins with a
>> service account.
>>
>> This went fine to start with, but after Christmas our tests started
>> failing due to random setup issues with 'tox', and the script started
>> making -1 votes. Those -1's created major headaches for the
>> infrastructure team and others upstream, I am sorry to say, and ended
>> up requiring a messy process of manual cleanup, and a public flogging
>> for us on IRC. Bad feeling all around ...
>
> The part to keep in mind is that random issues crop up all the time. For
> us in -infra they are a weekly event, and some weeks they are a daily
> event. The part that was the most difficult was the lack of
> responsiveness to our efforts to communicate, to get an acknowledgement
> from the maintainers that this account was experiencing some issues.
> Then to get the issues addressed. In the end we failed and had to resort
> to revoking verify voting permissions for this account.
>
> It has become evident through an IRC conversation with Tobbe Tornqvist
> [0] that the human resources required to stay available to maintain this
> testing system is an issue.
>
> While understanding of the need of different sized companies to make
> decisions to provide value for their customers, in -infra, we can not
> lose sight of the fact that our customers are our developers and we need
> to be responsive to their requirements.
>
> We are moving to be able to merge 200 patches a day with our system, by
> making a commitment to that rate of development as our target, we have
> to ensure any dependencies for testing also are able to move at a
> similar pace or at least be responsive to ours.
>
> Third-party CI [1] service accounts can still place comments on open
> reviews even if they lack the ability to set a verify vote (-1/+1), and
> this can be used to show whether the backend is reliably representing
> the changes it tests in an effort to build community trust in its
> results. Your account, Tail-f, can vote on our sandbox repo [2] as a way
> of testing voting for the system.
>
> Thank you,
> Anita.
>
> [0]
> http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2014-01-10.log
> timestamp ~ 2014-01-10T14:48:01
> [1] https://review.openstack.org/#/admin/groups/270,members
> [2] http://git.openstack.org/cgit/openstack-dev/sandbox/
>
>
>>
>> And that's where we are today.
>>
>> Now, reviewing the situation, I would love to find a solution that
>> doesn't make us deprecated _and_ doesn't require us to be so deeply
>> hooked into the OpenStack Jenkins infrastructure.
>>
>> Could we, for example, write an accurate emulator for the external
>> system so that the MD code can be tested on OpenStack's own servers?
>> That would suit us very well. It seems like a reasonable request to
>> make given the simplicity of our driver code.
>>
>> Hoping for a simple solution...
>>
>> Cheers,
>> -Luke & friends at Tail-f
>>
>> [1] http://blog.ipspace.net/2013/05/tail-f-network-control-system-first.html
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>