Open Stack

On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec <openstack at nemebean.com> wrote:
> Tempest isn't really either of those things.  According to another message
> in this thread it takes around 15 minutes to run just the smoke tests.
> That's unacceptable for a lot of our CI jobs.

Ben, is the issue merely the time it takes? Is it the affect that time
taken has on hardware availability?

Should we focus on how much testing we can get into N time period?
Then how do we decide an optimal N
for our constraints?

I've been working on a full up functional test for OpenStack CI builds
for a long time now, it works but takes
more than 10 hours. IF you're interested in results kick through to
Kibana here [0]. Let me know off list if you
have any issues, the presentation of this data is all experimental still.

[0] http://elk.browbeatproject.org:8080/goto/f264dea75e91469940e5ceb46444f8a3

On Mon, Apr 17, 2017 at 12:28 PM, Ben Nemec <openstack at nemebean.com> wrote:
>
>
> On 04/17/2017 10:51 AM, Emilien Macchi wrote:
>>
>> We haven't got much feedback from TripleO core reviewers, who are
>> usually more involved on this topic. I'll give a chance to let them
>> talk because we take some actions based on the feedback brought in
>> this discussion.
>
>
> I started to write a response last week and realized I didn't have a
> coherent recommendation, but here are my semi-organized thoughts:
>
> The pingtest was created for two main reasons.  First, it's fast.  Less than
> three minutes in most CI jobs.  Second, it's simple.  We've added a bunch of
> stuff for resource cleanup and such, but in essence it's four commands:
> glance image-create, neutron net-create, neutron subnet-create, and heat
> stack-create.  It would be hard to come up with a useful test that is
> meaningfully simpler.
>
> Tempest isn't really either of those things.  According to another message
> in this thread it takes around 15 minutes to run just the smoke tests.
> That's unacceptable for a lot of our CI jobs.  It also tends to require a
> lot more configuration in my experience.
>
> All that said, the simplicity of the ping test means it isn't well suited to
> testing more complex things, and I don't think we should try to extend it to
> do so.  Tempest already provides an extensible, pluggable framework for
> complex testing.
>
> Personally I don't want to see the ping test go away completely.  I like
> having a stupid simple and fast way to sanity check a deployment.  If we
> deleted the ping test I would probably just maintain a script that does the
> same basic thing out of tree.
>
> However, if we're starting to duplicate Tempest functionality for scenario
> tests or advanced features then I do think we need to figure out how to
> integrate Tempest for that test coverage.  I will note that I don't think
> the requirement to have a Heat resource for things we test in TripleO is
> necessarily a bad thing, but I don't think that's the only reason this is
> being discussed so I'll leave it at that.
>
> There's also a separate issue of API coverage.  It's true that the ping test
> only provides a bare minimum of coverage for the services, but as I
> discussed above that's largely a function of job runtime.  We simply can't
> run full API tests in many of our CI jobs.  Maybe we could in the basic
> multinode jobs, but the updates, upgrades, and ovb jobs are all too long
> already.  In those the best we can hope to do is a minimal sanity check.
> I'm thinking one Tempest test per service, max.  Less than that if we can
> find tests that exercise multiple services in one (like the ping test does).
>
> That's my brain dump on this.  Do with it what you will. :-)
>
>
>>
>> Thanks,
>>
>> On Fri, Apr 7, 2017 at 4:26 AM, Ravi Sekhar Reddy Konda
>> <ravisekhar.konda at oracle.com> wrote:
>>>
>>> Hi Diarmuid,
>>>
>>> Even in our setup Sravanthi by mistake has deleted all the ports
>>> we are trying to bring up again, if it is done today wil ping you and
>>> schedule again
>>>
>>> Otherwise i will schedule for Monday
>>>
>>> Thanks,
>>> Ravi
>>> ----- Original Message -----
>>> From: jtaleric at redhat.com
>>> To: openstack-dev at lists.openstack.org, jkilpatr at redhat.com
>>> Sent: Thursday, April 6, 2017 4:15:58 PM GMT +05:30 Chennai, Kolkata,
>>> Mumbai, New Delhi
>>> Subject: Re: [openstack-dev] [tripleo] pingtest vs tempest
>>>
>>> On Thu, Apr 6, 2017 at 5:32 AM, Sagi Shnaidman <sshnaidm at redhat.com>
>>> wrote:
>>>>
>>>> HI,
>>>>
>>>> I think Rally or Browbeat and other performance oriented solutions won't
>>>> serve our needs, because we run TripleO CI on virtualized environment
>>>> with
>>>> very limited resources. Actually we are pretty close to full utilizing
>>>> these
>>>> resources when deploying openstack, so very little is available for
>>>> test.
>>>> It's not a problem to run tempest API tests because they are cheap -
>>>> take
>>>> little time, little resources, but also gives little coverage. Scenario
>>>> test
>>>> are more interesting and gives us more coverage, but also takes a lot of
>>>> resources (which we don't have sometimes).
>>>
>>>
>>> Sagi,
>>> In my original message I mentioned a "targeted" test, I should
>>> explained that more. We could configure the specific scenario so that
>>> the load on the virt overcloud would be minimal. Justin Kilpatrick
>>> already have Browbeat integrated with TripleO Quickstart[1], so there
>>> shouldn't be much work to try this proposed solution.
>>>
>>>>
>>>> It may be useful to run a "limited edition" of API tests that maximize
>>>> coverage and don't duplicate, for example just to check service working
>>>> basically, without covering all its functionality. It will take very
>>>> little
>>>> time (i.e. 5 tests for each service) and will give a general picture of
>>>> deployment success. It will cover fields that are not covered by
>>>> pingtest as
>>>> well.
>>>>
>>>> I think could be an option to develop a special scenario tempest tests
>>>> for
>>>> TripleO which would fit our needs.
>>>
>>>
>>> I haven't looked at Tempest in a long time, so maybe its functionality
>>> has improved. I just saw the opportunity to integrate Browbeat/Rally
>>> into CI to test the functionality of OpenStack services, while also
>>> capturing performance metrics.
>>>
>>> Joe
>>>
>>> [1]
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_browbeat_tree_master_ci-2Dscripts&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=rFCQ76TW5HZUgA7b20ApVcXgXru6mvz4fvCm1_H6w1k&m=c7EeLf1PQSsV2XbWBhv6CWOzUFDRnDiIheN4lDKjyq8&s=Z0jGFw40ezDmSb3F6ns5SXRvacH6AgU0TK5dKBSRgEs&e=
>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>> On Wed, Apr 5, 2017 at 11:49 PM, Emilien Macchi <emilien at redhat.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> Greetings dear owls,
>>>>>
>>>>> I would like to bring back an old topic: running tempest in the gate.
>>>>>
>>>>> == Context
>>>>>
>>>>> Right now, TripleO gate is running something called pingtest to
>>>>> validate that the OpenStack cloud is working. It's an Heat stack, that
>>>>> deploys a Nova server, some volumes, a glance image, a neutron network
>>>>> and sometimes a little bit more.
>>>>> To deploy the pingtest, you obviously need Heat deployed in your
>>>>> overcloud.
>>>>>
>>>>> == Problems:
>>>>>
>>>>> Although pingtest has been very helpful over the last years:
>>>>> - easy to understand, it's an Heat template, like an OpenStack user
>>>>> would do to deploy their apps.
>>>>> - fast: the stack takes a few minutes to be created and validated
>>>>>
>>>>> It has some limitations:
>>>>> - Limitation to what Heat resources support (example: some OpenStack
>>>>> resources can't be managed from Heat)
>>>>> - Impossible to run a dynamic workflow (test a live migration for
>>>>> example)
>>>>>
>>>>> == Solutions
>>>>>
>>>>> 1) Switch pingtest to Tempest run on some specific tests, with feature
>>>>> parity of what we had with pingtest.
>>>>> For example, we could imagine to run the scenarios that deploys VM and
>>>>> boot from volume. It would test the same thing as pingtest (details
>>>>> can be discussed here).
>>>>> Each scenario would run more tests depending on the service that they
>>>>> run (scenario001 is telemetry, so it would run some tempest tests for
>>>>> Ceilometer, Aodh, Gnocchi, etc).
>>>>> We should work at making the tempest run as short as possible, and the
>>>>> close as possible from what we have with a pingtest.
>>>>>
>>>>> 2) Run custom scripts in TripleO CI tooling, called from the pingtest
>>>>> (heat template), that would run some validations commands (API calls,
>>>>> etc).
>>>>> It has been investigated in the past but never implemented AFIK.
>>>>>
>>>>> 3) ?
>>>>>
>>>>> I tried to make this text short and go straight to the point, please
>>>>> bring feedback now. I hope we can make progress on $topic during Pike,
>>>>> so we can increase our testing coverage and detect deployment issues
>>>>> sooner.
>>>>>
>>>>> Thanks,
>>>>> --
>>>>> Emilien Macchi
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>
>>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=rFCQ76TW5HZUgA7b20ApVcXgXru6mvz4fvCm1_H6w1k&m=c7EeLf1PQSsV2XbWBhv6CWOzUFDRnDiIheN4lDKjyq8&s=W6U8XqdHMEh8UmMqQtZNoLw8eWnRvoFgYl-sKUgkiBY&e=
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Best regards
>>>> Sagi Shnaidman
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>
>>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=rFCQ76TW5HZUgA7b20ApVcXgXru6mvz4fvCm1_H6w1k&m=c7EeLf1PQSsV2XbWBhv6CWOzUFDRnDiIheN4lDKjyq8&s=W6U8XqdHMEh8UmMqQtZNoLw8eWnRvoFgYl-sKUgkiBY&e=
>>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.openstack.org_cgi-2Dbin_mailman_listinfo_openstack-2Ddev&d=DwIGaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=rFCQ76TW5HZUgA7b20ApVcXgXru6mvz4fvCm1_H6w1k&m=c7EeLf1PQSsV2XbWBhv6CWOzUFDRnDiIheN4lDKjyq8&s=W6U8XqdHMEh8UmMqQtZNoLw8eWnRvoFgYl-sKUgkiBY&e=
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev