Open Stack

> Von: LELOUP Julien [mailto:Julien.LELOUP at 3ds.com]
> Gesendet: Montag, 3. März 2014 11:21
> An: OpenStack Development Mailing List (not for usage questions); Koderer,
> Marc
> Betreff: [Tempest - Stress Test][qa] : implement a full SSH connection on
> "ssh_floating.py" and improve it
>
> Hello,
>
> It tourns out that this stress test maybe not implemented in the right
> place.
>
> At ther moment I have put the SSH stress test in large_ops but Joe Gordon
> pointed (in https://review.openstack.org/#/c/74067/) that the large_ops
> gates are not meant to launch real servers.

Yep, this is a discussion that came up in the last summit and I might
undervalue the consequences. IMHO stress test's should both work with
fake drivers and with real environments. If they don't and we really
want to support such cases we need to flag such tests or make these
checks optional. If we do so these check possibly never run in the
gate. Especially for stress tests it's might the case that there
are designed for in-house usage and not for the gate.

>
> So where should I put this test ? I'm thinking of creating a new scenario
> file inheriting of the class specified in the large_ops file
> (TestLargeOpsScenario) in order to avoid duplicating the "nova_boot"
> method.
> Is it OK for you ?
>
> Is there a better place where I should put my test ?

I mean we still have the tempest/stress/actions/ and we could put something
like that in there. In general I would like to discuss this topic in the next
QA meeting..
@Julien: are you able to join the next meeting? It would be 22:00 UTC.


>
> Best Regards,
>
> Julien LELOUP
> julien.leloup at 3ds.com
>

Regards,
Marc


>
> -----Original Message-----
> From: LELOUP Julien [mailto:Julien.LELOUP at 3ds.com]
> Sent: Monday, February 17, 2014 5:02 PM
> To: OpenStack Development Mailing List (not for usage questions); Koderer,
> Marc
> Subject: Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a
> full SSH connection on "ssh_floating.py" and improve it
>
> Hello Marc,
>
> I'm raising this subject again since I have pushed a patch set
> implementing this SSH stress test in Tempest.
> As you suggested, I wrote it as a scenario test : I'm currently using it
> to check the ability of my stack to launch x servers at the same time and
> having them fully available to use.
>
> As a reminder, here is the Blueprint :
> https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip
>
> Plese feel free to check it and make feedback :)
>
> Best Regards,
>
> Julien LELOUP
> julien.leloup at 3ds.com
>
> -----Original Message-----
> From: LELOUP Julien [mailto:Julien.LELOUP at 3ds.com]
> Sent: Monday, January 20, 2014 11:55 AM
> To: Koderer, Marc
> Cc: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [qa] [Tempest - Stress Test] : implement a
> full SSH connection on "ssh_floating.py" and improve it
>
> Hello Marc,
>
> Thanks again for your help, your blog post is helpful.
>
> So I will start writing a new scenario test to get this full SSH stress
> test on newly created VM.
> I will put more details about it in the blueprint I created for this :
> https://blueprints.launchpad.net/tempest/+spec/stress-test-ssh-floating-ip
>
> Best Regards,
>
> Julien LELOUP
> julien.leloup at 3ds.com
>
>
> -----Original Message-----
> From: Koderer, Marc [mailto:m.koderer at telekom.de]
> Sent: Saturday, January 18, 2014 10:11 AM
> To: LELOUP Julien
> Cc: openstack-dev at lists.openstack.org
> Subject: RE: [qa] [Tempest - Stress Test] : implement a full SSH
> connection on "ssh_floating.py" and improve it
>
> Hello Julien,
>
> maybe my blog post helps you with some more details:
>
> http://telekomcloud.github.io/2013/09/11/new-ways-of-tempest-stress-
> testing.html
>
> You can run single test if you add a new json file with the test function
> you want to test. Like:
> https://github.com/openstack/tempest/blob/master/tempest/stress/etc/sample
> -unit-test.json
>
> With that you can launch them with the parameters you already described.
>
> Regards,
> Marc
>
> ________________________________________
> From: LELOUP Julien [Julien.LELOUP at 3ds.com]
> Sent: Friday, January 17, 2014 3:49 PM
> To: Koderer, Marc
> Cc: openstack-dev at lists.openstack.org
> Subject: RE: [Tempest - Stress Test] : implement a full SSH connection on
> "ssh_floating.py" and improve it
>
> Hi Marc,
>
> The Etherpad you provided was helpful to know the current state of the
> stress tests.
>
> I admit that I have some difficulties to understand how I can run a single
> test built with the @stresstest decorator (even not a beginner in Python,
> I still have things to learn on this technology and a lot more on
> OpenStack/Tempest :) ).
> I used to run my test using "./run_stress.py -t <JSON configuration
> pointing at my action/.py script> -d <duration>", which allowed me to run
> only one test with a dedicated configuration (number of threads, ...)
>
> For what I understand now in Tempest, I only managed to run all tests,
> using "./run_tests.sh" and the only configuration I found related to
> stress tests was the [stress] section in tempest.conf.
>
> For example : let say I ported my SSH stress test as a scenario test with
> the @stresstest decorator.
> How can I launch this test (and only this one) and use a dedicated
> configuration file like ones we can found in "tempest/stress/etc" ?
>
> Another question I have now : in the case that I have to use "run_test.sh"
> and not "run_stress.py" anymore, how do I get the test runs statistics I
> used to have, and where should I put some code to improve them ?
>
> When I will have cleared my mind with all these kinds of practical
> details, maybe I should add some content on the Wiki about stress tests in
> Tempest.
>
> Best Regards,
>
> Julien LELOUP
> julien.leloup at 3ds.com
>
> -----Original Message-----
> From: Koderer, Marc [mailto:m.koderer at telekom.de]
> Sent: Friday, January 17, 2014 3:07 PM
> To: LELOUP Julien
> Cc: openstack-dev at lists.openstack.org
> Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH
> connection on "ssh_floating.py" and improve it
>
> Hi Julien,
>
> most of the cases in tempest/stress are already covered by exiting tests
> in /api or /scenario. The only thing that is missing is the decorator on
> them.
>
> BTW here is the Etherpad from the summit talk that we had:
> https://etherpad.openstack.org/p/icehouse-summit-qa-stress-tests
>
> It possible help to understand the state. I didn't managed to work on the
> action items that are left.
>
> Your suggestions sound good. So I'd happy so see some patches :)
>
> Regards
> Marc
> ________________________________________
> From: LELOUP Julien [Julien.LELOUP at 3ds.com]
> Sent: Friday, January 17, 2014 11:52 AM
> To: Koderer, Marc
> Cc: openstack-dev at lists.openstack.org
> Subject: RE: [qa] RE: [Tempest - Stress Test] : implement a full SSH
> connection on "ssh_floating.py" and improve it
>
> Hello Marc,
>
> Thanks for your answer.
>
> At the moment I'm willing to spend some time on this kind of scenario so I
> will see how to use the stress decorator inside a scenario test.
> Does this means that all stress tests available in "tempest/stress" should
> be ported as scenario tests with this decorator ?
>
> I do have some ideas about features on stress test that I find useful for
> my own use case : like adding more statistics on stress test runs in order
> to use them as benchmarks.
> I don't know if this kind of feature was already discussed in the
> OpenStack community but since stress tests are a bit deprecated now, maybe
> there is some room for this kind of improvement on "fresh" stress tests.
>
> Best Regards,
>
> Julien LELOUP
>
> -----Original Message-----
> From: Koderer, Marc [mailto:m.koderer at telekom.de]
> Sent: Friday, January 17, 2014 9:45 AM
> To: LELOUP Julien
> Cc: openstack-dev at lists.openstack.org
> Subject: [qa] RE: [Tempest - Stress Test] : implement a full SSH
> connection on "ssh_floating.py" and improve it
>
> Hello Juilen,
>
> I forwarded your mail to the correct mailing list. Please do not use the
> qa list any longer.
>
> I am happy that you are interested in stress tests. All the tests in
> tempest/stress/actions are more or less deprecated. So what you should use
> instead is the stress decorator (e.g.
> https://github.com/openstack/tempest/blob/master/tempest/api/volume/test_v
> olumes_actions.py#L55).
> Unfortunately it's not yet used for scenarios like you describe. I'd
> suggest to build a scenario test in tempest/scenario and use this
> decorator on it.
>
> Any patch like that is welcome on gerrit. If you are planning to work in
> that area for more than just a patch, a blueprint would be nice. A good
> way to coordinate your efforts is also in the QA meeting
> (https://wiki.openstack.org/wiki/Meetings/QATeamMeeting)
>
> Regards
> Marc
>
> ________________________________________
> From: LELOUP Julien [Julien.LELOUP at 3ds.com]
> Sent: Wednesday, January 15, 2014 5:57 PM
> To: openstack-qa at lists.openstack.org
> Subject: [openstack-qa] [Tempest - Stress Test] : implement a full SSH
> connection on "ssh_floating.py" and improve it
>
> Hi everyone,
>
> I’m quite new on OpenStack / Tempest and I’m actually working on stress
> tests. I want to suggest a new feature in a currently available stress
> test.
> Not sure if this email should be posted on the QA mailing list or the dev
> mailing list, but I give it a try here since it is about a Tempest stress
> test ☺
>
> At the moment the “ssh_floating.py” stress test seems really interesting
> but I would like to improve it a bit.
>
> By now this script is simulating an SSH connection by binding a TCP socket
> on the newly created instance. But this test don’t allow us to check if
> this instance is really available. I’m mostly thinking about the metadata
> service unable to provide the SSH key pair to the instance, but surely
> other scenarios can lead to an instance considered “ACTIVE” but actually
> unusable.
>
> So I’m implementing a full SSH connection test using the “paramiko” SSH
> library and a key pair generated in the same way the other test resources
> are managed in this script : either one SSH key pair for every test runs
> or a new key pair for each run (depends on the JSON configuration file).
> I don’t plan to remove the old test (TCP socket binding), rather move this
> one on a separate test function and put the full SSH connection test code
> instead.
>
> Is this feature interesting for the OpenStack community ?
> Should I create a blueprint on the Tempest project on Launchpad in order
> to provide my code through Gerrit ?
>
> On a second time, I plan to overhaul improve this “ssh_floating.py” script
> by clean the code a little bit, and add more cleaning code in order to
> avoid leaving instances/security groups/floating IP behind : I do have
> this kind of behavior right now and I already improved the teardown() on
> this way.
>
> Should I consider this code as a new functionality (thus create a
> blueprint) or should I create a defect and assign it to myself ?
>
> Cordialement / Best Regards,
>
> Julien LELOUP
>
> R&D 3DExperience Platform IaaS Factory Technology Engineer
>
>
>
>
>
> julien.leloup at 3ds.com<mailto:Julien.LELOUP at 3ds.com>
>
> [cid:image003.gif at 01CF1216.D43ECE20]
>
> 3DS.COM<http://www.3ds.com/>
>
>
> Dassault Systèmes | 10 rue Marcel Dassault, CS 40501 | 78946 Vélizy-
> Villacoublay Cedex | France
>
>
>
>
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential
> and/or privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
>
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential
> and/or privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential
> and/or privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential
> and/or privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential
> and/or privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential
> and/or privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer