Open Stack

Hi Boris,

 Thanks for comment.
 Sorry for the misleading context in the etherpad. It sounds like Eris
will do everything, but it is not.
 I think I need to add more details to etherpad (I will do that)
 Short answer is we are not into  "reinventing wheels" and we will use
all possible existing tools.
 # Me and Gautam are working on a document about what are the gaps,
and what tool should use on what purpose and why.
 # I think that would be the long answer and  we definitely need your
feed back on this.
 #  I will share that here..

 For, Rally,
  Actually we are using Rally in Eris. Here is the PoC we are working on (WIP).
 # At this point, PoC is in it's very early stage. Not all the
Concepts are implemented yet..
  code: https://github.com/gautamdivgi/eris
  doc: https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/22872097/Extreme+Testing+Demo

 You may find lot more doc's about Eris in here,
 https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/13393034/Eris+-+Extreme+Testing+Framework+for+OpenStack

 For an example, we are considering,
 Rally: For control plane load injection (Current PoC we don't use Rally
 Shaker: For data plane load injection
 os-fault: fault injection ( lot of work need to be done here)

 Here is a one example:
 Eris is focus on realistic outage scenarios such as, SDN controller
failure, Storage back-end failure, infra L2/L3 SW failure ,etc,,
 If those are vendor specific appliances, then how should we do
failure injection and metrics gathering?
 Eris will provide plugin mechanism to plug vendor drivers ( provide
by the vendor).
 In testing, Eris will use Rally and Shaker for load generation and
inject deterministic/random HW/SW failures in those
 appliances and gather the metrics. Then,  Eris (or Rally or any other
tool or mix of them) can analyze the result and
 create the final report.

 Sorry for ranting.....

--- Regards,
Sampath



On Wed, Nov 1, 2017 at 4:02 PM, Boris Pavlovic <boris at pavlovic.me> wrote:
> Sam,
>
>>  Etherpad: https://etherpad.openstack.org/p/SYD-extreme-testing
>
>
> I really don't want to sound like a person that say use Rally my best ever
> project blablbab and other BS.
> I think that "reinventing wheels" approach is how humanity evolves and
> that's why I like this effort in any case.
>
> But really, I read carefully etherpad and I really see in description of
> Eris just plain Rally as is:
>
> - Rally allows you to create tests as YAML
> - Rally allows you to inject in various actions during the load (Rally
> Hooks) which makes it easy to do chaos and other kind of testing
> - Rally is pluggable and you can write even your own Runners (scenario
> executors) that will generate load pattern that you need
> - Rally has SLAs plugins (that can deeply analyze result of test cases) and
> say whatever they pass or not
> - We are working on feature that allows you to mix different workloads in
> parallel (and generate more realistic load)
> -.....
>
> So it would be really nice if you can share gaps that you faced that are
> blocking you to use directly Rally..
>
> Thanks!
>
> Best regards,
> Boris Pavlovic
>
>
> On Tue, Oct 31, 2017 at 10:50 PM, Sam P <sam47priya at gmail.com> wrote:
>>
>> Hi All,
>>
>>  Sending out a gentle reminder of Sydney Summit Forum Session
>> regarding this topic.
>>
>>  Extreme/Destructive Testing
>>  Tuesday, November 7, 1:50pm-2:30pm
>>  Sydney Convention and Exhibition Centre - Level 4 - C4.11
>>
>> [https://www.openstack.org/summit/sydney-2017/summit-schedule/events/20470/extremedestructive-testing]
>>  Eatherpad: https://etherpad.openstack.org/p/SYD-extreme-testing
>>
>>  Your participation in this session would be greatly appreciated.
>> --- Regards,
>> Sampath
>>
>>
>>
>> On Mon, Aug 14, 2017 at 11:43 PM, Tim Bell <Tim.Bell at cern.ch> wrote:
>> > +1 for Boris’ suggestion. Many of us use Rally to probe our clouds and
>> > have
>> > significant tooling behind it to integrate with local availability
>> > reporting
>> > and trouble ticketing systems. It would be much easier to deploy new
>> > functionality such as you propose if it was integrated into an existing
>> > project framework (such as Rally).
>> >
>> >
>> >
>> > Tim
>> >
>> >
>> >
>> > From: Boris Pavlovic <boris at pavlovic.me>
>> > Reply-To: "OpenStack Development Mailing List (not for usage questions)"
>> > <openstack-dev at lists.openstack.org>
>> > Date: Monday, 14 August 2017 at 12:57
>> > To: "OpenStack Development Mailing List (not for usage questions)"
>> > <openstack-dev at lists.openstack.org>
>> > Cc: openstack-operators <openstack-operators at lists.openstack.org>
>> > Subject: Re: [openstack-dev] [QA][LCOO] MEX-ops-meetup: OpenStack
>> > Extreme
>> > Testing
>> >
>> >
>> >
>> > Sam,
>> >
>> >
>> >
>> > Seems like a good plan and huge topic ;)
>> >
>> >
>> >
>> > I would as well suggest to take a look at the similar efforts in
>> > OpenStack:
>> >
>> > - Failure injection: https://github.com/openstack/os-faults
>> >
>> > - Rally Hooks Mechanism (to inject in rally scenarios failures):
>> >
>> > https://rally.readthedocs.io/en/latest/plugins/implementation/hook_and_trigger_plugins.html
>> >
>> >
>> >
>> >
>> >
>> > Best regards,
>> > Boris Pavlovic
>> >
>> >
>> >
>> >
>> >
>> > On Mon, Aug 14, 2017 at 2:35 AM, Sam P <sam47priya at gmail.com> wrote:
>> >
>> > Hi All,
>> >
>> > This is a follow up for OpenStack Extreme Testing session[1]
>> > we did in MEX-ops-meetup.
>> >
>> > Quick intro for those who were not there:
>> > In this work, we proposed to add new testing framework for openstack.
>> > This framework will provides tool for create tests with destructive
>> > scenarios which will check for High Availability, failover and
>> > recovery of OpenStack cloud.
>> > Please refer the link on top of the [1] for further details.
>> >
>> > Follow up:
>> > We are planning periodic irc meeting and have an irc
>> > channel for discussion. I will get back to you with those details soon.
>> >
>> > At that session, we did not have time to discuss last 3 items,
>> > Reference architectures
>> >  We are discussing about the reference architecture in [2].
>> >
>> > What sort of failures do you see today in your environment?
>> >  Currently we are considering, service failures, backend services (mq,
>> > DB, etc.) failures,
>> >  Network sw failures..etc. To begin with the implementation, we are
>> > considering to start with
>> >  service failures. Please let us know what failures are more frequent
>> > in your environment.
>> >
>> > Emulation/Simulation mechanisms, etc.
>> >  Rather than doing actual scale, load, or performance tests, we are
>> > thinking to build a emulation/simulation mechanism
>> > to get the predictions or result of how will openstack behave on such
>> > situations.
>> > This interesting idea was proposed by the Gautam and need more
>> > discussion on this.
>> >
>> > Please let us know you questions or comments.
>> >
>> > Request to Mike Perez:
>> >  We discussed about synergies with openstack assertion tags and other
>> > efforts to do similar testing in openstack.
>> >  Could you please give some info or pointer of previous discussions.
>> >
>> > [1] https://etherpad.openstack.org/p/MEX-ops-extreme-testing
>> > [2]
>> >
>> > https://openstack-lcoo.atlassian.net/wiki/spaces/LCOO/pages/15477787/Extreme+Testing-Vision+Arch
>> >
>> > --- Regards,
>> > Sampath
>> >
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> >
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>