[openstack-dev] [Mistral] Local vs. Scalable Engine

W Chan m4d.coder at gmail.com
Mon Mar 24 17:37:44 UTC 2014


I have the following murano-ci failure for my last patch set.
https://murano-ci.mirantis.com/job/mistral_master_on_commit/194/  Since I
modified the API launch script in mistral, is that the cause of this
failure here?  Do I have to make changes to the tempest test?  Please
advise.  Thanks.


On Fri, Mar 21, 2014 at 3:20 AM, Renat Akhmerov <rakhmerov at mirantis.com>wrote:

> Alright, thanks Winson!
>
> Team, please review.
>
> Renat Akhmerov
> @ Mirantis Inc.
>
>
>
> On 21 Mar 2014, at 06:43, W Chan <m4d.coder at gmail.com> wrote:
>
> I submitted a rough draft for review @
> https://review.openstack.org/#/c/81941/.  Instead of using the pecan
> hook, I added a class property for the transport in the abstract engine
> class.  On the pecan app setup, I passed the shared transport to the engine
> on load.  Please provide feedback.  Thanks.
>
>
> On Mon, Mar 17, 2014 at 9:37 AM, Ryan Petrello <
> ryan.petrello at dreamhost.com> wrote:
>
>> Changing the configuration object at runtime is not thread-safe.  If you
>> want to share objects with controllers, I'd suggest checking out Pecan's
>> hook functionality.
>>
>>
>> http://pecan.readthedocs.org/en/latest/hooks.html#implementating-a-pecan-hook
>>
>> e.g.,
>>
>> class SpecialContextHook(object):
>>
>>     def __init__(self, some_obj):
>>         self.some_obj = some_obj
>>
>>     def before(self, state):
>>         # In any pecan controller, `pecan.request` is a thread-local
>> webob.Request instance,
>>         # allowing you to access `pecan.request.context['foo']` in your
>> controllers.  In this example,
>>         # self.some_obj could be just about anything - a Python
>> primitive, or an instance of some class
>>         state.request.context = {
>>             'foo': self.some_obj
>>         }
>>
>> ...
>>
>> wsgi_app = pecan.Pecan(
>>     my_package.controllers.root.RootController(),
>>     hooks=[SpecialContextHook(SomeObj(1, 2, 3))]
>> )
>>
>> ---
>> Ryan Petrello
>> Senior Developer, DreamHost
>> ryan.petrello at dreamhost.com
>>
>> On Mar 14, 2014, at 8:53 AM, Renat Akhmerov <rakhmerov at mirantis.com>
>> wrote:
>>
>> > Take a look at method get_pecan_config() in mistral/api/app.py. It's
>> where you can pass any parameters into pecan app (see a dictionary
>> 'cfg_dict' initialization). They can be then accessed via pecan.conf as
>> described here:
>> http://pecan.readthedocs.org/en/latest/configuration.html#application-configuration.
>> If I understood the problem correctly this should be helpful.
>> >
>> > Renat Akhmerov
>> > @ Mirantis Inc.
>> >
>> >
>> >
>> > On 14 Mar 2014, at 05:14, Dmitri Zimine <dz at stackstorm.com> wrote:
>> >
>> >> We have access to all configuration parameters in the context of
>> api.py. May be you don't pass it but just instantiate it where you need it?
>> Or I may misunderstand what you're trying to do...
>> >>
>> >> DZ>
>> >>
>> >> PS: can you generate and update mistral.config.example to include new
>> oslo messaging options? I forgot to mention it on review on time.
>> >>
>> >>
>> >> On Mar 13, 2014, at 11:15 AM, W Chan <m4d.coder at gmail.com> wrote:
>> >>
>> >>> On the transport variable, the problem I see isn't with passing the
>> variable to the engine and executor.  It's passing the transport into the
>> API layer.  The API layer is a pecan app and I currently don't see a way
>> where the transport variable can be passed to it directly.  I'm looking at
>> https://github.com/stackforge/mistral/blob/master/mistral/cmd/api.py#L50and
>> https://github.com/stackforge/mistral/blob/master/mistral/api/app.py#L44.
>>  Do you have any suggestion?  Thanks.
>> >>>
>> >>>
>> >>> On Thu, Mar 13, 2014 at 1:30 AM, Renat Akhmerov <
>> rakhmerov at mirantis.com> wrote:
>> >>>
>> >>> On 13 Mar 2014, at 10:40, W Chan <m4d.coder at gmail.com> wrote:
>> >>>
>> >>>>    * I can write a method in base test to start local executor.  I
>> will do that as a separate bp.
>> >>> Ok.
>> >>>
>> >>>>    * After the engine is made standalone, the API will communicate
>> to the engine and the engine to the executor via the oslo.messaging
>> transport.  This means that for the "local" option, we need to start all
>> three components (API, engine, and executor) on the same process.  If the
>> long term goal as you stated above is to use separate launchers for these
>> components, this means that the API launcher needs to duplicate all the
>> logic to launch the engine and the executor. Hence, my proposal here is to
>> move the logic to launch the components into a common module and either
>> have a single generic launch script that launch specific components based
>> on the CLI options or have separate launch scripts that reference the
>> appropriate launch function from the common module.
>> >>> Ok, I see your point. Then I would suggest we have one script which
>> we could use to run all the components (any subset of of them). So for
>> those components we specified when launching the script we use this local
>> transport. Btw, scheduler eventually should become a standalone component
>> too, so we have 4 components.
>> >>>
>> >>>>    * The RPC client/server in oslo.messaging do not determine the
>> transport.  The transport is determine via oslo.config and then given
>> explicitly to the RPC client/server.
>> https://github.com/stackforge/mistral/blob/master/mistral/engine/scalable/engine.py#L31and
>> https://github.com/stackforge/mistral/blob/master/mistral/cmd/task_executor.py#L63are examples for the client and server respectively.  The in process Queue
>> is instantiated within this transport object from the fake driver.  For the
>> "local" option, all three components need to share the same transport in
>> order to have the Queue in scope. Thus, we will need some method to have
>> this transport object visible to all three components and hence my proposal
>> to use a global variable and a factory method.
>> >>> I'm still not sure I follow your point here.. Looking at the links
>> you provided I see this:
>> >>>
>> >>> transport = messaging.get_transport(cfg.CONF)
>> >>>
>> >>> So my point here is we can make this call once in the launching
>> script and pass it to engine/executor (and now API too if we want it to be
>> launched by the same script). Of course, we'll have to change the way how
>> we initialize these components, but I believe we can do it. So it's just a
>> dependency injection. And in this case we wouldn't need to use a global
>> variable. Am I still missing something?
>> >>>
>> >>>
>> >>> Renat Akhmerov
>> >>> @ Mirantis Inc.
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> OpenStack-dev mailing list
>> >>> OpenStack-dev at lists.openstack.org
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> OpenStack-dev mailing list
>> >>> OpenStack-dev at lists.openstack.org
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140324/ff3f9b03/attachment.html>


More information about the OpenStack-dev mailing list