Open Stack

> 2) I think you misunderstand what is the difference between
> upstart/systemd and Pacemaker in this case. There are many cases when you
> need to have syncrhonized view of the cluster. Otherwise you will hit
> split-brain situations and have your cluster misfunctioning. Until
> OpenStack provides us with such means there is no other way than using
> Pacemaker/Zookeper/etc.
>

Could you please give some examples of those 'many cases' for openstack
specifically?
As for my 'misunderstanding' - openstack services only need to be always
up, not more than that.
Upstart does a perfect job there.


> 3) Regarding Neutron agents - we discussed it many times - you need to be
> able to control and clean up stuff after some service crashed. Currently,
> Neutron does not provide reliable ways to do it. If your agent dies and
> does not clean up ip addresses from the network namespace you will get into
> the situation of ARP duplication which will be a kind of split brain
> described in item #2. I personally as a system architect and administrator
> do not believe for this to change in at least several years for OpenStack
> so we will be using Pacemaker for a very long period of time.
>

This has been changed already, and a while ago.
OCF infrastructure around neutron agents has never helped neutron in any
meaningful way and is just an artifact from the dark past.
The reasons are: pacemaker/ocf doesn't have enough intelligence to know
when to engage, as a result, any cleanup could only be achieved through
manual operations. I don't need to remind you how many bugs were in ocf
scripts which brought whole clusters down after those manual operations.
So it's just a way better to go with simple standard tools with fine-grain
control.
Same applies to any other openstack service (again, not rabbitmq/galera)

> so we will be using Pacemaker for a very long period of time.
Not for neutron, sorry. As soon as we finish the last bit of such cleanup,
which is targeted for 8.0

Now, back to the topic - we may decide to use some more sophisticated
> integral node health attribute which can be used with Pacemaker as well as
> to put node into some kind of maintenance mode. We can leverage User
> Maintenance Mode feature here or just simply stop particular services and
> disable particular haproxy backends.
>

I think this kind of attribute, although being analyzed by pacemaker/ocf,
doesn't need any new OS service to be put under pacemaker control.

Thanks,
Eugene.


>
> On Mon, Oct 5, 2015 at 11:57 PM, Eugene Nikanorov <enikanorov at mirantis.com
> > wrote:
>
>>
>>>>
>>> Mirantis does control neither Rabbitmq or Galera. Mirantis cannot assure
>>> their quality as well.
>>>
>>
>> Correct, and rabbitmq was always the pain in the back, preventing any *real
>> *enterprise usage of openstack where reliability does matter.
>>
>>
>>> > 2) it has terrible UX
>>>>
>>>
>>> It looks like personal opinion. I'd like to see surveys or operators
>>> feedbacks. Also, this statement is not constructive as it doesn't have
>>> alternative solutions.
>>>
>>
>> The solution is to get rid of terrible UX wherever possible (i'm not
>> saying it is always possible, of course)
>> upstart is just so much better.
>> And yes, this is my personal opinion and is a summary of escalation
>> team's experience.
>>
>>
>>>
>>>> > 3) it is not reliable
>>>>
>>>
>>> I would say openstack services are not HA reliable. So OCF scripts are
>>> reaction of operators on these problems. Many of them have child-ish issues
>>> from release to release. Operators made OCF scripts to fix these problems.
>>> A lot of openstack are stateful, so they require some kind of stickiness or
>>> synchronization. Openstack services doesn't have simple health-check
>>> functionality so it's hard to say it's running well or not. Sighup is still
>>> a problem for many of openstack services. Etc/etc So, let's be constructive
>>> here.
>>>
>>
>> Well, I prefer to be responsible for what I know and maintain. Thus, I
>> state that neutron doesn't need to be managed by pacemaker, neither server,
>> nor all kinds of agents, and that's the path that neutron team will be
>> taking.
>>
>> Thanks,
>> Eugene.
>>
>>>
>>>
>>>> >
>>>>
>>>> I disagree with #1 as I do not agree that should be a criteria for an
>>>> open-source project.  Considering pacemaker is at the core of our
>>>> controller setup, I would argue that if these are in fact true we need
>>>> to be using something else.  I would agree that it is a terrible UX
>>>> but all the clustering software I've used fall in this category.  I'd
>>>> like more information on how it is not reliable. Do we have numbers to
>>>> backup these claims?
>>>>
>>>> > (3) is not evaluation of the project itself, but just a logical
>>>> consequence
>>>> > of (1) and (2).
>>>> > As a part of escalation team I can say that it has cost our team
>>>> thousands
>>>> > of man hours of head-scratching, staring at pacemaker logs which
>>>> value are
>>>> > usually slightly below zero.
>>>> >
>>>> > Most of openstack services (in fact, ALL api servers) are stateless,
>>>> they
>>>> > don't require any cluster management (also, they don't need to be
>>>> moved in
>>>> > case of lack of space).
>>>> > Statefull services like neutron agents have their states being a
>>>> function of
>>>> > db state and are able to syncronize it with the server without
>>>> external
>>>> > "help".
>>>> >
>>>>
>>>> So it's not an issue with moving services so much as being able to
>>>> stop the services when a condition is met. Have we tested all OS
>>>> services to ensure they do function 100% when out of disk space?  I
>>>> would assume that glance might have issues with image uploads if there
>>>> is no space to handle a request.
>>>>
>>>> > So now usage of pacemaker can be only justified for cases where
>>>> service's
>>>> > clustering mechanism requires active monitoring (rabbitmq, galera)
>>>> > But even there, examples when we are better off without pacemaker are
>>>> all
>>>> > around.
>>>> >
>>>> > Thanks,
>>>> > Eugene.
>>>> >
>>>>
>>>> After I sent this email, I had further discussions around the issues
>>>> that I'm facing and it may not be completely related to disk space. I
>>>> think we might be relying on the expectation that the local rabbitmq
>>>> is always available but I need to look into that. Either way, I
>>>> believe we still should continue to discuss this issue as we are
>>>> managing services in multiple ways on a single host. Additionally I do
>>>> not believe that we really perform quality health checks on our
>>>> services.
>>>>
>>>> Thanks,
>>>> -Alex
>>>>
>>>>
>>>> >
>>>> > On Mon, Oct 5, 2015 at 1:34 PM, Sergey Vasilenko <
>>>> svasilenko at mirantis.com>
>>>> > wrote:
>>>> >>
>>>> >>
>>>> >> On Mon, Oct 5, 2015 at 12:22 PM, Eugene Nikanorov
>>>> >> <enikanorov at mirantis.com> wrote:
>>>> >>>
>>>> >>> No pacemaker for os services, please.
>>>> >>> We'll be moving out neutron agents from pacemaker control in 8.0,
>>>> other
>>>> >>> os services don't need it too.
>>>> >>
>>>> >>
>>>> >> could you please provide your arguments.
>>>> >>
>>>> >>
>>>> >> /sv
>>>> >>
>>>> >>
>>>> __________________________________________________________________________
>>>> >> OpenStack Development Mailing List (not for usage questions)
>>>> >> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> __________________________________________________________________________
>>>> > OpenStack Development Mailing List (not for usage questions)
>>>> > Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Yours Faithfully,
> Vladimir Kuklin,
> Fuel Library Tech Lead,
> Mirantis, Inc.
> +7 (495) 640-49-04
> +7 (926) 702-39-68
> Skype kuklinvv
> 35bk3, Vorontsovskaya Str.
> Moscow, Russia,
> www.mirantis.com <http://www.mirantis.ru/>
> www.mirantis.ru
> vkuklin at mirantis.com
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151006/04ceaae6/attachment.html>