Open Stack

On Wed, Apr 19, 2017 at 02:51:28PM -0700, Dan Sneddon wrote:
> On 04/13/2017 12:01 AM, Rabi Mishra wrote:
> > On Thu, Apr 13, 2017 at 2:14 AM, Dan Sneddon <dsneddon at redhat.com
> > <mailto:dsneddon at redhat.com>> wrote:
> > 
> >     On 04/12/2017 01:22 PM, Thomas Herve wrote:
> >     > On Wed, Apr 12, 2017 at 9:00 PM, Dan Sneddon <dsneddon at redhat.com
> >     <mailto:dsneddon at redhat.com>> wrote:
> >     >> I'm implementing predictable control plane IPs for spine/leaf,
> >     and I'm
> >     >> running into a problem implementing this in the TripleO Heat
> >     templates.
> >     >>
> >     >> I have a review in progress [1] that works, but fails on upgrade,
> >     so I'm
> >     >> looking for an alternative approach. I'm trying to influence the IP
> >     >> address that is selected for overcloud nodes' Control Plane IP.
> >     Here is
> >     >> the current construct:
> >     >>
> >     >>   Controller:
> >     >>     type: OS::TripleO::Server
> >     >>     metadata:
> >     >>       os-collect-config:
> >     >>         command: {get_param: ConfigCommand}
> >     >>     properties:
> >     >>       image: {get_param: controllerImage}
> >     >>       image_update_policy: {get_param: ImageUpdatePolicy}
> >     >>       flavor: {get_param: OvercloudControlFlavor}
> >     >>       key_name: {get_param: KeyName}
> >     >>       networks:
> >     >>         - network: ctlplane  # <- Here's where the port is created
> >     >>
> >     >> If I add fixed_ip: to the networks element at the end of the above, I
> >     >> can select an IP address from the 'ctlplane' network, like this:
> >     >>
> >     >>       networks:
> >     >>         - network: ctlplane
> >     >>           fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
> >     >>
> >     >> But the problem is that if I pass a blank string to fixed_ip, I
> >     get an
> >     >> error on deployment. This means that the old behavior of
> >     automatically
> >     >> selecting an IP doesn't work.
> >     >>
> >     >> I thought I has solved this by passing an external Neutron port,
> >     like this:
> >     >>
> >     >>       networks:
> >     >>         - network: ctlplane
> >     >>           port: {get_attr: [ControlPlanePort, port_id]}
> >     >>
> >     >> Which works for deployments, but that fails on upgrades, since the
> >     >> original port was created as part of the Nova::Server resource,
> >     instead
> >     >> of being an external resource.
> >     >
> >     > Can you detail how it fails? I was under the impression we never
> >     > replaced servers no matter what (or we try to do that, at least). Is
> >     > the issue that your new port is not the correct one?
> >     >
> >     >> I'm now looking for a way to use Heat conditionals to apply the
> >     fixed_ip
> >     >> only if the value is not unset. Looking at the intrinsic
> >     functions [2],
> >     >> I don't see a way to do this. Is what I'm trying to do with Heat
> >     possible?
> >     >
> >     > You should be able to write something like that (not tested):
> >     >
> >     > networks:
> >     >   if:
> >     >     - <my condition>
> >     >     - network: ctlplane
> >     >       fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
> >     >     - network: ctlplane
> >     >
> >     > The question is how to define your condition. Maybe:
> >     >
> >     > conditions:
> >     >   fixed_ip_condition:
> >     >      not:
> >     >         equals:
> >     >           - {get_attr: [ControlPlanePort, ip_address]}
> >     >           - ''
> >     >
> >     > To get back to the problem you stated first.
> >     >
> >     >
> >     >> Another option I'm exploring is conditionally applying resources. It
> >     >> appears that would require duplicating the entire TripleO::Server
> >     stanza
> >     >> in *-role.yaml so that there is one that uses fixed_ip and one
> >     that does
> >     >> not. Which one is applied would be based on a condition that tested
> >     >> whether fixed_ip was blank or not. The downside of that is that
> >     it would
> >     >> make the role definition confusing because there would be a large
> >     >> resource that was implemented twice, with only one line difference
> >     >> between them.
> >     >
> >     > You can define properties with conditions, so you shouldn't need to
> >     > rewrite everything.
> >     >
> > 
> >     Thomas,
> > 
> >     Thanks, I will try your suggestions and that should get me closer.
> > 
> >     The full error log is available here:
> >     http://logs.openstack.org/78/413278/11/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/8d91762/console.html
> >     <http://logs.openstack.org/78/413278/11/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/8d91762/console.html>
> > 
> > We do an interface_detach/attach when a port is replaced.
> > It seems to be failing[1] as this is not implemented for
> > ironic/baremetal driver.  I could see a patch[2] to add that
> > functionality though.
> > 
> > [1]
> > http://logs.openstack.org/78/413278/11/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/8d91762/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-04-12_00_26_15_475
> > 
> > [2] https://review.openstack.org/#/c/419975/
> > 
> > We retry a few times to check whether the detach/attach is complete(it's
> > an async operation in nova and takes time), so the cryptic error below
> > is coming from tenacity library which fails after the configured number
> > of attempts.
> > 
> >     Here are the errors I am getting:
> > 
> >     2017-04-12 00:26:34.436655 | 2017-04-12 00:26:29Z
> >     [overcloud-CephStorage-bkucn6ign34i-0-2yq2jbtwuu7k.CephStorage]:
> >     UPDATE_FAILED  RetryError: resources.CephStorage: RetryError[<Future at
> >     0xdd62550 state=finished returned bool>]
> >     2017-04-12 00:26:34.436808 | 2017-04-12 00:26:29Z
> >     [overcloud-CephStorage-bkucn6ign34i-0-2yq2jbtwuu7k]: UPDATE_FAILED
> >     RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
> >     state=finished returned bool>]
> >     2017-04-12 00:26:34.436903 | 2017-04-12 00:26:29Z
> >     [overcloud-CephStorage-bkucn6ign34i.0]: UPDATE_FAILED  resources[0]:
> >     RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
> >     state=finished returned bool>]
> >     2017-04-12 00:26:34.436989 | 2017-04-12 00:26:29Z
> >     [overcloud-CephStorage-bkucn6ign34i]: UPDATE_FAILED  resources[0]:
> >     RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
> >     state=finished returned bool>]
> >     2017-04-12 00:26:34.437078 | 2017-04-12 00:26:30Z
> >     [overcloud-Controller-3lf3jauv4cbc-0-ydowkb3nwsso.Controller]:
> >     UPDATE_FAILED  RetryError: resources.Controller: RetryError[<Future at
> >     0xdc79b50 state=finished returned bool>]
> >     2017-04-12 00:26:34.437173 | 2017-04-12 00:26:30Z
> >     [overcloud-Controller-3lf3jauv4cbc-0-ydowkb3nwsso]: UPDATE_FAILED
> >     RetryError: resources.Controller: RetryError[<Future at 0xdc79b50
> >     state=finished returned bool>]
> >     2017-04-12 00:26:34.437269 | 2017-04-12 00:26:30Z [CephStorage]:
> >     UPDATE_FAILED  resources.CephStorage: resources[0]: RetryError:
> >     resources.CephStorage: RetryError[<Future at 0xdd62550 state=finished
> >     returned bool>]
> > 
> >     --
> >     Dan Sneddon         |  Senior Principal Software Engineer
> >     dsneddon at redhat.com <mailto:dsneddon at redhat.com> | 
> >     redhat.com/openstack <http://redhat.com/openstack>
> >     dsneddon:irc        |  @dxs:twitter
> > 
> > -- 
> > Regards,
> > Rabi Misra
> 
> Rabi,
> 
> Thanks for the explanation on why the port replacement isn't working.
> 
> Unfortunately, I tried the recommendation that Thomas Herve made about
> using conditionals, but that failed. It appears that you can't use a
> get_attr inside of a conditional statement, I get an error to that
> effect. I can use a get_param, but that doesn't help me check a value in
> a nested stack.

Yes I noticed the same thing recently, which is a limitation of conditions
that IMO we should look at fixing - everywhere else in HOT templates
get_param and get_attr can be used interchangably.

As a workaround could you move the conditional into the port resource
nested stack?  E.g if you returned the list we pass to "networks" for the
server resource from the new ControlPlanePort nested stack perhaps, and put
whatever logic we need to generate it inside the port nested stack?

Attributes from e.g *-role.yaml would then be parameters in the nested
stack, so you could do the conditional as Thomas suggested, or perhaps
it's even simpler if you can just return different data from the
ctlplane.yaml and ctlplane_from_pool.yaml templates?

Steve