Open Stack

On 04/13/2017 12:01 AM, Rabi Mishra wrote:
> On Thu, Apr 13, 2017 at 2:14 AM, Dan Sneddon <dsneddon at redhat.com
> <mailto:dsneddon at redhat.com>> wrote:
> 
>     On 04/12/2017 01:22 PM, Thomas Herve wrote:
>     > On Wed, Apr 12, 2017 at 9:00 PM, Dan Sneddon <dsneddon at redhat.com
>     <mailto:dsneddon at redhat.com>> wrote:
>     >> I'm implementing predictable control plane IPs for spine/leaf,
>     and I'm
>     >> running into a problem implementing this in the TripleO Heat
>     templates.
>     >>
>     >> I have a review in progress [1] that works, but fails on upgrade,
>     so I'm
>     >> looking for an alternative approach. I'm trying to influence the IP
>     >> address that is selected for overcloud nodes' Control Plane IP.
>     Here is
>     >> the current construct:
>     >>
>     >>   Controller:
>     >>     type: OS::TripleO::Server
>     >>     metadata:
>     >>       os-collect-config:
>     >>         command: {get_param: ConfigCommand}
>     >>     properties:
>     >>       image: {get_param: controllerImage}
>     >>       image_update_policy: {get_param: ImageUpdatePolicy}
>     >>       flavor: {get_param: OvercloudControlFlavor}
>     >>       key_name: {get_param: KeyName}
>     >>       networks:
>     >>         - network: ctlplane  # <- Here's where the port is created
>     >>
>     >> If I add fixed_ip: to the networks element at the end of the above, I
>     >> can select an IP address from the 'ctlplane' network, like this:
>     >>
>     >>       networks:
>     >>         - network: ctlplane
>     >>           fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
>     >>
>     >> But the problem is that if I pass a blank string to fixed_ip, I
>     get an
>     >> error on deployment. This means that the old behavior of
>     automatically
>     >> selecting an IP doesn't work.
>     >>
>     >> I thought I has solved this by passing an external Neutron port,
>     like this:
>     >>
>     >>       networks:
>     >>         - network: ctlplane
>     >>           port: {get_attr: [ControlPlanePort, port_id]}
>     >>
>     >> Which works for deployments, but that fails on upgrades, since the
>     >> original port was created as part of the Nova::Server resource,
>     instead
>     >> of being an external resource.
>     >
>     > Can you detail how it fails? I was under the impression we never
>     > replaced servers no matter what (or we try to do that, at least). Is
>     > the issue that your new port is not the correct one?
>     >
>     >> I'm now looking for a way to use Heat conditionals to apply the
>     fixed_ip
>     >> only if the value is not unset. Looking at the intrinsic
>     functions [2],
>     >> I don't see a way to do this. Is what I'm trying to do with Heat
>     possible?
>     >
>     > You should be able to write something like that (not tested):
>     >
>     > networks:
>     >   if:
>     >     - <my condition>
>     >     - network: ctlplane
>     >       fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
>     >     - network: ctlplane
>     >
>     > The question is how to define your condition. Maybe:
>     >
>     > conditions:
>     >   fixed_ip_condition:
>     >      not:
>     >         equals:
>     >           - {get_attr: [ControlPlanePort, ip_address]}
>     >           - ''
>     >
>     > To get back to the problem you stated first.
>     >
>     >
>     >> Another option I'm exploring is conditionally applying resources. It
>     >> appears that would require duplicating the entire TripleO::Server
>     stanza
>     >> in *-role.yaml so that there is one that uses fixed_ip and one
>     that does
>     >> not. Which one is applied would be based on a condition that tested
>     >> whether fixed_ip was blank or not. The downside of that is that
>     it would
>     >> make the role definition confusing because there would be a large
>     >> resource that was implemented twice, with only one line difference
>     >> between them.
>     >
>     > You can define properties with conditions, so you shouldn't need to
>     > rewrite everything.
>     >
> 
>     Thomas,
> 
>     Thanks, I will try your suggestions and that should get me closer.
> 
>     The full error log is available here:
>     http://logs.openstack.org/78/413278/11/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/8d91762/console.html
>     <http://logs.openstack.org/78/413278/11/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/8d91762/console.html>
> 
> We do an interface_detach/attach when a port is replaced.
> It seems to be failing[1] as this is not implemented for
> ironic/baremetal driver.  I could see a patch[2] to add that
> functionality though.
> 
> [1]
> http://logs.openstack.org/78/413278/11/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/8d91762/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-04-12_00_26_15_475
> 
> [2] https://review.openstack.org/#/c/419975/
> 
> We retry a few times to check whether the detach/attach is complete(it's
> an async operation in nova and takes time), so the cryptic error below
> is coming from tenacity library which fails after the configured number
> of attempts.
> 
>     Here are the errors I am getting:
> 
>     2017-04-12 00:26:34.436655 | 2017-04-12 00:26:29Z
>     [overcloud-CephStorage-bkucn6ign34i-0-2yq2jbtwuu7k.CephStorage]:
>     UPDATE_FAILED  RetryError: resources.CephStorage: RetryError[<Future at
>     0xdd62550 state=finished returned bool>]
>     2017-04-12 00:26:34.436808 | 2017-04-12 00:26:29Z
>     [overcloud-CephStorage-bkucn6ign34i-0-2yq2jbtwuu7k]: UPDATE_FAILED
>     RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
>     state=finished returned bool>]
>     2017-04-12 00:26:34.436903 | 2017-04-12 00:26:29Z
>     [overcloud-CephStorage-bkucn6ign34i.0]: UPDATE_FAILED  resources[0]:
>     RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
>     state=finished returned bool>]
>     2017-04-12 00:26:34.436989 | 2017-04-12 00:26:29Z
>     [overcloud-CephStorage-bkucn6ign34i]: UPDATE_FAILED  resources[0]:
>     RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
>     state=finished returned bool>]
>     2017-04-12 00:26:34.437078 | 2017-04-12 00:26:30Z
>     [overcloud-Controller-3lf3jauv4cbc-0-ydowkb3nwsso.Controller]:
>     UPDATE_FAILED  RetryError: resources.Controller: RetryError[<Future at
>     0xdc79b50 state=finished returned bool>]
>     2017-04-12 00:26:34.437173 | 2017-04-12 00:26:30Z
>     [overcloud-Controller-3lf3jauv4cbc-0-ydowkb3nwsso]: UPDATE_FAILED
>     RetryError: resources.Controller: RetryError[<Future at 0xdc79b50
>     state=finished returned bool>]
>     2017-04-12 00:26:34.437269 | 2017-04-12 00:26:30Z [CephStorage]:
>     UPDATE_FAILED  resources.CephStorage: resources[0]: RetryError:
>     resources.CephStorage: RetryError[<Future at 0xdd62550 state=finished
>     returned bool>]
> 
>     --
>     Dan Sneddon         |  Senior Principal Software Engineer
>     dsneddon at redhat.com <mailto:dsneddon at redhat.com> | 
>     redhat.com/openstack <http://redhat.com/openstack>
>     dsneddon:irc        |  @dxs:twitter
> 
> -- 
> Regards,
> Rabi Misra

Rabi,

Thanks for the explanation on why the port replacement isn't working.

Unfortunately, I tried the recommendation that Thomas Herve made about
using conditionals, but that failed. It appears that you can't use a
get_attr inside of a conditional statement, I get an error to that
effect. I can use a get_param, but that doesn't help me check a value in
a nested stack.

So in order for me to achieve the goal of being able to conditionally
assign the IP automatically or from a fixed_ip, I need either a Heat
feature (to resolve get_attr inside of conditional statements), or the
Nova patch to implement attach/detach in Ironic.

Thanks to both of you for the assistance.

-- 
Dan Sneddon         |  Senior Principal Software Engineer
dsneddon at redhat.com |  redhat.com/openstack
dsneddon:irc        |  @dxs:twitter