Open Stack

On Thu, Apr 13, 2017 at 2:14 AM, Dan Sneddon <dsneddon at redhat.com> wrote:

> On 04/12/2017 01:22 PM, Thomas Herve wrote:
> > On Wed, Apr 12, 2017 at 9:00 PM, Dan Sneddon <dsneddon at redhat.com>
> wrote:
> >> I'm implementing predictable control plane IPs for spine/leaf, and I'm
> >> running into a problem implementing this in the TripleO Heat templates.
> >>
> >> I have a review in progress [1] that works, but fails on upgrade, so I'm
> >> looking for an alternative approach. I'm trying to influence the IP
> >> address that is selected for overcloud nodes' Control Plane IP. Here is
> >> the current construct:
> >>
> >>   Controller:
> >>     type: OS::TripleO::Server
> >>     metadata:
> >>       os-collect-config:
> >>         command: {get_param: ConfigCommand}
> >>     properties:
> >>       image: {get_param: controllerImage}
> >>       image_update_policy: {get_param: ImageUpdatePolicy}
> >>       flavor: {get_param: OvercloudControlFlavor}
> >>       key_name: {get_param: KeyName}
> >>       networks:
> >>         - network: ctlplane  # <- Here's where the port is created
> >>
> >> If I add fixed_ip: to the networks element at the end of the above, I
> >> can select an IP address from the 'ctlplane' network, like this:
> >>
> >>       networks:
> >>         - network: ctlplane
> >>           fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
> >>
> >> But the problem is that if I pass a blank string to fixed_ip, I get an
> >> error on deployment. This means that the old behavior of automatically
> >> selecting an IP doesn't work.
> >>
> >> I thought I has solved this by passing an external Neutron port, like
> this:
> >>
> >>       networks:
> >>         - network: ctlplane
> >>           port: {get_attr: [ControlPlanePort, port_id]}
> >>
> >> Which works for deployments, but that fails on upgrades, since the
> >> original port was created as part of the Nova::Server resource, instead
> >> of being an external resource.
> >
> > Can you detail how it fails? I was under the impression we never
> > replaced servers no matter what (or we try to do that, at least). Is
> > the issue that your new port is not the correct one?
> >
> >> I'm now looking for a way to use Heat conditionals to apply the fixed_ip
> >> only if the value is not unset. Looking at the intrinsic functions [2],
> >> I don't see a way to do this. Is what I'm trying to do with Heat
> possible?
> >
> > You should be able to write something like that (not tested):
> >
> > networks:
> >   if:
> >     - <my condition>
> >     - network: ctlplane
> >       fixed_ip: {get_attr: [ControlPlanePort, ip_address]}
> >     - network: ctlplane
> >
> > The question is how to define your condition. Maybe:
> >
> > conditions:
> >   fixed_ip_condition:
> >      not:
> >         equals:
> >           - {get_attr: [ControlPlanePort, ip_address]}
> >           - ''
> >
> > To get back to the problem you stated first.
> >
> >
> >> Another option I'm exploring is conditionally applying resources. It
> >> appears that would require duplicating the entire TripleO::Server stanza
> >> in *-role.yaml so that there is one that uses fixed_ip and one that does
> >> not. Which one is applied would be based on a condition that tested
> >> whether fixed_ip was blank or not. The downside of that is that it would
> >> make the role definition confusing because there would be a large
> >> resource that was implemented twice, with only one line difference
> >> between them.
> >
> > You can define properties with conditions, so you shouldn't need to
> > rewrite everything.
> >
>
> Thomas,
>
> Thanks, I will try your suggestions and that should get me closer.
>
> The full error log is available here:
> http://logs.openstack.org/78/413278/11/check-tripleo/gate-
> tripleo-ci-centos-7-ovb-updates/8d91762/console.html
>
> We do an interface_detach/attach when a port is replaced.
It seems to be failing[1] as this is not implemented for ironic/baremetal
driver.  I could see a patch[2] to add that functionality though.

[1]
http://logs.openstack.org/78/413278/11/check-tripleo/gate-tripleo-ci-centos-7-ovb-updates/8d91762/logs/undercloud/var/log/nova/nova-compute.txt.gz#_2017-04-12_00_26_15_475

[2] https://review.openstack.org/#/c/419975/

We retry a few times to check whether the detach/attach is complete(it's an
async operation in nova and takes time), so the cryptic error below is
coming from tenacity library which fails after the configured number of
attempts.

Here are the errors I am getting:
>
> 2017-04-12 00:26:34.436655 | 2017-04-12 00:26:29Z
> [overcloud-CephStorage-bkucn6ign34i-0-2yq2jbtwuu7k.CephStorage]:
> UPDATE_FAILED  RetryError: resources.CephStorage: RetryError[<Future at
> 0xdd62550 state=finished returned bool>]
> 2017-04-12 00:26:34.436808 | 2017-04-12 00:26:29Z
> [overcloud-CephStorage-bkucn6ign34i-0-2yq2jbtwuu7k]: UPDATE_FAILED
> RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
> state=finished returned bool>]
> 2017-04-12 00:26:34.436903 | 2017-04-12 00:26:29Z
> [overcloud-CephStorage-bkucn6ign34i.0]: UPDATE_FAILED  resources[0]:
> RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
> state=finished returned bool>]
> 2017-04-12 00:26:34.436989 | 2017-04-12 00:26:29Z
> [overcloud-CephStorage-bkucn6ign34i]: UPDATE_FAILED  resources[0]:
> RetryError: resources.CephStorage: RetryError[<Future at 0xdd62550
> state=finished returned bool>]
> 2017-04-12 00:26:34.437078 | 2017-04-12 00:26:30Z
> [overcloud-Controller-3lf3jauv4cbc-0-ydowkb3nwsso.Controller]:
> UPDATE_FAILED  RetryError: resources.Controller: RetryError[<Future at
> 0xdc79b50 state=finished returned bool>]
> 2017-04-12 00:26:34.437173 | 2017-04-12 00:26:30Z
> [overcloud-Controller-3lf3jauv4cbc-0-ydowkb3nwsso]: UPDATE_FAILED
> RetryError: resources.Controller: RetryError[<Future at 0xdc79b50
> state=finished returned bool>]
> 2017-04-12 00:26:34.437269 | 2017-04-12 00:26:30Z [CephStorage]:
> UPDATE_FAILED  resources.CephStorage: resources[0]: RetryError:
> resources.CephStorage: RetryError[<Future at 0xdd62550 state=finished
> returned bool>]
>
> --
> Dan Sneddon         |  Senior Principal Software Engineer
> dsneddon at redhat.com |  redhat.com/openstack
> dsneddon:irc        |  @dxs:twitter
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards,
Rabi Misra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170413/9531759e/attachment.html>