Open Stack

Hi Saravanan,

Ah, ok. I'd assumed ironic didn't touch the bootloader as the docs
show a mkconfig & reboot to customize grub for localboot:
http://docs.openstack.org/project-install-guide/baremetal/draft/advanced.html#appending-kernel-parameters-to-boot-instances.

Thanks,
Ollie

On 13 December 2016 at 11:03, Saravanan KR <skramaja at redhat.com> wrote:
> Hi Oliver,
>
> During the deployment, Ironic will start the node with IPA ramdisk,
> which will copy the overcloud image to the node's disk, then configure
> the grub cfg [1], set the node to local boot and reboot the node.
> After which the node will be boot with the overcloud image. So no
> reboot required in the overcloud image, as the "deploy steps" will be
> run by the IPA itself.
>
> Regards,
> Saravanan KR
>
> [1] https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/extensions/image.py#L136
>
>
> On Tue, Dec 13, 2016 at 4:18 PM, Oliver Walsh <owalsh at redhat.com> wrote:
>> Hi Yolanda,
>>
>>> these changes will be created by ironic before the image is deployed
>>
>> The question I'm asking is how are the changed created without a reboot?
>>
>> Typically when setting this via a manual change or via tuned the
>> process is to modify /etc/default/grub, run grub2-mkconfig, and then
>> reboot. Are you suggesting we drop in a pre-build grub cfg before
>> deployment?
>>
>> Thanks,
>> Ollie
>>
>> On 13 December 2016 at 10:33, Yolanda Robla Mota <yroblamo at redhat.com> wrote:
>>> It won't need a reboot, because these changes will be created by ironic
>>> before the image is deployed. So it will boot with the right parameters. The
>>> alternative of doing with puppet after the image was deployed, needed a
>>> reboot, because the changes were done post-deploy.
>>> So ironic build steps are pre-deploy without reboot, puppet changes are
>>> post-deploy with a reboot.
>>>
>>> On Tue, Dec 13, 2016 at 11:24 AM, Oliver Walsh <owalsh at redhat.com> wrote:
>>>>
>>>> Hi,
>>>>
>>>> Saravanan wrote:
>>>> > If ironic "deploy steps" can configure this "tuned" setting and run the
>>>> > command
>>>>
>>>> How does this avoid the reboot?
>>>>
>>>> Yolanda wrote:
>>>> > The idea will be to define custom deployment steps for ironic, like
>>>> > including the kernel boot parameters.
>>>>
>>>> Again, is this avoiding the reboot or just moving it?
>>>>
>>>> Thanks,
>>>> Ollie
>>>>
>>>> On 13 December 2016 at 09:02, Saravanan KR <skramaja at redhat.com> wrote:
>>>> > Hi Yolanda,
>>>> >
>>>> > The flow for "tuned" will be like set the "tuned" configuration files,
>>>> > and then activate the profile by running the command "tuned-adm
>>>> > tuned-profile-nfv". This command will actually write the required
>>>> > configuration files for tuning the host. If ironic "deploy steps" can
>>>> > configure this "tuned" setting and run the command, then it is good
>>>> > enough.
>>>> >
>>>> > Regards,
>>>> > Saravanan KR
>>>> >
>>>> > On Tue, Dec 13, 2016 at 1:04 PM, Yolanda Robla Mota
>>>> > <yroblamo at redhat.com> wrote:
>>>> >> Hi Saravanan
>>>> >> Thanks for your comments. With this new module, I guess a reboot is
>>>> >> still
>>>> >> needed after os-host-config ?
>>>> >> Right now we have been guided by TripleO and Ironic people to start
>>>> >> using
>>>> >> what in Ironic is called "custom deployment steps". An initial spec is
>>>> >> reflected here:
>>>> >> https://review.openstack.org/#/c/382091
>>>> >>
>>>> >> The idea will be to define custom deployment steps for ironic, like
>>>> >> including the kernel boot parameters. Can that be a solution for your
>>>> >> "tuned" needs as well?
>>>> >>
>>>> >> Best
>>>> >> Yolanda
>>>> >>
>>>> >> On Tue, Dec 13, 2016 at 7:59 AM, Saravanan KR <skramaja at redhat.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Hello,
>>>> >>>
>>>> >>> Thanks Yolanda for starting the thread. The list of requirements in
>>>> >>> the host configuration, related to boot parameters and reboot are:
>>>> >>>
>>>> >>> * DPDK - For vfio-pci driver binding, iommu support on kernel args is
>>>> >>> mandatory, which has to be configured before os-net-config runs
>>>> >>> * DPDK & RealTime - Enabling "tuned" profile for nfv or rt, will
>>>> >>> update the boot parameters and a reboot is required
>>>> >>> * Other items mentioned by Yolanda
>>>> >>>
>>>> >>> If it is configuring only, the boot parameters, then ironic's deploy
>>>> >>> feature may help, but there are other requirement to enable the
>>>> >>> "tuned" profile which tunes the host for the required configuration,
>>>> >>> which also requires reboot, as it will alter the boot parameters. If
>>>> >>> we can collate the all the configurations which requires reboot
>>>> >>> together, we will improve the reboot time. And if we reboot before the
>>>> >>> actual openstack services are started, then the reboot time _may_
>>>> >>> improve.
>>>> >>>
>>>> >>> Can I propose a *new* module for TripleO deployments, like >
>>>> >>> os-host-config <, which will run after os-collect-config and before
>>>> >>> os-net-config, then we can collate all the host specific configuration
>>>> >>> inside this module. This module can be a set of ansible scripts, which
>>>> >>> will only configure the host. Ofcource the parameter to this module
>>>> >>> should be provided via os-collect-config. Separating the host
>>>> >>> configuration will help in the containerized TripleO deployment also.
>>>> >>>
>>>> >>> Or any other better alternatives are welcome.
>>>> >>>
>>>> >>> Please pour in your views if you think for/against it.
>>>> >>>
>>>> >>> Regards,
>>>> >>> Saravanan KR
>>>> >>>
>>>> >>>
>>>> >>> On Fri, Dec 2, 2016 at 9:31 PM, Yolanda Robla Mota
>>>> >>> <yroblamo at redhat.com>
>>>> >>> wrote:
>>>> >>> > Hi , Dmitry
>>>> >>> > That's what i didn't get very clear. If all the deployment steps are
>>>> >>> > pre-imaging as that statement says, or every deploy step could be
>>>> >>> > isolated
>>>> >>> > and configured somehow.
>>>> >>> > I'm also a bit confused with that spec, because it mixes the concept
>>>> >>> > of
>>>> >>> > "deployment steps", will all the changes needed for runtime RAID.
>>>> >>> > Could it
>>>> >>> > be possible to separate into two separate ones?
>>>> >>> >
>>>> >>> > ----- Original Message -----
>>>> >>> > From: "Dmitry Tantsur" <dtantsur at redhat.com>
>>>> >>> > To: openstack-dev at lists.openstack.org
>>>> >>> > Sent: Friday, December 2, 2016 3:51:30 PM
>>>> >>> > Subject: Re: [openstack-dev] [tripleo] [ironic] Need to update
>>>> >>> > kernel
>>>> >>> > parameters on local boot
>>>> >>> >
>>>> >>> > On 12/02/2016 01:28 PM, Yolanda Robla Mota wrote:
>>>> >>> >> Hi Dmitry
>>>> >>> >>
>>>> >>> >> So we've been looking at that spec you suggested, but we are
>>>> >>> >> wondering
>>>> >>> >> if that will be useful for our use case. As the text says:
>>>> >>> >>
>>>> >>> >> The ``ironic-python-agent`` project and ``agent`` driver will be
>>>> >>> >> adjusted to
>>>> >>> >> support ``get_deploy_steps``. That way, ``ironic-python-agent``
>>>> >>> >> will be
>>>> >>> >> able
>>>> >>> >> to declare deploy steps to run prior to disk imaging, and operators
>>>> >>> >> will be
>>>> >>> >> able to extend ``ironic-python-agent`` to add any custom step.
>>>> >>> >>
>>>> >>> >> Our needs are different, actually we need to create a deployment
>>>> >>> >> step
>>>> >>> >> after imaging. We'd need an step that drops config on
>>>> >>> >> /etc/default/grub ,
>>>> >>> >> and updates it. This is a post-imaging deploy step, that modifies
>>>> >>> >> the base
>>>> >>> >> image. Could ironic support these kind of steps, if there is a base
>>>> >>> >> system
>>>> >>> >> to just define per-user steps?
>>>> >>> >
>>>> >>> > I thought that all deployment operations are converted to steps,
>>>> >>> > with
>>>> >>> > partitioning, writing the image, writing the configdrive and
>>>> >>> > installing
>>>> >>> > the boot
>>>> >>> > loader being four default ones (as you see, two steps actually
>>>> >>> > happen
>>>> >>> > after the
>>>> >>> > image is written).
>>>> >>> >
>>>> >>> >>
>>>> >>> >> The idea we had on mind is:
>>>> >>> >> - from tripleo, add a property to each flavor, that defines the
>>>> >>> >> boot
>>>> >>> >> parameters:  openstack flavor set compute --property
>>>> >>> >> os:kernel_boot_params='abc'
>>>> >>> >> - define a "ironic post-imaging deploy step", that will grab this
>>>> >>> >> property from the flavor, drop it on /etc/default/grub and
>>>> >>> >> regenerate it
>>>> >>> >> - then on local boot, the proper kernel parameters will be applied
>>>> >>> >>
>>>> >>> >> What is your feedback there?
>>>> >>> >>
>>>> >>> >> ----- Original Message -----
>>>> >>> >> From: "Dmitry Tantsur" <dtantsur at redhat.com>
>>>> >>> >> To: openstack-dev at lists.openstack.org
>>>> >>> >> Sent: Friday, December 2, 2016 12:44:29 PM
>>>> >>> >> Subject: Re: [openstack-dev] [tripleo] [ironic] Need to update
>>>> >>> >> kernel
>>>> >>> >> parameters on local boot
>>>> >>> >>
>>>> >>> >> On 11/28/2016 04:46 PM, Jay Faulkner wrote:
>>>> >>> >>>
>>>> >>> >>>> On Nov 28, 2016, at 7:36 AM, Yolanda Robla Mota
>>>> >>> >>>> <yroblamo at redhat.com>
>>>> >>> >>>> wrote:
>>>> >>> >>>>
>>>> >>> >>>> Hi, good afternoon
>>>> >>> >>>>
>>>> >>> >>>> I wanted to start an email thread about how to properly setup
>>>> >>> >>>> kernel
>>>> >>> >>>> parameters on local boot, for our overcloud images on TripleO.
>>>> >>> >>>> These parameters may vary depending on the needs of our end
>>>> >>> >>>> users,
>>>> >>> >>>> and even can be different ( for different roles ) per deployment.
>>>> >>> >>>> As an
>>>> >>> >>>> example, we need it for:
>>>> >>> >>>> - enable FIPS kernel in terms of security
>>>> >>> >>>> (https://bugs.launchpad.net/tripleo/+bug/1640235)
>>>> >>> >>>> - enable functionality for DPDK/SR-IOV
>>>> >>> >>>> (https://review.openstack.org/#/c/331564/)
>>>> >>> >>>> - enable rd.iscsi.firmware=1 flag (this for the ramdisk image)
>>>> >>> >>>> - etc..
>>>> >>> >>>>
>>>> >>> >>>> So far, the solutions we got were on several directions:
>>>> >>> >>>>
>>>> >>> >>>> 1. Update the golden overcloud-full image with virt-customize,
>>>> >>> >>>> modifying /etc/default/grub settings according to our needs: this
>>>> >>> >>>> is a
>>>> >>> >>>> manual process, not really driven by TripleO. End users will want
>>>> >>> >>>> to avoid
>>>> >>> >>>> manual steps as much as possible. Also if we announce that
>>>> >>> >>>> OpenStack ships
>>>> >>> >>>> features in TripleO like DPDK, SR-IOV... doesn't make sense to
>>>> >>> >>>> tell end
>>>> >>> >>>> users that if they want to consume that feature, they need to do
>>>> >>> >>>> manual
>>>> >>> >>>> updates on the image. It shall be natively supported, or
>>>> >>> >>>> configurable per
>>>> >>> >>>> TripleO environments.
>>>> >>> >>>>
>>>> >>> >>>> 2. Create our own images using diskimage-builder and custom
>>>> >>> >>>> elements:
>>>> >>> >>>> in this case, we have the problem that the partners will loose
>>>> >>> >>>> support, as
>>>> >>> >>>> building their own images is good for upstream, but not accepted
>>>> >>> >>>> into the
>>>> >>> >>>> OSP environment. Also the combination of images needed can be
>>>> >>> >>>> huge, that can
>>>> >>> >>>> be a blocker for QA.
>>>> >>> >>>>
>>>> >>> >>>> 3. Add Ironic support for it. Images can be uploaded to glance,
>>>> >>> >>>> and
>>>> >>> >>>> some properties can be set on metadata, like a json with kernel
>>>> >>> >>>> parameters.
>>>> >>> >>>> Ironic will modify these kernel parameters when deploying the
>>>> >>> >>>> image (in a
>>>> >>> >>>> similar way that when it installs bootloader, or generates
>>>> >>> >>>> partitions).
>>>> >>> >>>>
>>>> >>> >>>
>>>> >>> >>> This has been proposed before in ironic-specs
>>>> >>> >>> (https://review.openstack.org/#/c/331564/) and was rejected, as it
>>>> >>> >>> would
>>>> >>> >>> require Ironic to reach out and modify image contents, which
>>>> >>> >>> traditionally
>>>> >>> >>> has been considered out of scope for Ironic. I would personally
>>>> >>> >>> recommend
>>>> >>> >>> #4, as post-boot automation is the safest way to configure
>>>> >>> >>> node-specific
>>>> >>> >>> options inside an image.
>>>> >>> >>
>>>> >>> >> I'm still a bit divided about our decision back then.. On one hand,
>>>> >>> >> this does
>>>> >>> >> seem somewhat out of scope. On the other, I quite understand why
>>>> >>> >> reboot
>>>> >>> >> is
>>>> >>> >> suboptimal. I wonder if the ongoing deploy steps work will actually
>>>> >>> >> solve it by
>>>> >>> >> allowing hardware managers to provide additional deploy steps.
>>>> >>> >>
>>>> >>> >> Yolanda, you may want to check the spec
>>>> >>> >> https://review.openstack.org/#/c/382091/
>>>> >>> >> as it lays the foundation for the deploy steps idea.
>>>> >>> >>
>>>> >>> >>>
>>>> >>> >>> Thanks,
>>>> >>> >>> Jay Faulkner
>>>> >>> >>> OSIC
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>>> 4. Configure it post-deployment: there can be some puppet element
>>>> >>> >>>> that updates kernel parameters. But it will need a node reboot to
>>>> >>> >>>> be
>>>> >>> >>>> applied, and it's very far from being optimal and acceptable for
>>>> >>> >>>> the end
>>>> >>> >>>> users. Reboots are slow, they can be a problem depending on the
>>>> >>> >>>> number of
>>>> >>> >>>> nodes/hardware, and also the timing of reboot shall be totally
>>>> >>> >>>> controlled
>>>> >>> >>>> (after all puppet has been applied properly).
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> In the first three cases, we also hit the problem that TripleO
>>>> >>> >>>> only
>>>> >>> >>>> accepts one single overcloud image for all deployments - there is
>>>> >>> >>>> no way to
>>>> >>> >>>> instruct TripleO to upload and use several images, depending on
>>>> >>> >>>> the node
>>>> >>> >>>> type (although Ironic supports it). Also, we are worried about
>>>> >>> >>>> upgrade paths
>>>> >>> >>>> if we do image customizations. We need a clear way to move
>>>> >>> >>>> forward on it.
>>>> >>> >>>>
>>>> >>> >>>> So, we'd like to discuss the possible options there and the
>>>> >>> >>>> action
>>>> >>> >>>> items to take (raise bugs, create some blueprints...). To
>>>> >>> >>>> summarize, our end
>>>> >>> >>>> goal is the following:
>>>> >>> >>>>
>>>> >>> >>>> - need to map overcloud-full images to roles
>>>> >>> >>>> - need to be done in an automated way, no manual steps enforced,
>>>> >>> >>>> and
>>>> >>> >>>> in a way that can pass properly quality controls
>>>> >>> >>>> - reboots are sub-optimal
>>>> >>> >>>>
>>>> >>> >>>> What are your thoughts there?
>>>> >>> >>>>
>>>> >>> >>>> Best,
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> Yolanda Robla
>>>> >>> >>>> yroblamo at redhat.com
>>>> >>> >>>> Principal Software Engineer - NFV Partner Engineer
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>>
>>>> >>> >>>> __________________________________________________________________________
>>>> >>> >>>> OpenStack Development Mailing List (not for usage questions)
>>>> >>> >>>> Unsubscribe:
>>>> >>> >>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>>
>>>> >>> >>> __________________________________________________________________________
>>>> >>> >>> OpenStack Development Mailing List (not for usage questions)
>>>> >>> >>> Unsubscribe:
>>>> >>> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>> >>>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> __________________________________________________________________________
>>>> >>> >> OpenStack Development Mailing List (not for usage questions)
>>>> >>> >> Unsubscribe:
>>>> >>> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> __________________________________________________________________________
>>>> >>> >> OpenStack Development Mailing List (not for usage questions)
>>>> >>> >> Unsubscribe:
>>>> >>> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>> >>
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> > __________________________________________________________________________
>>>> >>> > OpenStack Development Mailing List (not for usage questions)
>>>> >>> > Unsubscribe:
>>>> >>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>> >
>>>> >>> >
>>>> >>> >
>>>> >>> > __________________________________________________________________________
>>>> >>> > OpenStack Development Mailing List (not for usage questions)
>>>> >>> > Unsubscribe:
>>>> >>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>>
>>>> >>>
>>>> >>> __________________________________________________________________________
>>>> >>> OpenStack Development Mailing List (not for usage questions)
>>>> >>> Unsubscribe:
>>>> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Yolanda Robla Mota
>>>> >> NFV Partner Engineer
>>>> >> yroblamo at redhat.com
>>>> >> +34 605641639
>>>> >>
>>>> >>
>>>> >> __________________________________________________________________________
>>>> >> OpenStack Development Mailing List (not for usage questions)
>>>> >> Unsubscribe:
>>>> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> >>
>>>> >
>>>> >
>>>> > __________________________________________________________________________
>>>> > OpenStack Development Mailing List (not for usage questions)
>>>> > Unsubscribe:
>>>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> --
>>> Yolanda Robla Mota
>>> NFV Partner Engineer
>>> yroblamo at redhat.com
>>> +34 605641639
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev