[openstack-dev] [Ironic] Fuel agent proposal
Roman Prykhodchenko
rprikhodchenko at mirantis.com
Tue Dec 9 14:52:08 UTC 2014
So regarding to the 3rd-party CI. There is FuelCI that runs tests for some
stackforge projects. Attaching it to Ironic to run required tests that
detect problems in FuelAgent driver is not a big deal at all and will be
done as soon as it's necessary.
On Tue, Dec 9, 2014 at 3:45 PM, Vladimir Kozhukalov <
vkozhukalov at mirantis.com> wrote:
> s/though/throw/g
>
> Vladimir Kozhukalov
>
> On Tue, Dec 9, 2014 at 5:40 PM, Vladimir Kozhukalov <
> vkozhukalov at mirantis.com> wrote:
>
>>
>>
>> Vladimir Kozhukalov
>>
>> On Tue, Dec 9, 2014 at 3:51 PM, Dmitry Tantsur <dtantsur at redhat.com>
>> wrote:
>>
>>> Hi folks,
>>>
>>> Thank you for additional explanation, it does clarify things a bit. I'd
>>> like to note, however, that you talk a lot about how _different_ Fuel Agent
>>> is from what Ironic does now. I'd like actually to know how well it's going
>>> to fit into what Ironic does (in additional to your specific use cases).
>>> Hence my comments inline:
>>
>>
>>>
>>> On 12/09/2014 01:01 PM, Vladimir Kozhukalov wrote:
>>>
>>>> Just a short explanation of Fuel use case.
>>>>
>>>> Fuel use case is not a cloud. Fuel is a deployment tool. We install OS
>>>> on bare metal servers and on VMs
>>>> and then configure this OS using Puppet. We have been using Cobbler as
>>>> our OS provisioning tool since the beginning of Fuel.
>>>> However, Cobbler assumes using native OS installers (Anaconda and
>>>> Debian-installer). For some reasons we decided to
>>>> switch to image based approach for installing OS.
>>>>
>>>> One of Fuel features is the ability to provide advanced partitioning
>>>> schemes (including software RAIDs, LVM).
>>>> Native installers are quite difficult to customize in the field of
>>>> partitioning
>>>> (that was one of the reasons to switch to image based approach).
>>>> Moreover, we'd like to implement even more
>>>> flexible user experience. We'd like to allow user to choose which hard
>>>> drives to use for root FS, for
>>>> allocating DB. We'd like user to be able to put root FS over LV or MD
>>>> device (including stripe, mirror, multipath).
>>>> We'd like user to be able to choose which hard drives are bootable (if
>>>> any), which options to use for mounting file systems.
>>>> Many many various cases are possible. If you ask why we'd like to
>>>> support all those cases, the answer is simple:
>>>> because our users want us to support all those cases.
>>>> Obviously, many of those cases can not be implemented as image
>>>> internals, some cases can not be also implemented on
>>>> configuration stage (placing root fs on lvm device).
>>>>
>>>> As far as those use cases were rejected to be implemented in term of
>>>> IPA, we implemented so called Fuel Agent.
>>>> Important Fuel Agent features are:
>>>>
>>>> * It does not have REST API
>>>>
>>> I would not call it a feature :-P
>>>
>>> Speaking seriously, if you agent is a long-running thing and it gets
>>> it's configuration from e.g. JSON file, how can Ironic notify it of any
>>> changes?
>>>
>>> Fuel Agent is not long-running service. Currently there is no need to
>> have REST API. If we deal with kind of keep alive stuff of
>> inventory/discovery then we probably add API. Frankly, IPA REST API is not
>> REST at all. However that is not a reason to not to call it a feature and
>> through it away. It is a reason to work on it and improve. That is how I
>> try to look at things (pragmatically).
>>
>> Fuel Agent has executable entry point[s] like /usr/bin/provision. You can
>> run this entry point with options (oslo.config) and point out where to find
>> input json data. It is supposed Ironic will use ssh (currently in Fuel we
>> use mcollective) connection and run this waiting for exit code. If exit
>> code is equal to 0, provisioning is done. Extremely simple.
>>
>>
>>> * it has executable entry point[s]
>>>> * It uses local json file as it's input
>>>> * It is planned to implement ability to download input data via HTTP
>>>> (kind of metadata service)
>>>> * It is designed to be agnostic to input data format, not only Fuel
>>>> format (data drivers)
>>>> * It is designed to be agnostic to image format (tar images, file system
>>>> images, disk images, currently fs images)
>>>> * It is designed to be agnostic to image compression algorithm
>>>> (currently gzip)
>>>> * It is designed to be agnostic to image downloading protocol (currently
>>>> local file and HTTP link)
>>>>
>>> Does it support Glance? I understand it's HTTP, but it requires
>>> authentication.
>>>
>>>
>>>> So, it is clear that being motivated by Fuel, Fuel Agent is quite
>>>> independent and generic. And we are open for
>>>> new use cases.
>>>>
>>> My favorite use case is hardware introspection (aka getting data
>>> required for scheduling from a node automatically). Any ideas on this?
>>> (It's not a priority for this discussion, just curious).
>>
>>
>> That is exactly what we do in Fuel. Currently we use so called 'Default'
>> pxelinux config and all nodes being powered on are supposed to boot with so
>> called 'Bootstrap' ramdisk where Ohai based agent (not Fuel Agent) runs
>> periodically and sends hardware report to Fuel master node.
>> User then is able to look at CPU, hard drive and network info and choose
>> which nodes to use for controllers, which for computes, etc. That is what
>> nova scheduler is supposed to do (look at hardware info and choose a
>> suitable node).
>>
>> Talking about future, we are planning to re-implement inventory/discovery
>> stuff in terms of Fuel Agent (currently, this stuff is implemented as Ohai
>> based independent script). Estimation for that is March 2015.
>>
>>>
>>>
>>>
>>>> According Fuel itself, our nearest plan is to get rid of Cobbler because
>>>> in the case of image based approach it is huge overhead. The question is
>>>> which tool we can use instead of Cobbler. We need power management,
>>>> we need TFTP management, we need DHCP management. That is
>>>> exactly what Ironic is able to do. Frankly, we can implement
>>>> power/TFTP/DHCP
>>>> management tool independently, but as Devananda said, we're all working
>>>> on the same problems,
>>>> so let's do it together. Power/TFTP/DHCP management is where we are
>>>> working on the same problems,
>>>> but IPA and Fuel Agent are about different use cases. This case is not
>>>> just Fuel, any mature
>>>> deployment case require advanced partition/fs management.
>>>>
>>> Taking into consideration that you're doing a generic OS installation
>>> tool... yeah, it starts to make some sense. For cloud advanced partition is
>>> definitely a "pet" case.
>>
>>
>> Generic image based OS installation tool.
>>
>>
>>>
>>> However, for
>>>
>>>> me it is OK, if it is easily possible
>>>> to use Ironic with external drivers (not merged to Ironic and not tested
>>>> on Ironic CI).
>>>>
>>>> AFAIU, this spec https://review.openstack.org/#/c/138115/ does not
>>>> assume changing Ironic API and core.
>>>> Jim asked about how Fuel Agent will know about advanced disk
>>>> partitioning scheme if API is not
>>>> changed. The answer is simple: Ironic is supposed to send a link to
>>>> metadata service (http or local file)
>>>> where Fuel Agent can download input json data.
>>>>
>>> That's not about not changing Ironic. Changing Ironic is ok for
>>> reasonable use cases - we do a huge change right now to accommodate
>>> zapping, hardware introspection and RAID configuration.
>>>
>>> Minimal changes because we don't want to break anything. It is clear how
>> difficult to convince people to do even minimal changes. Again it is just a
>> pragmatic approach. We want to do things iteratively so as not to break
>> Ironic as well as Fuel. We just can not change all at once.
>>
>>
>>> I actually have problems with this particular statement. It does not
>>> sound like Fuel Agent will integrate enough with Ironic. This JSON file:
>>> who is going to generate it? In the most popular use case we're driven by
>>> Nova. Will Nova generate this file?
>>>
>>> If the answer is "generate it manually for every node", it's too much a
>>> "pet" case for me personally.
>>>
>>> That is how this provision data look like right now
>> https://github.com/stackforge/fuel-web/blob/master/fuel_agent_ci/samples/provision.json
>> Do you still think it is written manually? Currently Fuel Agent works as a
>> part of Fuel ecosystem. We have a service which serializes provision data
>> for us into json. Fuel Agent is agnostic to data format (data drivers). If
>> someone wants to use another format, they are welcome to implement a
>> driver.
>>
>> We assume next step will be to put provision data (disk partition scheme,
>> maybe other data) into driver_info and make Fuel Agent driver able to
>> serialize those data (special format) and implement a corresponding data
>> driver in Fuel Agent for this format. Again very simple. Maybe it is time
>> to think of having Ironic metadata service (just maybe).
>>
>> Another point is that currently Fuel stores hardware info in its own
>> database but when it is possible to get those data from Ironic (when
>> inventory stuff is implemented) we will be glad to use Ironic API for that.
>> That is what I mean when I say 'to make Fuel stuff closer to Ironic
>> abstractions'
>>
>>
>>
>>>
>>>> As Roman said, we try to be pragmatic and suggest something which does
>>>> not break anything. All changes
>>>> are supposed to be encapsulated into a driver. No API and core changes.
>>>> We have resources to support, test
>>>> and improve this driver. This spec is just a zero step. Further steps
>>>> are supposed to improve driver
>>>> so as to make it closer to Ironic abstractions.
>>>>
>>> Honestly I think you should at least write a roadmap for it - see my
>>> comments above.
>>>
>>
>> Honestly, I think writing roadmap right now is not very rational as far
>> as I am not even sure people are interested in widening Ironic use cases.
>> Some of the comments were not even constructive like "I don't understand
>> what your use case is, please use IPA".
>>
>>
>>>
>>> About testing and support: are you providing a 3rdparty CI for it? It
>>> would be a big plus as to me: we already have troubles with drivers broken
>>> accidentally.
>>
>>
>> We are flexible here but I'm not ready to answer this question right now.
>> We will try to fit Ironic requirements wherever it is possible.
>>
>>
>>>
>>>
>>>> For Ironic that means widening use cases and user community. But, as I
>>>> already said,
>>>> we are OK if Ironic does not need this feature.
>>>>
>>> I don't think we should through away your hardware provision use case,
>>> but I personally would like to see how well Fuel Agent is going to play
>>> with how Ironic and Nova operate.
>>>
>>
>> Nova is not our case. Fuel is totally about deployment. There is some in
>> common
>>
>> As I already explained, currently we need power/tftp/dhcp management
>> Ironic capabilities. Again, it is not a problem to implement this stuff
>> independently like it happened with Fuel Agent (because this use case was
>> rejected several months ago). Our suggestion is not about "let's compete
>> with IPA" it is totally about "let's work on the same problems together".
>>
>>
>>
>>>
>>>> Vladimir Kozhukalov
>>>>
>>>> On Tue, Dec 9, 2014 at 1:09 PM, Roman Prykhodchenko
>>>> <rprikhodchenko at mirantis.com <mailto:rprikhodchenko at mirantis.com>>
>>>> wrote:
>>>>
>>>> It is true that IPA and FuelAgent share a lot of functionality in
>>>> common. However there is a major difference between them which is
>>>> that they are intended to be used to solve a different problem.
>>>>
>>>> IPA is a solution for provision-use-destroy-use_by_different_user
>>>> use-case and is really great for using it for providing BM nodes for
>>>> other OS services or in services like Rackspace OnMetal. FuelAgent
>>>> itself serves for provision-use-use-…-use use-case like Fuel or
>>>> TripleO have.
>>>>
>>>> Those two use-cases require concentration on different details in
>>>> first place. For instance for IPA proper decommissioning is more
>>>> important than advanced disk management, but for FuelAgent
>>>> priorities are opposite because of obvious reasons.
>>>>
>>>> Putting all functionality to a single driver and a single agent may
>>>> cause conflicts in priorities and make a lot of mess inside both the
>>>> driver and the agent. Actually previously changes to IPA were
>>>> blocked right because of this conflict of priorities. Therefore
>>>> replacing FuelAgent by IPA in where FuelAgent is used currently does
>>>> not seem like a good option because come people (and I’m not talking
>>>> about Mirantis) might loose required features because of different
>>>> priorities.
>>>>
>>>> Having two separate drivers along with two separate agents for those
>>>> different use-cases will allow to have two independent teams that
>>>> are concentrated on what’s really important for a specific use-case.
>>>> I don’t see any problem in overlapping functionality if it’s used
>>>> differently.
>>>>
>>>>
>>>> P. S.
>>>> I realise that people may be also confused by the fact that
>>>> FuelAgent is actually called like that and is used only in Fuel atm.
>>>> Our point is to make it a simple, powerful and what’s more important
>>>> a generic tool for provisioning. It is not bound to Fuel or Mirantis
>>>> and if it will cause confusion in the future we will even be happy
>>>> to give it a different and less confusing name.
>>>>
>>>> P. P. S.
>>>> Some of the points of this integration do not look generic enough or
>>>> nice enough. We look pragmatic on the stuff and are trying to
>>>> implement what’s possible to implement as the first step. For sure
>>>> this is going to have a lot more steps to make it better and more
>>>> generic.
>>>>
>>>>
>>>> On 09 Dec 2014, at 01:46, Jim Rollenhagen <jim at jimrollenhagen.com
>>>>> <mailto:jim at jimrollenhagen.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On December 8, 2014 2:23:58 PM PST, Devananda van der Veen
>>>>> <devananda.vdv at gmail.com <mailto:devananda.vdv at gmail.com>> wrote:
>>>>>
>>>>>> I'd like to raise this topic for a wider discussion outside of the
>>>>>> hallway
>>>>>> track and code reviews, where it has thus far mostly remained.
>>>>>>
>>>>>> In previous discussions, my understanding has been that the Fuel
>>>>>> team
>>>>>> sought to use Ironic to manage "pets" rather than "cattle" - and
>>>>>> doing
>>>>>> so
>>>>>> required extending the API and the project's functionality in
>>>>>> ways that
>>>>>> no
>>>>>> one else on the core team agreed with. Perhaps that understanding
>>>>>> was
>>>>>> wrong
>>>>>> (or perhaps not), but in any case, there is now a proposal to add
>>>>>> a
>>>>>> FuelAgent driver to Ironic. The proposal claims this would meet
>>>>>> that
>>>>>> teams'
>>>>>> needs without requiring changes to the core of Ironic.
>>>>>>
>>>>>> https://review.openstack.org/#/c/138115/
>>>>>>
>>>>>
>>>>> I think it's clear from the review that I share the opinions
>>>>> expressed in this email.
>>>>>
>>>>> That said (and hopefully without derailing the thread too much),
>>>>> I'm curious how this driver could do software RAID or LVM without
>>>>> modifying Ironic's API or data model. How would the agent know how
>>>>> these should be built? How would an operator or user tell Ironic
>>>>> what the disk/partition/volume layout would look like?
>>>>>
>>>>> And before it's said - no, I don't think vendor passthru API calls
>>>>> are an appropriate answer here.
>>>>>
>>>>> // jim
>>>>>
>>>>>
>>>>>> The Problem Description section calls out four things, which have
>>>>>> all
>>>>>> been
>>>>>> discussed previously (some are here [0]). I would like to address
>>>>>> each
>>>>>> one,
>>>>>> invite discussion on whether or not these are, in fact, problems
>>>>>> facing
>>>>>> Ironic (not whether they are problems for someone, somewhere),
>>>>>> and then
>>>>>> ask
>>>>>> why these necessitate a new driver be added to the project.
>>>>>>
>>>>>>
>>>>>> They are, for reference:
>>>>>>
>>>>>> 1. limited partition support
>>>>>>
>>>>>> 2. no software RAID support
>>>>>>
>>>>>> 3. no LVM support
>>>>>>
>>>>>> 4. no support for hardware that lacks a BMC
>>>>>>
>>>>>> #1.
>>>>>>
>>>>>> When deploying a partition image (eg, QCOW format), Ironic's PXE
>>>>>> deploy
>>>>>> driver performs only the minimal partitioning necessary to
>>>>>> fulfill its
>>>>>> mission as an OpenStack service: respect the user's request for
>>>>>> root,
>>>>>> swap,
>>>>>> and ephemeral partition sizes. When deploying a whole-disk image,
>>>>>> Ironic
>>>>>> does not perform any partitioning -- such is left up to the
>>>>>> operator
>>>>>> who
>>>>>> created the disk image.
>>>>>>
>>>>>> Support for arbitrarily complex partition layouts is not required
>>>>>> by,
>>>>>> nor
>>>>>> does it facilitate, the goal of provisioning physical servers via
>>>>>> a
>>>>>> common
>>>>>> cloud API. Additionally, as with #3 below, nothing prevents a
>>>>>> user from
>>>>>> creating more partitions in unallocated disk space once they have
>>>>>> access to
>>>>>> their instance. Therefor, I don't see how Ironic's minimal
>>>>>> support for
>>>>>> partitioning is a problem for the project.
>>>>>>
>>>>>> #2.
>>>>>>
>>>>>> There is no support for defining a RAID in Ironic today, at all,
>>>>>> whether
>>>>>> software or hardware. Several proposals were floated last cycle;
>>>>>> one is
>>>>>> under review right now for DRAC support [1], and there are
>>>>>> multiple
>>>>>> call
>>>>>> outs for RAID building in the state machine mega-spec [2]. Any
>>>>>> such
>>>>>> support
>>>>>> for hardware RAID will necessarily be abstract enough to support
>>>>>> multiple
>>>>>> hardware vendor's driver implementations and both in-band
>>>>>> creation (via
>>>>>> IPA) and out-of-band creation (via vendor tools).
>>>>>>
>>>>>> Given the above, it may become possible to add software RAID
>>>>>> support to
>>>>>> IPA
>>>>>> in the future, under the same abstraction. This would closely tie
>>>>>> the
>>>>>> deploy agent to the images it deploys (the latter image's kernel
>>>>>> would
>>>>>> be
>>>>>> dependent upon a software RAID built by the former), but this
>>>>>> would
>>>>>> necessarily be true for the proposed FuelAgent as well.
>>>>>>
>>>>>> I don't see this as a compelling reason to add a new driver to the
>>>>>> project.
>>>>>> Instead, we should (plan to) add support for software RAID to the
>>>>>> deploy
>>>>>> agent which is already part of the project.
>>>>>>
>>>>>> #3.
>>>>>>
>>>>>> LVM volumes can easily be added by a user (after provisioning)
>>>>>> within
>>>>>> unallocated disk space for non-root partitions. I have not yet
>>>>>> seen a
>>>>>> compelling argument for doing this within the provisioning phase.
>>>>>>
>>>>>> #4.
>>>>>>
>>>>>> There are already in-tree drivers [3] [4] [5] which do not
>>>>>> require a
>>>>>> BMC.
>>>>>> One of these uses SSH to connect and run pre-determined commands.
>>>>>> Like
>>>>>> the
>>>>>> spec proposal, which states at line 122, "Control via SSH access
>>>>>> feature
>>>>>> intended only for experiments in non-production environment," the
>>>>>> current
>>>>>> SSHPowerDriver is only meant for testing environments. We could
>>>>>> probably
>>>>>> extend this driver to do what the FuelAgent spec proposes, as far
>>>>>> as
>>>>>> remote
>>>>>> power control for cheap always-on hardware in testing
>>>>>> environments with
>>>>>> a
>>>>>> pre-shared key.
>>>>>>
>>>>>> (And if anyone wonders about a use case for Ironic without
>>>>>> external
>>>>>> power
>>>>>> control ... I can only think of one situation where I would
>>>>>> rationally
>>>>>> ever
>>>>>> want to have a control-plane agent running inside a
>>>>>> user-instance: I am
>>>>>> both the operator and the only user of the cloud.)
>>>>>>
>>>>>>
>>>>>> ----------------
>>>>>>
>>>>>> In summary, as far as I can tell, all of the problem statements
>>>>>> upon
>>>>>> which
>>>>>> the FuelAgent proposal are based are solvable through incremental
>>>>>> changes
>>>>>> in existing drivers, or out of scope for the project entirely. As
>>>>>> another
>>>>>> software-based deploy agent, FuelAgent would duplicate the
>>>>>> majority of
>>>>>> the
>>>>>> functionality which ironic-python-agent has today.
>>>>>>
>>>>>> Ironic's driver ecosystem benefits from a diversity of
>>>>>> hardware-enablement
>>>>>> drivers. Today, we have two divergent software deployment drivers
>>>>>> which
>>>>>> approach image deployment differently: "agent" drivers use a local
>>>>>> agent to
>>>>>> prepare a system and download the image; "pxe" drivers use a
>>>>>> remote
>>>>>> agent
>>>>>> and copy the image over iSCSI. I don't understand how a second
>>>>>> driver
>>>>>> which
>>>>>> duplicates the functionality we already have, and shares the same
>>>>>> goals
>>>>>> as
>>>>>> the drivers we already have, is beneficial to the project.
>>>>>>
>>>>>> Doing the same thing twice just increases the burden on the team;
>>>>>> we're
>>>>>> all
>>>>>> working on the same problems, so let's do it together.
>>>>>>
>>>>>> -Devananda
>>>>>>
>>>>>>
>>>>>> [0]
>>>>>> https://blueprints.launchpad.net/ironic/+spec/ironic-
>>>>>> python-agent-partition
>>>>>>
>>>>>> [1] https://review.openstack.org/#/c/107981/
>>>>>>
>>>>>> [2]
>>>>>> https://review.openstack.org/#/c/133828/11/specs/kilo/new-
>>>>>> ironic-state-machine.rst
>>>>>>
>>>>>>
>>>>>> [3]
>>>>>> http://git.openstack.org/cgit/openstack/ironic/tree/ironic/
>>>>>> drivers/modules/snmp.py
>>>>>>
>>>>>> [4]
>>>>>> http://git.openstack.org/cgit/openstack/ironic/tree/ironic/
>>>>>> drivers/modules/iboot.py
>>>>>>
>>>>>> [5]
>>>>>> http://git.openstack.org/cgit/openstack/ironic/tree/ironic/
>>>>>> drivers/modules/ssh.py
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------
>>>>>> ------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> <mailto:OpenStack-dev at lists.openstack.org>
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141209/9cdd402f/attachment-0001.html>
More information about the OpenStack-dev
mailing list