[openstack-dev] [Ironic] Proposal to add a new repository
Devananda van der Veen
devananda.vdv at gmail.com
Tue Jun 23 00:11:57 UTC 2015
Oh - one more thing. If ahc-tools depends on data gathered by
enovance/hardware, then I'm not sure it makes sense to import one to
openstack/ without the other.
-Deva
On Mon, Jun 22, 2015 at 5:08 PM Devananda van der Veen <
devananda.vdv at gmail.com> wrote:
> I'm
> On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge <trown at redhat.com> wrote:
>
>>
>>
>> On 06/22/2015 10:40 AM, Dmitry Tantsur wrote:
>> > On 06/22/2015 04:19 PM, Devananda van der Veen wrote:
>> >> Hi John,
>> >>
>> >> Thanks for the excellent summary! I found it very helpful to get caught
>> >> up. I'd like to make sure I understand the direction ahc is going. A
>> >> couple questions...
>>
>> Thanks for your interest.
>>
>> >
>> > Let me add my $0.5 :)
>> >
>> >>
>> >> I see that ahc is storing its information in swift. That's clever, but
>> >> if Ironic provided a blob store for each node, would that be better?
>>
>> If the blob is large enough, this would be better. Originally we stored
>> the data in the extra column of the Ironic db, but that proved disastrous:
>>
>> https://bugs.launchpad.net/ironic-inspector/+bug/1461252
>>
>
>
>> >>
>> >> We discussed adding a search API to Ironic at the Vancouver summit,
>> >> though no work has been done on that yet, afaik. If ahc is going to
>> grow
>> >> a REST API for searching for nodes based on specific criteria that it
>> >> discovered, could/should we combine these within Ironic's API?
>> >
>> > I think John meant having API to replace scripts, so I guess search
>> > won't help. When we're talking about advanced matching, we're talking
>> > about the following:
>> > 1. We have a ramdisk tool (based on [8]) to get some insane of facts
>> > from withing the ramdisk (say, 1000 of them)
>> > 2. We have an Inspector plugin to put them all in Swift (or Ironic blob
>> > storage as above)
>> > 3. We have config files (aka rules) written in special JSON-alike DSL to
>> > do matching (one of the weak points is that these are files - I'd like
>> > API endpoint to accept these rules instead).
>> > 4. We have a script to run this DSL and get some output (match/not match
>> > + some matched variables - similar to what regexps do).
>> > As I understood it John want the latter to become an API endpoint,
>> > accepting rules (and maybe node UUIDs) and outputting some result.
>> >
>> > Not sure about benchmarking here, but again, it's probably an API
>> > endpoint that accepts some minimal expectations, and puts failed nodes
>> > to maintenance mode, if they fail to comply (again, that's how I
>> > understood it).
>> >
>> > It's not hard to make these API endpoints part of Inspector, but it's
>> > somewhat undesirable to have them optional...
>> >
>> >>
>> >> From a service coupling perspective, I like the approach that ahc is
>> >> optional, and also that Ironic-inspector is optional, because this
>> keeps
>> >> the simple use-case for Ironic, well, simple! That said, this seems
>> more
>> >> like a configuration setting (should inspector do extra things?) than
>> an
>> >> entirely separate service, and separating them might be unnecessarily
>> >> complicated.
>> >
>> > We keep thinking about it as well. After all, right now it's just a
>> > couple of utilities. There are 2 more concerns that initially made me
>> > pull out this code:
>> > 1. ahc-tools currently depends on the library [8], which I wish would be
>> > developed much more openly
>>
>
> > 2. it's cool that inspector is pluggable, but it has its cost: there's a
>> > poor feedback loop from inspector processing plugins to a user - like
>> > with all highly asynchronous code
>> > 3. it's also not possible (at least for now) to request a set of
>> > processing plugins when starting introspection via inspector.
>> >
>> > We solved the latter 2 problems by moving code to scripts. So now
>> > Inspector only puts some data to Swift, and scripts can do everything
>> else.
>> >
>> > So now we've left with
>> > 1. dependency on "hardware" library
>> > 2. not very stable interface, much less stable than one of Inspector
>> >
>> > We still wonder how to solve these 2 without creating one more
>> > repository. Any ideas are welcome :)
>>
>>
> Oh - good point. There's some neat looking functionality in
> enovance/hardware repository, but yea, until this is not a
> single-vendor-controlled repository, I think you made the right call.
>
> How would folks feel about moving that into openstack/ ?
>
>
>> It is a goal of mine to solve issue 1 incrementally over time. Either by
>> improving the library (both in function and in openness), or by slowly
>> moving the implementation. That does not seem impossible to do within
>> the inspector tree.
>>
>
> Or that :) Either way, I agree with the direction -- moving the hardware
> inspection functionality into a common repository is good. If it makes
> sense to have two repositories (one for inspector, one for a hardware utils
> library) that's just fine with me.
>
>
> However, issue 2 is a fact. We currently have scripts, and we want to
>> have a REST API. I do not see a transition between the two that does not
>> involve a large amount of churn.
>>
>
> From a quick read of the code, it looks like ahc-tools merely analyzes
> data already gathered by inspector & enovance/hardware, providing some more
> advanced search/filtering/error-checking capabilities on the client side.
> If that read is correct, then I would like to align that work with my
> interest in developing a query-API for Ironic. It might take some time to
> do that, and so having a repo for client-side scripts is fine for now.
>
> (If that read is incorrect, please explain what I missed)
>
>
>>
>> I am not sure how to solve issue 2 without a separate repository. I do
>> think there is a logical separation of concerns though, so we may not
>> need to completely merge the two in the future. Inspector collects data,
>> and ahc-tools (or whatever it is eventually named) is used to act on the
>> data.
>>
>
> Right -- I see inspector as a separate process from Ironic, but it sounds
> like ahc-tools could eventually be functionality provided by Ironic itself,
> eg, GET /v1/search/ with a body containing some query document. Or
> something....
>
> -Deva
>
>
> >
>> >>
>> >> It sounds like this is the direction you'd like to go, and you took the
>> >> current approach for expediency. If so, I'd like us to discuss a path
>> to
>> >> merge the functionality as it matures, and decide whether a separate
>> >> repository is the right way to go long term.
>> >>
>> >> Thanks much,
>> >> Devananda
>> >>
>> >>
>> >> On Mon, Jun 22, 2015, 05:40 John Trowbridge <trown at redhat.com
>> >> <mailto:trown at redhat.com>> wrote:
>> >>
>> >> This is a proposal to add a new repository governed by the ironic
>> >> inspector subteam. The current repository is named ahc-tools[1],
>> >> however
>> >> there is no attachment to this name. "ironic-inspector-extra"
>> >> would seem
>> >> to fit if this is moved under the Ironic umbrella.
>> >>
>> >> What is AHC?
>> >> ------------
>> >> * AHC as a term comes from the enovance edeploy installation
>> >> method[2].
>> >> * The general concept is that we want to have a very granular
>> >> picture of
>> >> the physical hardware being used in a deployment in order to be
>> >> able to
>> >> match specific hardware to specific roles, as well as the ability
>> to
>> >> find poor performing outliers before we attempt to deploy.
>> >> * For example: As a cloud operator, I want to make sure all logical
>> >> disks have random read IOPs within 15% variance of each other.
>> >> * The huge benefit of this tooling over current inspection is the
>> >> number
>> >> of facts collected (~1000 depending on the hardware), all of which
>> >> can
>> >> be used for matching.
>> >> * Another example: As an end user, I would like to request a bare
>> >> metal
>> >> machine with a specific model GPU.
>> >>
>> >> What is ahc-tools?
>> >> ------------------
>> >> * We first tried to place all of this logic into a plugin in
>> >> inspector[3] (discoverd at the time). [4]
>> >> * This worked fine for just collecting some of the simple facts,
>> >> however
>> >> we now had a coupling between booting a ramdisk, and matching
>> against
>> >> the collected data.
>> >> * ahc-tools started as a way to uncouple these two steps[5].
>> >> * We also added a wrapper around the enovance report tooling[6],
>> >> as it
>> >> already had the ability to generate reports based on the collected
>> >> data,
>> >> but was designed to read in the data from the filesystem.
>> >> * The report tool has two functions.
>> >> * First, it can group the systems by category (NICs, Firmware,
>> >> Processors, etc.).
>> >> * Second, it can use statistical analysis to find performance
>> >> outliers.
>> >>
>> >> Why is ahc-tools useful to Ironic?
>> >> ----------------------------------
>> >> * If we run benchmarks on hardware whenever it is turned back in
>> by a
>> >> tenant, we can easily put nodes into maintenance if the hardware is
>> >> performing below some set threshold. This would allow us to have
>> >> better
>> >> certainty that the end user is getting what we promised them.
>> >> * The advanced matching could also prove very useful. For VMs, I
>> >> think
>> >> the pets vs cattle analogy holds up very well, however many use
>> cases
>> >> for having cloud based bare metal involve access to specific
>> hardware
>> >> capabilities. I think advanced matching could help bridge this gap.
>> >>
>> >> Why not just put this code directly into inspector?
>> >> ---------------------------------------------------
>> >> * Clearly this code is 100% dependent on inspector. However,
>> >> inspector
>> >> is quite stable, and works great without any of this extra tooling.
>> >> * ahc-tools is very immature, and will need many breaking changes
>> >> to get
>> >> to the same stability level of inspector.
>> >>
>> >> Why aren't you following the downstream->stackforge->openstack
>> path?
>> >>
>> --------------------------------------------------------------------
>> >> * This was the initial plan[7], however we were told that under
>> >> the new
>> >> "big tent", that the openstack namespace is no longer meant to
>> >> signify
>> >> maturity of a project.
>> >> * Instead, we were told we should propose the project directly to
>> >> Ironic, or make a new separate project.
>> >>
>> >> What is the plan to make ahc-tools better?
>> >> ------------------------------------------
>> >> * The first major overhaul we would like to do is to put the
>> >> reporting
>> >> and matching functionality behind a REST API.
>> >> * Reporting in particular will require significant work, as the
>> >> current
>> >> wrapper script wraps code that was never designed to be a library
>> >> (Its
>> >> output is just a series of print statements). One option is to
>> >> improve
>> >> the library[8] to be more library like, and the other is to
>> >> reimplement
>> >> the logic itself. Personally, while reimplementing the library is a
>> >> large amount of work, I think it is probably worth the effort.
>> >> * We would also like to add an API endpoint to coordinate
>> distributed
>> >> checks. For instance, if we want to confirm that there is physical
>> >> network connectivity between a set of nodes, or if we would like to
>> >> confirm the bandwidth of those connections.
>> >> * The distributed checks and REST API will hopefully be completed
>> >> in the
>> >> Liberty timeframe.
>> >> * Overhaul of the reporting will likely be an M target, unless
>> >> there is
>> >> interest from new contributors in working on this feature.
>> >> * We are planning a talk for Tokyo on inspector that will also
>> >> include
>> >> details about this project.
>> >>
>> >> Thank you very much for your consideration.
>> >>
>> >> Respectfully,
>> >> John Trowbridge
>> >>
>> >> [1] https://github.com/rdo-management/ahc-tools
>> >> [2] https://github.com/enovance/edeploy/blob/master/docs/AHC.rst
>> >> [3]
>> >>
>> >>
>> https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c
>> >>
>> >> [4] We can have out-of-tree plugins for the inspector, so some of
>> >> this
>> >> code might become a plugin again, but within the new repository
>> tree.
>> >> [5]
>> >>
>> >>
>> https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422
>> >>
>> >> [6]
>> >>
>> >>
>> https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py
>> >>
>> >> [7] https://review.openstack.org/#/c/193392/
>> >> [8] https://github.com/enovance/hardware
>> >>
>> >>
>> >>
>> __________________________________________________________________________
>> >>
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe:
>> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >>
>> >> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>
>> >>
>> >>
>> __________________________________________________________________________
>> >>
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe:
>> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150623/8f7edf9f/attachment.html>
More information about the OpenStack-dev
mailing list