[openstack-dev] [Ironic] Proposal to add a new repository
Devananda van der Veen
devananda.vdv at gmail.com
Tue Jun 23 00:08:27 UTC 2015
I'm
On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge <trown at redhat.com> wrote:
>
>
> On 06/22/2015 10:40 AM, Dmitry Tantsur wrote:
> > On 06/22/2015 04:19 PM, Devananda van der Veen wrote:
> >> Hi John,
> >>
> >> Thanks for the excellent summary! I found it very helpful to get caught
> >> up. I'd like to make sure I understand the direction ahc is going. A
> >> couple questions...
>
> Thanks for your interest.
>
> >
> > Let me add my $0.5 :)
> >
> >>
> >> I see that ahc is storing its information in swift. That's clever, but
> >> if Ironic provided a blob store for each node, would that be better?
>
> If the blob is large enough, this would be better. Originally we stored
> the data in the extra column of the Ironic db, but that proved disastrous:
>
> https://bugs.launchpad.net/ironic-inspector/+bug/1461252
>
> >>
> >> We discussed adding a search API to Ironic at the Vancouver summit,
> >> though no work has been done on that yet, afaik. If ahc is going to grow
> >> a REST API for searching for nodes based on specific criteria that it
> >> discovered, could/should we combine these within Ironic's API?
> >
> > I think John meant having API to replace scripts, so I guess search
> > won't help. When we're talking about advanced matching, we're talking
> > about the following:
> > 1. We have a ramdisk tool (based on [8]) to get some insane of facts
> > from withing the ramdisk (say, 1000 of them)
> > 2. We have an Inspector plugin to put them all in Swift (or Ironic blob
> > storage as above)
> > 3. We have config files (aka rules) written in special JSON-alike DSL to
> > do matching (one of the weak points is that these are files - I'd like
> > API endpoint to accept these rules instead).
> > 4. We have a script to run this DSL and get some output (match/not match
> > + some matched variables - similar to what regexps do).
> > As I understood it John want the latter to become an API endpoint,
> > accepting rules (and maybe node UUIDs) and outputting some result.
> >
> > Not sure about benchmarking here, but again, it's probably an API
> > endpoint that accepts some minimal expectations, and puts failed nodes
> > to maintenance mode, if they fail to comply (again, that's how I
> > understood it).
> >
> > It's not hard to make these API endpoints part of Inspector, but it's
> > somewhat undesirable to have them optional...
> >
> >>
> >> From a service coupling perspective, I like the approach that ahc is
> >> optional, and also that Ironic-inspector is optional, because this keeps
> >> the simple use-case for Ironic, well, simple! That said, this seems more
> >> like a configuration setting (should inspector do extra things?) than an
> >> entirely separate service, and separating them might be unnecessarily
> >> complicated.
> >
> > We keep thinking about it as well. After all, right now it's just a
> > couple of utilities. There are 2 more concerns that initially made me
> > pull out this code:
> > 1. ahc-tools currently depends on the library [8], which I wish would be
> > developed much more openly
>
> 2. it's cool that inspector is pluggable, but it has its cost: there's a
> > poor feedback loop from inspector processing plugins to a user - like
> > with all highly asynchronous code
> > 3. it's also not possible (at least for now) to request a set of
> > processing plugins when starting introspection via inspector.
> >
> > We solved the latter 2 problems by moving code to scripts. So now
> > Inspector only puts some data to Swift, and scripts can do everything
> else.
> >
> > So now we've left with
> > 1. dependency on "hardware" library
> > 2. not very stable interface, much less stable than one of Inspector
> >
> > We still wonder how to solve these 2 without creating one more
> > repository. Any ideas are welcome :)
>
>
Oh - good point. There's some neat looking functionality in
enovance/hardware repository, but yea, until this is not a
single-vendor-controlled repository, I think you made the right call.
How would folks feel about moving that into openstack/ ?
> It is a goal of mine to solve issue 1 incrementally over time. Either by
> improving the library (both in function and in openness), or by slowly
> moving the implementation. That does not seem impossible to do within
> the inspector tree.
>
Or that :) Either way, I agree with the direction -- moving the hardware
inspection functionality into a common repository is good. If it makes
sense to have two repositories (one for inspector, one for a hardware utils
library) that's just fine with me.
However, issue 2 is a fact. We currently have scripts, and we want to
> have a REST API. I do not see a transition between the two that does not
> involve a large amount of churn.
>
>From a quick read of the code, it looks like ahc-tools merely analyzes data
already gathered by inspector & enovance/hardware, providing some more
advanced search/filtering/error-checking capabilities on the client side.
If that read is correct, then I would like to align that work with my
interest in developing a query-API for Ironic. It might take some time to
do that, and so having a repo for client-side scripts is fine for now.
(If that read is incorrect, please explain what I missed)
>
> I am not sure how to solve issue 2 without a separate repository. I do
> think there is a logical separation of concerns though, so we may not
> need to completely merge the two in the future. Inspector collects data,
> and ahc-tools (or whatever it is eventually named) is used to act on the
> data.
>
Right -- I see inspector as a separate process from Ironic, but it sounds
like ahc-tools could eventually be functionality provided by Ironic itself,
eg, GET /v1/search/ with a body containing some query document. Or
something....
-Deva
>
> >>
> >> It sounds like this is the direction you'd like to go, and you took the
> >> current approach for expediency. If so, I'd like us to discuss a path to
> >> merge the functionality as it matures, and decide whether a separate
> >> repository is the right way to go long term.
> >>
> >> Thanks much,
> >> Devananda
> >>
> >>
> >> On Mon, Jun 22, 2015, 05:40 John Trowbridge <trown at redhat.com
> >> <mailto:trown at redhat.com>> wrote:
> >>
> >> This is a proposal to add a new repository governed by the ironic
> >> inspector subteam. The current repository is named ahc-tools[1],
> >> however
> >> there is no attachment to this name. "ironic-inspector-extra"
> >> would seem
> >> to fit if this is moved under the Ironic umbrella.
> >>
> >> What is AHC?
> >> ------------
> >> * AHC as a term comes from the enovance edeploy installation
> >> method[2].
> >> * The general concept is that we want to have a very granular
> >> picture of
> >> the physical hardware being used in a deployment in order to be
> >> able to
> >> match specific hardware to specific roles, as well as the ability to
> >> find poor performing outliers before we attempt to deploy.
> >> * For example: As a cloud operator, I want to make sure all logical
> >> disks have random read IOPs within 15% variance of each other.
> >> * The huge benefit of this tooling over current inspection is the
> >> number
> >> of facts collected (~1000 depending on the hardware), all of which
> >> can
> >> be used for matching.
> >> * Another example: As an end user, I would like to request a bare
> >> metal
> >> machine with a specific model GPU.
> >>
> >> What is ahc-tools?
> >> ------------------
> >> * We first tried to place all of this logic into a plugin in
> >> inspector[3] (discoverd at the time). [4]
> >> * This worked fine for just collecting some of the simple facts,
> >> however
> >> we now had a coupling between booting a ramdisk, and matching
> against
> >> the collected data.
> >> * ahc-tools started as a way to uncouple these two steps[5].
> >> * We also added a wrapper around the enovance report tooling[6],
> >> as it
> >> already had the ability to generate reports based on the collected
> >> data,
> >> but was designed to read in the data from the filesystem.
> >> * The report tool has two functions.
> >> * First, it can group the systems by category (NICs, Firmware,
> >> Processors, etc.).
> >> * Second, it can use statistical analysis to find performance
> >> outliers.
> >>
> >> Why is ahc-tools useful to Ironic?
> >> ----------------------------------
> >> * If we run benchmarks on hardware whenever it is turned back in by
> a
> >> tenant, we can easily put nodes into maintenance if the hardware is
> >> performing below some set threshold. This would allow us to have
> >> better
> >> certainty that the end user is getting what we promised them.
> >> * The advanced matching could also prove very useful. For VMs, I
> >> think
> >> the pets vs cattle analogy holds up very well, however many use
> cases
> >> for having cloud based bare metal involve access to specific
> hardware
> >> capabilities. I think advanced matching could help bridge this gap.
> >>
> >> Why not just put this code directly into inspector?
> >> ---------------------------------------------------
> >> * Clearly this code is 100% dependent on inspector. However,
> >> inspector
> >> is quite stable, and works great without any of this extra tooling.
> >> * ahc-tools is very immature, and will need many breaking changes
> >> to get
> >> to the same stability level of inspector.
> >>
> >> Why aren't you following the downstream->stackforge->openstack path?
> >> --------------------------------------------------------------------
> >> * This was the initial plan[7], however we were told that under
> >> the new
> >> "big tent", that the openstack namespace is no longer meant to
> >> signify
> >> maturity of a project.
> >> * Instead, we were told we should propose the project directly to
> >> Ironic, or make a new separate project.
> >>
> >> What is the plan to make ahc-tools better?
> >> ------------------------------------------
> >> * The first major overhaul we would like to do is to put the
> >> reporting
> >> and matching functionality behind a REST API.
> >> * Reporting in particular will require significant work, as the
> >> current
> >> wrapper script wraps code that was never designed to be a library
> >> (Its
> >> output is just a series of print statements). One option is to
> >> improve
> >> the library[8] to be more library like, and the other is to
> >> reimplement
> >> the logic itself. Personally, while reimplementing the library is a
> >> large amount of work, I think it is probably worth the effort.
> >> * We would also like to add an API endpoint to coordinate
> distributed
> >> checks. For instance, if we want to confirm that there is physical
> >> network connectivity between a set of nodes, or if we would like to
> >> confirm the bandwidth of those connections.
> >> * The distributed checks and REST API will hopefully be completed
> >> in the
> >> Liberty timeframe.
> >> * Overhaul of the reporting will likely be an M target, unless
> >> there is
> >> interest from new contributors in working on this feature.
> >> * We are planning a talk for Tokyo on inspector that will also
> >> include
> >> details about this project.
> >>
> >> Thank you very much for your consideration.
> >>
> >> Respectfully,
> >> John Trowbridge
> >>
> >> [1] https://github.com/rdo-management/ahc-tools
> >> [2] https://github.com/enovance/edeploy/blob/master/docs/AHC.rst
> >> [3]
> >>
> >>
> https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c
> >>
> >> [4] We can have out-of-tree plugins for the inspector, so some of
> >> this
> >> code might become a plugin again, but within the new repository
> tree.
> >> [5]
> >>
> >>
> https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422
> >>
> >> [6]
> >>
> >>
> https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py
> >>
> >> [7] https://review.openstack.org/#/c/193392/
> >> [8] https://github.com/enovance/hardware
> >>
> >>
> >>
> __________________________________________________________________________
> >>
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>
> >> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>
> >>
> __________________________________________________________________________
> >>
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150623/dea35924/attachment.html>
More information about the OpenStack-dev
mailing list