[openstack-dev] [Ironic] Proposal to add a new repository
Dmitry Tantsur
dtantsur at redhat.com
Tue Jun 23 07:49:02 UTC 2015
On 06/23/2015 02:11 AM, Devananda van der Veen wrote:
> Oh - one more thing. If ahc-tools depends on data gathered by
> enovance/hardware, then I'm not sure it makes sense to import one to
> openstack/ without the other.
Maybe. We'll chat with our folks about it.
>
> -Deva
>
> On Mon, Jun 22, 2015 at 5:08 PM Devananda van der Veen
> <devananda.vdv at gmail.com <mailto:devananda.vdv at gmail.com>> wrote:
>
> I'm
> On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge <trown at redhat.com
> <mailto:trown at redhat.com>> wrote:
>
>
>
> On 06/22/2015 10:40 AM, Dmitry Tantsur wrote:
> > On 06/22/2015 04:19 PM, Devananda van der Veen wrote:
> >> Hi John,
> >>
> >> Thanks for the excellent summary! I found it very helpful to
> get caught
> >> up. I'd like to make sure I understand the direction ahc is
> going. A
> >> couple questions...
>
> Thanks for your interest.
>
> >
> > Let me add my $0.5 :)
> >
> >>
> >> I see that ahc is storing its information in swift. That's
> clever, but
> >> if Ironic provided a blob store for each node, would that be
> better?
>
> If the blob is large enough, this would be better. Originally we
> stored
> the data in the extra column of the Ironic db, but that proved
> disastrous:
>
> https://bugs.launchpad.net/ironic-inspector/+bug/1461252
>
> >>
> >> We discussed adding a search API to Ironic at the Vancouver
> summit,
> >> though no work has been done on that yet, afaik. If ahc is
> going to grow
> >> a REST API for searching for nodes based on specific
> criteria that it
> >> discovered, could/should we combine these within Ironic's API?
> >
> > I think John meant having API to replace scripts, so I guess
> search
> > won't help. When we're talking about advanced matching, we're
> talking
> > about the following:
> > 1. We have a ramdisk tool (based on [8]) to get some insane
> of facts
> > from withing the ramdisk (say, 1000 of them)
> > 2. We have an Inspector plugin to put them all in Swift (or
> Ironic blob
> > storage as above)
> > 3. We have config files (aka rules) written in special
> JSON-alike DSL to
> > do matching (one of the weak points is that these are files -
> I'd like
> > API endpoint to accept these rules instead).
> > 4. We have a script to run this DSL and get some output
> (match/not match
> > + some matched variables - similar to what regexps do).
> > As I understood it John want the latter to become an API
> endpoint,
> > accepting rules (and maybe node UUIDs) and outputting some
> result.
> >
> > Not sure about benchmarking here, but again, it's probably an API
> > endpoint that accepts some minimal expectations, and puts
> failed nodes
> > to maintenance mode, if they fail to comply (again, that's how I
> > understood it).
> >
> > It's not hard to make these API endpoints part of Inspector,
> but it's
> > somewhat undesirable to have them optional...
> >
> >>
> >> From a service coupling perspective, I like the approach
> that ahc is
> >> optional, and also that Ironic-inspector is optional,
> because this keeps
> >> the simple use-case for Ironic, well, simple! That said,
> this seems more
> >> like a configuration setting (should inspector do extra
> things?) than an
> >> entirely separate service, and separating them might be
> unnecessarily
> >> complicated.
> >
> > We keep thinking about it as well. After all, right now it's
> just a
> > couple of utilities. There are 2 more concerns that initially
> made me
> > pull out this code:
> > 1. ahc-tools currently depends on the library [8], which I
> wish would be
> > developed much more openly
>
>
> > 2. it's cool that inspector is pluggable, but it has its
> cost: there's a
> > poor feedback loop from inspector processing plugins to a
> user - like
> > with all highly asynchronous code
> > 3. it's also not possible (at least for now) to request a set of
> > processing plugins when starting introspection via inspector.
> >
> > We solved the latter 2 problems by moving code to scripts. So now
> > Inspector only puts some data to Swift, and scripts can do
> everything else.
> >
> > So now we've left with
> > 1. dependency on "hardware" library
> > 2. not very stable interface, much less stable than one of
> Inspector
> >
> > We still wonder how to solve these 2 without creating one more
> > repository. Any ideas are welcome :)
>
>
> Oh - good point. There's some neat looking functionality in
> enovance/hardware repository, but yea, until this is not a
> single-vendor-controlled repository, I think you made the right call.
>
> How would folks feel about moving that into openstack/ ?
>
> It is a goal of mine to solve issue 1 incrementally over time.
> Either by
> improving the library (both in function and in openness), or by
> slowly
> moving the implementation. That does not seem impossible to do
> within
> the inspector tree.
>
>
> Or that :) Either way, I agree with the direction -- moving the
> hardware inspection functionality into a common repository is good.
> If it makes sense to have two repositories (one for inspector, one
> for a hardware utils library) that's just fine with me.
>
>
> However, issue 2 is a fact. We currently have scripts, and we
> want to
> have a REST API. I do not see a transition between the two that
> does not
> involve a large amount of churn.
>
>
> From a quick read of the code, it looks like ahc-tools merely
> analyzes data already gathered by inspector & enovance/hardware,
> providing some more advanced search/filtering/error-checking
> capabilities on the client side. If that read is correct, then I
> would like to align that work with my interest in developing a
> query-API for Ironic. It might take some time to do that, and so
> having a repo for client-side scripts is fine for now.
>
> (If that read is incorrect, please explain what I missed)
>
>
> I am not sure how to solve issue 2 without a separate
> repository. I do
> think there is a logical separation of concerns though, so we
> may not
> need to completely merge the two in the future. Inspector
> collects data,
> and ahc-tools (or whatever it is eventually named) is used to
> act on the
> data.
>
>
> Right -- I see inspector as a separate process from Ironic, but it
> sounds like ahc-tools could eventually be functionality provided by
> Ironic itself, eg, GET /v1/search/ with a body containing some
> query document. Or something....
>
> -Deva
>
> >
> >>
> >> It sounds like this is the direction you'd like to go, and
> you took the
> >> current approach for expediency. If so, I'd like us to
> discuss a path to
> >> merge the functionality as it matures, and decide whether a
> separate
> >> repository is the right way to go long term.
> >>
> >> Thanks much,
> >> Devananda
> >>
> >>
> >> On Mon, Jun 22, 2015, 05:40 John Trowbridge
> <trown at redhat.com <mailto:trown at redhat.com>
> >> <mailto:trown at redhat.com <mailto:trown at redhat.com>>> wrote:
> >>
> >> This is a proposal to add a new repository governed by
> the ironic
> >> inspector subteam. The current repository is named
> ahc-tools[1],
> >> however
> >> there is no attachment to this name.
> "ironic-inspector-extra"
> >> would seem
> >> to fit if this is moved under the Ironic umbrella.
> >>
> >> What is AHC?
> >> ------------
> >> * AHC as a term comes from the enovance edeploy installation
> >> method[2].
> >> * The general concept is that we want to have a very
> granular
> >> picture of
> >> the physical hardware being used in a deployment in
> order to be
> >> able to
> >> match specific hardware to specific roles, as well as
> the ability to
> >> find poor performing outliers before we attempt to deploy.
> >> * For example: As a cloud operator, I want to make sure
> all logical
> >> disks have random read IOPs within 15% variance of each
> other.
> >> * The huge benefit of this tooling over current
> inspection is the
> >> number
> >> of facts collected (~1000 depending on the hardware),
> all of which
> >> can
> >> be used for matching.
> >> * Another example: As an end user, I would like to
> request a bare
> >> metal
> >> machine with a specific model GPU.
> >>
> >> What is ahc-tools?
> >> ------------------
> >> * We first tried to place all of this logic into a plugin in
> >> inspector[3] (discoverd at the time). [4]
> >> * This worked fine for just collecting some of the
> simple facts,
> >> however
> >> we now had a coupling between booting a ramdisk, and
> matching against
> >> the collected data.
> >> * ahc-tools started as a way to uncouple these two steps[5].
> >> * We also added a wrapper around the enovance report
> tooling[6],
> >> as it
> >> already had the ability to generate reports based on the
> collected
> >> data,
> >> but was designed to read in the data from the filesystem.
> >> * The report tool has two functions.
> >> * First, it can group the systems by category (NICs,
> Firmware,
> >> Processors, etc.).
> >> * Second, it can use statistical analysis to find
> performance
> >> outliers.
> >>
> >> Why is ahc-tools useful to Ironic?
> >> ----------------------------------
> >> * If we run benchmarks on hardware whenever it is turned
> back in by a
> >> tenant, we can easily put nodes into maintenance if the
> hardware is
> >> performing below some set threshold. This would allow us
> to have
> >> better
> >> certainty that the end user is getting what we promised
> them.
> >> * The advanced matching could also prove very useful.
> For VMs, I
> >> think
> >> the pets vs cattle analogy holds up very well, however
> many use cases
> >> for having cloud based bare metal involve access to
> specific hardware
> >> capabilities. I think advanced matching could help
> bridge this gap.
> >>
> >> Why not just put this code directly into inspector?
> >> ---------------------------------------------------
> >> * Clearly this code is 100% dependent on inspector. However,
> >> inspector
> >> is quite stable, and works great without any of this
> extra tooling.
> >> * ahc-tools is very immature, and will need many
> breaking changes
> >> to get
> >> to the same stability level of inspector.
> >>
> >> Why aren't you following the
> downstream->stackforge->openstack path?
> >>
> --------------------------------------------------------------------
> >> * This was the initial plan[7], however we were told
> that under
> >> the new
> >> "big tent", that the openstack namespace is no longer
> meant to
> >> signify
> >> maturity of a project.
> >> * Instead, we were told we should propose the project
> directly to
> >> Ironic, or make a new separate project.
> >>
> >> What is the plan to make ahc-tools better?
> >> ------------------------------------------
> >> * The first major overhaul we would like to do is to put the
> >> reporting
> >> and matching functionality behind a REST API.
> >> * Reporting in particular will require significant work,
> as the
> >> current
> >> wrapper script wraps code that was never designed to be
> a library
> >> (Its
> >> output is just a series of print statements). One option
> is to
> >> improve
> >> the library[8] to be more library like, and the other is to
> >> reimplement
> >> the logic itself. Personally, while reimplementing the
> library is a
> >> large amount of work, I think it is probably worth the
> effort.
> >> * We would also like to add an API endpoint to
> coordinate distributed
> >> checks. For instance, if we want to confirm that there
> is physical
> >> network connectivity between a set of nodes, or if we
> would like to
> >> confirm the bandwidth of those connections.
> >> * The distributed checks and REST API will hopefully be
> completed
> >> in the
> >> Liberty timeframe.
> >> * Overhaul of the reporting will likely be an M target,
> unless
> >> there is
> >> interest from new contributors in working on this feature.
> >> * We are planning a talk for Tokyo on inspector that
> will also
> >> include
> >> details about this project.
> >>
> >> Thank you very much for your consideration.
> >>
> >> Respectfully,
> >> John Trowbridge
> >>
> >> [1] https://github.com/rdo-management/ahc-tools
> >> [2]
> https://github.com/enovance/edeploy/blob/master/docs/AHC.rst
> >> [3]
> >>
> >>
> https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c
> >>
> >> [4] We can have out-of-tree plugins for the inspector,
> so some of
> >> this
> >> code might become a plugin again, but within the new
> repository tree.
> >> [5]
> >>
> >>
> https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422
> >>
> >> [6]
> >>
> >>
> https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py
> >>
> >> [7] https://review.openstack.org/#/c/193392/
> >> [8] https://github.com/enovance/hardware
> >>
> >>
> >>
> __________________________________________________________________________
> >>
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >>
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >>
> >>
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>
> >>
> __________________________________________________________________________
> >>
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >>
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list