<div dir="ltr">Oh - one more thing. <span style="line-height:1.5;font-size:13.1999998092651px">If ahc-tools depends on data gathered by enovance/hardware, then I'm not sure it makes sense to import one to openstack/ without the other. </span><div><span style="line-height:1.5;font-size:13.1999998092651px"><br></span></div><div><span style="line-height:1.5;font-size:13.1999998092651px">-Deva</span></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jun 22, 2015 at 5:08 PM Devananda van der Veen <<a href="mailto:devananda.vdv@gmail.com">devananda.vdv@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I'm <br><div class="gmail_quote"></div></div><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Mon, Jun 22, 2015 at 8:19 AM John Trowbridge <<a href="mailto:trown@redhat.com" target="_blank">trown@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
On 06/22/2015 10:40 AM, Dmitry Tantsur wrote:<br>
> On 06/22/2015 04:19 PM, Devananda van der Veen wrote:<br>
>> Hi John,<br>
>><br>
>> Thanks for the excellent summary! I found it very helpful to get caught<br>
>> up. I'd like to make sure I understand the direction ahc is going. A<br>
>> couple questions...<br>
<br>
Thanks for your interest.<br>
<br>
><br>
> Let me add my $0.5 :)<br>
><br>
>><br>
>> I see that ahc is storing its information in swift. That's clever, but<br>
>> if Ironic provided a blob store for each node, would that be better?<br>
<br>
If the blob is large enough, this would be better. Originally we stored<br>
the data in the extra column of the Ironic db, but that proved disastrous:<br>
<br>
<a href="https://bugs.launchpad.net/ironic-inspector/+bug/1461252" rel="noreferrer" target="_blank">https://bugs.launchpad.net/ironic-inspector/+bug/1461252</a><br></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
>><br>
>> We discussed adding a search API to Ironic at the Vancouver summit,<br>
>> though no work has been done on that yet, afaik. If ahc is going to grow<br>
>> a REST API for searching for nodes based on specific criteria that it<br>
>> discovered, could/should we combine these within Ironic's API?<br>
><br>
> I think John meant having API to replace scripts, so I guess search<br>
> won't help. When we're talking about advanced matching, we're talking<br>
> about the following:<br>
> 1. We have a ramdisk tool (based on [8]) to get some insane of facts<br>
> from withing the ramdisk (say, 1000 of them)<br>
> 2. We have an Inspector plugin to put them all in Swift (or Ironic blob<br>
> storage as above)<br>
> 3. We have config files (aka rules) written in special JSON-alike DSL to<br>
> do matching (one of the weak points is that these are files - I'd like<br>
> API endpoint to accept these rules instead).<br>
> 4. We have a script to run this DSL and get some output (match/not match<br>
> + some matched variables - similar to what regexps do).<br>
> As I understood it John want the latter to become an API endpoint,<br>
> accepting rules (and maybe node UUIDs) and outputting some result.<br>
><br>
> Not sure about benchmarking here, but again, it's probably an API<br>
> endpoint that accepts some minimal expectations, and puts failed nodes<br>
> to maintenance mode, if they fail to comply (again, that's how I<br>
> understood it).<br>
><br>
> It's not hard to make these API endpoints part of Inspector, but it's<br>
> somewhat undesirable to have them optional...<br>
><br>
>><br>
>> From a service coupling perspective, I like the approach that ahc is<br>
>> optional, and also that Ironic-inspector is optional, because this keeps<br>
>> the simple use-case for Ironic, well, simple! That said, this seems more<br>
>> like a configuration setting (should inspector do extra things?) than an<br>
>> entirely separate service, and separating them might be unnecessarily<br>
>> complicated.<br>
><br>
> We keep thinking about it as well. After all, right now it's just a<br>
> couple of utilities. There are 2 more concerns that initially made me<br>
> pull out this code:<br>
> 1. ahc-tools currently depends on the library [8], which I wish would be<br>
> developed much more openly<br></blockquote><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
> 2. it's cool that inspector is pluggable, but it has its cost: there's a<br>
> poor feedback loop from inspector processing plugins to a user - like<br>
> with all highly asynchronous code<br>
> 3. it's also not possible (at least for now) to request a set of<br>
> processing plugins when starting introspection via inspector.<br>
><br>
> We solved the latter 2 problems by moving code to scripts. So now<br>
> Inspector only puts some data to Swift, and scripts can do everything else.<br>
><br>
> So now we've left with<br>
> 1. dependency on "hardware" library<br>
> 2. not very stable interface, much less stable than one of Inspector<br>
><br>
> We still wonder how to solve these 2 without creating one more<br>
> repository. Any ideas are welcome :)<br>
<br></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div><div>Oh - good point. There's some neat looking functionality in enovance/hardware repository, but yea, until this is not a single-vendor-controlled repository, I think you made the right call.</div><div><br></div><div>How would folks feel about moving that into openstack/ ? </div></div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It is a goal of mine to solve issue 1 incrementally over time. Either by<br>
improving the library (both in function and in openness), or by slowly<br>
moving the implementation. That does not seem impossible to do within<br>
the inspector tree.<br></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>Or that :) Either way, I agree with the direction -- moving the hardware inspection functionality into a common repository is good. If it makes sense to have two repositories (one for inspector, one for a hardware utils library) that's just fine with me.</div></div></div><div dir="ltr"><div class="gmail_quote"><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">However, issue 2 is a fact. We currently have scripts, and we want to<br>
have a REST API. I do not see a transition between the two that does not<br>
involve a large amount of churn.<br></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>From a quick read of the code, it looks like ahc-tools merely analyzes data already gathered by inspector & enovance/hardware, providing some more advanced search/filtering/error-checking capabilities on the client side. If that read is correct, then I would like to align that work with my interest in developing a query-API for Ironic. It might take some time to do that, and so having a repo for client-side scripts is fine for now.</div><div><br></div><div>(If that read is incorrect, please explain what I missed)</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
I am not sure how to solve issue 2 without a separate repository. I do<br>
think there is a logical separation of concerns though, so we may not<br>
need to completely merge the two in the future. Inspector collects data,<br>
and ahc-tools (or whatever it is eventually named) is used to act on the<br>
data.<br></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>Right -- I see inspector as a separate process from Ironic, but it sounds like ahc-tools could eventually be functionality provided by Ironic itself, eg, GET /v1/search/ with a body containing some query document. Or something....</div><div><br></div><div>-Deva</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
><br>
>><br>
>> It sounds like this is the direction you'd like to go, and you took the<br>
>> current approach for expediency. If so, I'd like us to discuss a path to<br>
>> merge the functionality as it matures, and decide whether a separate<br>
>> repository is the right way to go long term.<br>
>><br>
>> Thanks much,<br>
>> Devananda<br>
>><br>
>><br>
>> On Mon, Jun 22, 2015, 05:40 John Trowbridge <<a href="mailto:trown@redhat.com" target="_blank">trown@redhat.com</a><br>
>> <mailto:<a href="mailto:trown@redhat.com" target="_blank">trown@redhat.com</a>>> wrote:<br>
>><br>
>> This is a proposal to add a new repository governed by the ironic<br>
>> inspector subteam. The current repository is named ahc-tools[1],<br>
>> however<br>
>> there is no attachment to this name. "ironic-inspector-extra"<br>
>> would seem<br>
>> to fit if this is moved under the Ironic umbrella.<br>
>><br>
>> What is AHC?<br>
>> ------------<br>
>> * AHC as a term comes from the enovance edeploy installation<br>
>> method[2].<br>
>> * The general concept is that we want to have a very granular<br>
>> picture of<br>
>> the physical hardware being used in a deployment in order to be<br>
>> able to<br>
>> match specific hardware to specific roles, as well as the ability to<br>
>> find poor performing outliers before we attempt to deploy.<br>
>> * For example: As a cloud operator, I want to make sure all logical<br>
>> disks have random read IOPs within 15% variance of each other.<br>
>> * The huge benefit of this tooling over current inspection is the<br>
>> number<br>
>> of facts collected (~1000 depending on the hardware), all of which<br>
>> can<br>
>> be used for matching.<br>
>> * Another example: As an end user, I would like to request a bare<br>
>> metal<br>
>> machine with a specific model GPU.<br>
>><br>
>> What is ahc-tools?<br>
>> ------------------<br>
>> * We first tried to place all of this logic into a plugin in<br>
>> inspector[3] (discoverd at the time). [4]<br>
>> * This worked fine for just collecting some of the simple facts,<br>
>> however<br>
>> we now had a coupling between booting a ramdisk, and matching against<br>
>> the collected data.<br>
>> * ahc-tools started as a way to uncouple these two steps[5].<br>
>> * We also added a wrapper around the enovance report tooling[6],<br>
>> as it<br>
>> already had the ability to generate reports based on the collected<br>
>> data,<br>
>> but was designed to read in the data from the filesystem.<br>
>> * The report tool has two functions.<br>
>> * First, it can group the systems by category (NICs, Firmware,<br>
>> Processors, etc.).<br>
>> * Second, it can use statistical analysis to find performance<br>
>> outliers.<br>
>><br>
>> Why is ahc-tools useful to Ironic?<br>
>> ----------------------------------<br>
>> * If we run benchmarks on hardware whenever it is turned back in by a<br>
>> tenant, we can easily put nodes into maintenance if the hardware is<br>
>> performing below some set threshold. This would allow us to have<br>
>> better<br>
>> certainty that the end user is getting what we promised them.<br>
>> * The advanced matching could also prove very useful. For VMs, I<br>
>> think<br>
>> the pets vs cattle analogy holds up very well, however many use cases<br>
>> for having cloud based bare metal involve access to specific hardware<br>
>> capabilities. I think advanced matching could help bridge this gap.<br>
>><br>
>> Why not just put this code directly into inspector?<br>
>> ---------------------------------------------------<br>
>> * Clearly this code is 100% dependent on inspector. However,<br>
>> inspector<br>
>> is quite stable, and works great without any of this extra tooling.<br>
>> * ahc-tools is very immature, and will need many breaking changes<br>
>> to get<br>
>> to the same stability level of inspector.<br>
>><br>
>> Why aren't you following the downstream->stackforge->openstack path?<br>
>> --------------------------------------------------------------------<br>
>> * This was the initial plan[7], however we were told that under<br>
>> the new<br>
>> "big tent", that the openstack namespace is no longer meant to<br>
>> signify<br>
>> maturity of a project.<br>
>> * Instead, we were told we should propose the project directly to<br>
>> Ironic, or make a new separate project.<br>
>><br>
>> What is the plan to make ahc-tools better?<br>
>> ------------------------------------------<br>
>> * The first major overhaul we would like to do is to put the<br>
>> reporting<br>
>> and matching functionality behind a REST API.<br>
>> * Reporting in particular will require significant work, as the<br>
>> current<br>
>> wrapper script wraps code that was never designed to be a library<br>
>> (Its<br>
>> output is just a series of print statements). One option is to<br>
>> improve<br>
>> the library[8] to be more library like, and the other is to<br>
>> reimplement<br>
>> the logic itself. Personally, while reimplementing the library is a<br>
>> large amount of work, I think it is probably worth the effort.<br>
>> * We would also like to add an API endpoint to coordinate distributed<br>
>> checks. For instance, if we want to confirm that there is physical<br>
>> network connectivity between a set of nodes, or if we would like to<br>
>> confirm the bandwidth of those connections.<br>
>> * The distributed checks and REST API will hopefully be completed<br>
>> in the<br>
>> Liberty timeframe.<br>
>> * Overhaul of the reporting will likely be an M target, unless<br>
>> there is<br>
>> interest from new contributors in working on this feature.<br>
>> * We are planning a talk for Tokyo on inspector that will also<br>
>> include<br>
>> details about this project.<br>
>><br>
>> Thank you very much for your consideration.<br>
>><br>
>> Respectfully,<br>
>> John Trowbridge<br>
>><br>
>> [1] <a href="https://github.com/rdo-management/ahc-tools" rel="noreferrer" target="_blank">https://github.com/rdo-management/ahc-tools</a><br>
>> [2] <a href="https://github.com/enovance/edeploy/blob/master/docs/AHC.rst" rel="noreferrer" target="_blank">https://github.com/enovance/edeploy/blob/master/docs/AHC.rst</a><br>
>> [3]<br>
>><br>
>> <a href="https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c" rel="noreferrer" target="_blank">https://github.com/openstack/ironic-inspector/commit/22a0e24efbef149377ea1e020f2d81968c10b58c</a><br>
>><br>
>> [4] We can have out-of-tree plugins for the inspector, so some of<br>
>> this<br>
>> code might become a plugin again, but within the new repository tree.<br>
>> [5]<br>
>><br>
>> <a href="https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422" rel="noreferrer" target="_blank">https://github.com/openstack/ironic-inspector/commit/eaad7e09b99ab498e080e6e0ab71e69d00275422</a><br>
>><br>
>> [6]<br>
>><br>
>> <a href="https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py" rel="noreferrer" target="_blank">https://github.com/rdo-management/ahc-tools/blob/master/ahc_tools/report.py</a><br>
>><br>
>> [7] <a href="https://review.openstack.org/#/c/193392/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/193392/</a><br>
>> [8] <a href="https://github.com/enovance/hardware" rel="noreferrer" target="_blank">https://github.com/enovance/hardware</a><br>
>><br>
>><br>
>> __________________________________________________________________________<br>
>><br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe:<br>
>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>><br>
>> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
>><br>
>><br>
>> __________________________________________________________________________<br>
>><br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe:<br>
>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div></blockquote></div>