<div dir="ltr">Very familiar list Tim, and we end up working around a lot of them with horrible hardware specific code. Our bugbears also include :<div><br></div><div>Required configuration only being available via a web interface - eg. setting hostname of the BMC on Supermicro hardware</div><div>IPMI hanging and requiring complete removal and reload of the kernel modules to enable resetting</div><div>Undocumented functions requiring raw IPMI commands - again on Supermicro there is some black magic to set dedicated ports, check power supply status etc. </div><div>Web interfaces requiring Java, and totally broken on mainstream browsers - HP ILO's in particular, which are almost impossible to use with a Mac. </div><div>Firmware and BIOS'es which don't allow command line updating from inside a running OS</div><div><br></div><div>We're used to being able to flash BIOS images and CMOS settings by writing directly to the memory addresses, but more and more modern hardware won't let you do this anymore :(</div><div><br></div><div>We're hoping Redfish will solve some of the configuration related issues, although obviously it won't make any difference to flaky BMC implementations and proprietary tooling to update firmware. </div></div><div class="gmail_extra"><br><div class="gmail_quote">On 12 May 2016 at 06:25, Tim Bell <span dir="ltr"><<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
<br>
On 12/05/16 06:22, "Stig Telfer" <<a href="mailto:stig.openstack@telfer.org">stig.openstack@telfer.org</a>> wrote:<br>
<br>
>Hi All -<br>
><br>
>Jim Rollenhagen from the Ironic project has just posted a great summit report of Ironic team activities on the openstack-devs mailing list[1], which included this item which will be of interest to the Scientific WG members who are looking to work on bare metal activities this cycle:<br>
><br>
>> # Making ops less worse<br>
>><br>
>> [Etherpad](<a href="https://etherpad.openstack.org/p/ironic-newton-summit-ops" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/ironic-newton-summit-ops</a>)<br>
>><br>
>> We discussed some common failure cases that operators see, and how we<br>
>> can solve them in code.<br>
>><br>
>> We discussed flaky BMCs, which end with the node in maintenance mode,<br>
>> and if Ironic can get them out of that mode automagically. We identified<br>
>> the need to distinguish between maintenance set by ironic and set by<br>
>> operators, and do things like attempt to connect to the BMC on a power<br>
>> state request, and turn off maintenance mode if successful. JayF is<br>
>> going to write a spec for this differentiation.<br>
>><br>
>> Folks also expressed the desire to be able to reset the BMC via APIs. We<br>
>> have a BMC reset function in the vendor interface for the ipmitool<br>
>> driver; dtantsur volunteered to write a spec to promote that method to<br>
>> an official ManagementInterface method.<br>
>><br>
>> We also talked for a while about stuck states. This has been mostly<br>
>> solved in code, but is still a problem for some deployers. We decided<br>
>> that we should not have a "reset-state" API like nova does, but rather a<br>
>> command line tool to handle this. lintan has volunteered to write a<br>
>> proposal for this; I have also posted some [straw man<br>
>> code](<a href="https://review.openstack.org/#/c/311273/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/311273/</a>) that someone is welcome<br>
>> to take over or use.<br>
><br>
>The operator issues already identified cover some things we’ve hit at Cambridge, please do scan through and contribute if there is anything they have not covered.<br>
><br>
<br>
</div></div>We have certainly had our share of BMC problems through the years. It is often frustrating as the very time you find you need the console, it is not working. Having Ironic doing an active monitoring (without overloading) would be a real help.<br>
<br>
The other item we’ve found difficult has been in the configuration:<br>
<br>
- Software maintenance is very limited. Some vendors choose to produce new versions of the BMC microcode without changing the version number reported by the BMC which makes consistent management difficult. There is no common API defined for updating the code.<br>
- Implementations between IPMI 1.5 and IPMI 2.0 vary significantly and between commodity white boxes and blades<br>
- BMCs have different Lan channels according to manufacturer for remote access<br>
- The tty speeds vary which means that the booted OS needs to have different cmdlines for the kernel according to the underlying hardware<br>
- the number of additional accounts is limited in some BMCs and password management is very basic. Currently, we define distinct users for read-only access to the SDRs (e.g. monitoring), console and power operations since these need to be kept in different systems. We also have unique passwords for each machine, all of which requires tracking. Foreman helps here but it is not ideal.<br>
- BMC replacement is also frequent. A process to re-import a replacement BMC (new MAC, no user accounts defined) would re-installing the box is needed.<br>
- we have a fairly complex reset process which hits the BMC with different levels of reset. We’ve also sometimes found the need to reset the IPMI kernel modules at the same time which go into a loop.<br>
<br>
I’m not expecting Ironic to fix all of this but it would be great to have a block of code which we can gradually improve together. There are other good initiatives like OpenBMC but they won’t help with the existing boxes.<br>
<br>
I think my best advice to Ironic for BMC management would be consider the BMC as a potentially unreliable device. Thus, along with performing the actions, checking they completed and probing that a function which was working an hour ago is still working now (but not overloading it)… we’ll be looking at Ironic this year so we’ll be able to help on the failure cases.<br>
<span class="HOEnZb"><font color="#888888"><br>
Tim<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
>Best wishes,<br>
>Stig<br>
><br>
>[1] <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-May/094658.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2016-May/094658.html</a><br>
>_______________________________________________<br>
>OpenStack-operators mailing list<br>
><a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</div></div></blockquote></div><br></div>
<br>
<span style="color:rgb(34,34,34);font-family:arial,sans-serif;background-color:rgb(255,255,255)">DataCentred Limited registered in England and Wales no. 05611763</span>