Open Stack

On Mon, 2020-08-24 at 10:32 +0200, Dmitry Tantsur wrote:
> Hi,
> 
> On Mon, Aug 24, 2020 at 10:24 AM Arne Wiebalck <arne.wiebalck at cern.ch>
> wrote:
> 
> > Hi!
> > 
> > CERN's deployment is using the iscsi deploy interface since we started
> > with Ironic a couple of years ago (and we installed around 5000 nodes
> > with it by now). The reason we chose it at the time was simplicity: we
> > did not (and still do not) have a Swift backend to Glance, and the iscsi
> > interface provided a straightforward alternative.
> > 
> > While we have not seen obscure bugs/issues with it, I can certainly back
> > the scalability issues mentioned by Dmitry: the tunneling of the images
> > through the controllers can create issues when deploying hundreds of
> > nodes at the same time. The security of the iscsi interface is less of a
> > concern in our specific environment.
> > 
> > So, why did we not move to direct (yet)? In addition to the lack of
> > Swift, mostly since iscsi works for us and the scalability issues were
> > not that much of a burning problem ... so we focused on other things :)
> > 
> > Here are some thoughts/suggestions for this discussion:
> > 
> > How would 'direct' work with other Glance backends (like Ceph/RBD in our
> > case)? If using direct requires to duplicate images from Glance to
> > Ironic (or somewhere else) to be served, I think this would be an
> > argument against deprecating iscsi.
> > 
> 
> With image_download_source=http ironic will download the image to the
> conductor to be able serve it to the node. Which is exactly what the iscsi
> is doing, so not much of a change for you (except for s/iSCSI/HTTP/ as a
> means of serving the image).
> 
> Would it be an option for you to test direct deploy with
> image_download_source=http?
i think if there is still an option to not force deployemnt to altere any of there
other sevices this is likely ok but i think the onious shoudl be on the ironic
and ooo teams to ensure there is an upgrade path for those useres before this deprecation
becomes a removal without deploying swift or a swift compatibale api e.g. RadosGW

perhaps a ci job could be put in place maybe using grenade that starts with iscsi and moves
to direct with http porvided to show that just setting that weill allow the conductor to download
the image from glance and server it to the ipa.


unlike cern i just use ironic in a tiny home deployment where i have an all in one deployment + 4 addtional
nodes for ironic. i cant deploy swift as all my disks are already in use for cinder so down the line when 
i eventually upgrade to vicortia and wallaby  i would either have to drop ironic or not upgrade it
if there is not a option to just pull the image from glance or glance via the conductor. enhancing the ipa
to pull directly from glance would also proably work for many who use iscsi today but that would depend on your network
toplogy i guess.
> 
> 
> > 
> > Equally, if this would require to completely move the Glance backend to
> > something else, like from RBD to RadosGW, I would not expect happy
> > operators. (Does anyone know if RadosGW could even replace Swift for
> > this specific use case?)
> > 
> 
> AFAIK ironic works with RadosGW, we have some support code for it.
> 
> 
> > 
> > Do we have numbers on how many deployments use iscsi vs direct? If many
> > rely on iscsi, I would also suggest to establish a migration guide for
> > operators on how to move from iscsi to direct, for the various configs.
> > Recent versions of Glance support multiple backends, so a migration path
> > may be to add a new (direct compatible) backend for new images.
> > 
> 
> I don't have any numbers, but a migration guide is a must in any case.
> 
> I expect most TripleO consumers to use the iscsi deploy, but only because
> it's the default. Their Edge solution uses the direct deploy. I've polled a
> few operators I know, they all (except for you, obviously :) seem to use
> the direct deploy. Metal3 uses direct deploy.
> 
> Dmitry
> 
> 
> > 
> > Cheers,
> >   Arne
> > 
> > On 20.08.20 17:49, Julia Kreger wrote:
> > > I'm having a sense of deja vu!
> > > 
> > > Because of the way the mechanics work, the iscsi deploy driver is in
> > > an unfortunate position of being harder to troubleshoot and diagnose
> > > failures. Which basically means we've not been able to really identify
> > > common failures and add logic to handle them appropriately, like we
> > > are able to with a tcp socket and file download. Based on this alone,
> > > I think it makes a solid case for us to seriously consider
> > > deprecation.
> > > 
> > > Overall, I'm +1 for the proposal and I believe over two cycles is the
> > > right way to go.
> > > 
> > > I suspect we're going to have lots of push back from the TripleO
> > > community because there has been resistance to change their default
> > > usage in the past. As such I'm adding them to the subject so hopefully
> > > they will be at least aware.
> > > 
> > > I guess my other worry is operators who already have a substantial
> > > operational infrastructure investment built around the iscsi deploy
> > > interface. I wonder why they didn't use direct, but maybe they have
> > > all migrated in the past ?5? years. This could just be a non-concern
> > > in reality, I'm just not sure.
> > > 
> > > Of course, if someone is willing to step up and make the iscsi
> > > deployment interface their primary focus, that also shifts the
> > > discussion to making direct the default interface?
> > > 
> > > -Julia
> > > 
> > > 
> > > On Thu, Aug 20, 2020 at 1:57 AM Dmitry Tantsur <dtantsur at redhat.com>
> > 
> > wrote:
> > > > 
> > > > Hi all,
> > > > 
> > > > Side note for those lacking context: this proposal concerns deprecating
> > 
> > one of the ironic deploy interfaces detailed in
> > https://docs.openstack.org/ironic/latest/admin/interfaces/deploy.html. It
> > does not affect the boot-from-iSCSI feature.
> > > > 
> > > > I would like to propose deprecating and removing the 'iscsi' deploy
> > 
> > interface over the course of the next 2 cycles. The reasons are:
> > > > 1) The iSCSI deploy is a source of occasional cryptic bugs when a
> > 
> > target cannot be discovered or mounted properly.
> > > > 2) Its security is questionable: I don't think we even use
> > 
> > authentication.
> > > > 3) Operators confusion: right now we default to the iSCSI deploy but
> > 
> > pretty much direct everyone who cares about scalability or security to the
> > 'direct' deploy.
> > > > 4) Cost of maintenance: our feature set is growing, our team - not so
> > 
> > much. iscsi_deploy.py is 800 lines of code that can be removed, and some
> > dependencies that can be dropped as well.
> > > > 
> > > > As far as I can remember, we've kept the iSCSI deploy for two reasons:
> > > > 1) The direct deploy used to require Glance with Swift backend. The
> > 
> > recently added [agent]image_download_source option allows caching and
> > serving images via the ironic's HTTP server, eliminating this problem. I
> > guess we'll have to switch to 'http' by default for this option to keep the
> > out-of-box experience.
> > > > 2) Memory footprint of the direct deploy. With the raw images streaming
> > 
> > we no longer have to cache the downloaded images in the agent memory,
> > removing this problem as well (I'm not even sure how much of a problem it
> > is in 2020, even my phone has 4GiB of RAM).
> > > > 
> > > > If this proposal is accepted, I suggest to execute it as follows:
> > > > Victoria release:
> > > > 1) Put an early deprecation warning in the release notes.
> > > > 2) Announce the future change of the default value for
> > 
> > [agent]image_download_source.
> > > > W release:
> > > > 3) Change [agent]image_download_source to 'http' by default.
> > > > 4) Remove iscsi from the default enabled_deploy_interfaces and move it
> > 
> > to the back of the supported list (effectively making direct deploy the
> > default).
> > > > X release:
> > > > 5) Remove the iscsi deploy code from both ironic and IPA.
> > > > 
> > > > Thoughts, opinions, suggestions?
> > > > 
> > > > Dmitry
> > 
> >