Open Stack

On Tue, Sep 18, 2018 at 2:09 PM, Peter Penchev <openstack-dev at storpool.com>
wrote:

> On Tue, Sep 18, 2018 at 11:32:37AM +0200, Attila Fazekas wrote:
> [format recovered; top-posting after an inline reply looks confusing]
> > On Mon, Sep 17, 2018 at 11:43 PM, Jay Pipes <jaypipes at gmail.com> wrote:
> >
> > > On 09/17/2018 09:39 AM, Peter Penchev wrote:
> > >
> > >> Hi,
> > >>
> > >> So here's a possibly stupid question - or rather, a series of such :)
> > >> Let's say a company has two (or five, or a hundred) datacenters in
> > >> geographically different locations and wants to deploy OpenStack in
> both.
> > >> What would be a deployment scenario that would allow relatively easy
> > >> migration (cold, not live) of instances from one datacenter to
> another?
> > >>
> > >> My understanding is that for servers located far away from one another
> > >> regions would be a better metaphor than availability zones, if only
> > >> because it would be faster for the various storage, compute, etc.
> > >> services to communicate with each other for the common case of doing
> > >> actions within the same datacenter.  Is this understanding wrong - is
> it
> > >> considered all right for groups of servers located in far away places
> to
> > >> be treated as different availability zones in the same cluster?
> > >>
> > >> If the groups of servers are put in different regions, though, this
> > >> brings me to the real question: how can an instance be migrated across
> > >> regions?  Note that the instance will almost certainly have some
> > >> shared-storage volume attached, and assume (not quite the common case,
> > >> but still) that the underlying shared storage technology can be taught
> > >> about another storage cluster in another location and can transfer
> > >> volumes and snapshots to remote clusters.  From what I've found, there
> > >> are three basic ways:
> > >>
> > >> - do it pretty much by hand: create snapshots of the volumes used in
> > >>    the underlying storage system, transfer them to the other storage
> > >>    cluster, then tell the Cinder volume driver to manage them, and
> spawn
> > >>    an instance with the newly-managed newly-transferred volumes
> > >>
> > >
> > > Yes, this is a perfectly reasonable solution. In fact, when I was at
> AT&T,
> > > this was basically how we allowed tenants to spin up instances in
> multiple
> > > regions: snapshot the instance, it gets stored in the Swift storage
> for the
> > > region, tenant starts the instance in a different region, and Nova
> pulls
> > > the image from the Swift storage in the other region. It's slow the
> first
> > > time it's launched in the new region, of course, since the bits need
> to be
> > > pulled from the other region's Swift storage, but after that, local
> image
> > > caching speeds things up quite a bit.
> > >
> > > This isn't migration, though. Namely, the tenant doesn't keep their
> > > instance ID, their instance's IP addresses, or anything like that.
>
> Right, sorry, I should have clarified that what we're interested in is
> technically creating a new instance with the same disk contents, so
> that's fine.  Thanks for confirming that there is not a simpler way that
> I've missed, I guess :)
>
> > > I've heard some users care about that stuff, unfortunately, which is
> why
> > > we have shelve [offload]. There's absolutely no way to perform a
> > > cross-region migration that keeps the instance ID and instance IP
> addresses.
> > >
> > > - use Cinder to backup the volumes from one region, then restore them
> to
> > >>    the other; if this is combined with a storage-specific Cinder
> backup
> > >>    driver that knows that "backing up" is "creating a snapshot" and
> > >>    "restoring to the other region" is "transferring that snapshot to
> the
> > >>    remote storage cluster", it seems to be the easiest way forward
> (once
> > >>    the Cinder backup driver has been written)
> > >>
> > >
> > > Still won't have the same instance ID and IP address, which is what
> > > certain users tend to complain about needing with move operations.
> > >
> > > - use Nova's "server image create" command, transfer the resulting
> > >>    Glance image somehow (possibly by downloading it from the Glance
> > >>    storage in one region and simulateneously uploading it to the
> Glance
> > >>    instance in the other), then spawn an instance off that image
> > >>
> > >
> > > Still won't have the same instance ID and IP address :)
> > >
> > > Best,
> > > -jay
> > >
> > > The "server image create" approach seems to be the simplest one,
> > >> although it is a bit hard to imagine how it would work without
> > >> transferring data unnecessarily (the online articles I've seen
> > >> advocating it seem to imply that a Nova instance in a region cannot be
> > >> spawned off a Glance image in another region, so there will need to be
> > >> at least one set of "download the image and upload it to the other
> > >> side", even if the volume-to-image and image-to-volume transfers are
> > >> instantaneous, e.g. using glance-cinderclient).  However, when I tried
> > >> it with a Nova instance backed by a StorPool volume (no ephemeral
> image
> > >> at all), the Glance image was zero bytes in length and only its
> metadata
> > >> contained some information about a volume snapshot created at that
> > >> point, so this seems once again to go back to options 1 and 2 for the
> > >> different ways to transfer a Cinder volume or snapshot to the other
> > >> region.  Or have I missed something, is there a way to get the "server
> > >> image create / image download / image create" route to handle volumes
> > >> attached to the instance?
> > >>
> > >> So... have I missed something else, too, or are these the options for
> > >> transferring a Nova instance between two distant locations?
> > >>
> > >> Thanks for reading this far, and thanks in advance for your help!
> > >>
> > >> Best regards,
> > >> Peter
> >
> > Create a volume transfer VM/machine in each region.
> > attache the volume -> dd -> compress  -> internet   ->decompress -> new
> > volume, attache(/boot with) to the volume to the final machine.
> > In case you have frequent transfers you may keep up the machines for the
> > next one..
>
> Thanks for the advice, but this would involve transferring *a lot* more
> data than if we leave it to the underlying storage :)  As I mentioned,
> the underlying storage can be taught about remote clusters and can be told
> to create a remote snapshot of a volume; this will be the base on which
> we will write our Cinder backup driver.  So both my options 1 (do it "by
> hand" with the underlying storage) and 2 (cinder volume backup/restore)
> would be preferable.
>

Cinder might get a feature for `rescue` a volume in case accidentally
someone
deleted the DB record or some other bad thing happened.
This needs to be admin only op where you would need to specify where is the
volume,
If just a new volume `shows up` on the storage, but  without
the knowledge of cinder, it could be rescued as well.

Among same storage types probably cinder could have an admin only
API for transfer.

I am not sure is volume backup/restore is really better across regions
than the above steps properly piped however
it is very infrastructure dependent,
bandwidth and latency across regions matters.

The direct storage usage likely better than the pipe/dd on close proximity,
but in case of good internal networking on longer distance (external net)
the diff will not be big,
you might wait more on openstack api/client than on the
actual data transfer, in case of small the size.
The internal network total bandwith nowadays is very huge
a little internal data move (storage->vm->internetet->vm->storage)
might not even shows ups on the internal monitors ;-)
The internet is the high latency thing even
if you have the best internet connection on the world.

The light is not getting faster. ;-)


>
> > In case the storage is just on the compute node: snapshot ->glance
> download
> > ->glance upload
>
> Right, as I mentioned in my description of the third option, this does
> not really work with attached volumes (thus your "just on the compute
> node")
> and as I mentioned before listing the options, the instances will almost
> certainly have attached volumes.
>
> yes, you need to use both way.

> Would be nice if cinder/glance could take the credentials for another
> > openstack and move the volume/image to another cinder/glance.
> >
> > If you want the same IP , specify the ip at instance boot time (port
> > create),
> > but you cannot be sure the same ip is always available or really
> route-able
> > to different region.. unless... VPN like solution in place...
> >
> > The uuid not expected to be changed by the users or admins (unsafe),
> > but you can use other metadata for description/your uuid.
>
> Best regards,
> Peter
>
> --
> Peter Penchev  openstack-dev at storpool.com  https://storpool.com/
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20180918/7a66c798/attachment.html>