Open Stack

```
import openstack

conn = openstack.connect()

conn.image.download_image(image_name, stream=True, output="data.iso”)
```

This gives me max performance of the network. Actually using stream=True may be slower (around 40%), but may be crucially necessary when dealing with huge images. Additionally you can specify chunk_size as param to download_image function, what aligns performance of stream vs non stream (for me stream=True and chunk_size=8192 resulted 2.3G image to be downloaded in 14 sec)


> On 13. Oct 2022, at 23:24, Lucio Seki <lucioseki at gmail.com> wrote:
> 
> Yes, I'm using tqdm to monitor the progress and speed.
> I removed it, and it improved slightly (120kB/s -> 131kB/s) but not significantly :-/
> 
> On Thu, Oct 13, 2022, 16:54 Sean Mooney <smooney at redhat.com <mailto:smooney at redhat.com>> wrote:
> On Thu, 2022-10-13 at 16:21 -0300, Lucio Seki wrote:
> > Thanks Sean, that makes much easier to code!
> > 
> > ```
> > ...
> > conn = openstack.connect(cloud_name)
> > 
> > with open(path, 'wb') as image_file:
> >     response = conn.image.download_image(image_name)
> >     for chunk in tqdm(response.iter_content(), **tqdm_params):
> >         image_file.write(chunk)
> > ```
> > 
> > And it gave me some performance improvement (3kB/s -> 120kB/s).
> > ... though it would still take several days to download an image.
> > 
> > Is there some tuning that I could apply?
> this is what nova does
> https://github.com/openstack/nova/blob/master/nova/image/glance.py#L344 <https://github.com/openstack/nova/blob/master/nova/image/glance.py#L344>
> 
> we get the image chunks by calling the data method on the glance client
> https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L373-L377 <https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L373-L377>
> then bwe basiclly just loop over the chunks and write them to a file like you are
> https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L413-L437 <https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L413-L437>
> we have some extra code for doing image verification but its basically the same as what you are doing
> we use eventlets to monkeypatch python io which can imporve performce but i woudl not expect it to be that dramatic
> and i dont think the glance clinet or opesntack client use eventlet so its sound liek something else is limiting the transfer speed.
> 
> this is the glance client method we are invokeing 
> https://github.com/openstack/python-glanceclient/blob/56186d6d5aa1a0c8fde99eeb535a650b0495925d/glanceclient/v2/images.py#L201-L271 <https://github.com/openstack/python-glanceclient/blob/56186d6d5aa1a0c8fde99eeb535a650b0495925d/glanceclient/v2/images.py#L201-L271>
> 
> 
> im not sure what tqdm is by the way is it meusrign the transfer speed of something linke that?
> does the speed increase if you remvoe that?
> i.ie <http://i.ie/> can you test this via a simple time script and see how much downloads say in up to 60 seconds by lookign at the file size?
> 
> assuming its https://github.com/tqdm/tqdm <https://github.com/tqdm/tqdm> perhaps the addtional io that woudl be doing to standard out is slowign it down?
> 
> 
> 
> 
> > 
> > On Thu, Oct 13, 2022, 14:18 Sean Mooney <smooney at redhat.com <mailto:smooney at redhat.com>> wrote:
> > 
> > > On Thu, 2022-10-13 at 13:30 -0300, Lucio Seki wrote:
> > > > Hi glance experts,
> > > > 
> > > > I'm using the following code to download a glance image:
> > > > 
> > > > ```
> > > > from glanceapi import client
> > > > ...
> > > > glance = client.Client(GLANCE_API_VERSION, session=sess)
> > > > ...
> > > > with open(path, 'wb') as image_file:
> > > >     data = glance.images.data(image_id)
> > > >     for chunk in tqdm(data, unit='B', unit_scale=True,
> > > unit_divisor=1024):
> > > >         image_file.write(chunk)
> > > > ```
> > > > 
> > > > And I get a speed around 3kB/s. It would take months to download an
> > > image.
> > > > I'm using python3-glanceclient==3.6.0.
> > > > I even tried:
> > > > ```
> > > >     for chunk in tqdm(data, unit='B', unit_scale=True,
> > > unit_divisor=1024):
> > > >         pass
> > > > ```
> > > > to see if the bottleneck was the disk I/O, but didn't get any faster.
> > > > 
> > > > In the same environment, when I use the glance CLI instead:
> > > > 
> > > > ```
> > > > glance image-download --file $path $image_id
> > > > ```
> > > > I get hundreds of MB/s download speed, and it finishes in a few minutes.
> > > > 
> > > > Is there anything I can do to improve the glanceclient performance?
> > > > I'm considering using subprocess.Popen(['glance', 'image-download', ...])
> > > > if nothing helps...
> > > have you considered using the openstacksdk instead
> > > 
> > > the glanceclint is really only intendeted for other openstack service to
> > > use like
> > > nova or ironic.
> > > its not really ment to be used to write your onw code anymore.
> > > in the past it provided a programatic interface for interacting with glance
> > > but now you shoudl prefer the openstack sdk instead.
> > > https://github.com/openstack/openstacksdk <https://github.com/openstack/openstacksdk>
> > > 
> > > > 
> > > > Regards,
> > > > Lucio
> > > 
> > > 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221014/b667d289/attachment.htm>