<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">```</div><div class="">import openstack</div><div class=""><br class=""></div><div class="">conn = openstack.connect()</div><div class=""><br class=""></div><div class="">conn.image.download_image(image_name, stream=True, output="data.iso”)</div><div class="">```</div><div class=""><br class=""></div><div class="">This gives me max performance of the network. Actually using stream=True may be slower (around 40%), but may be crucially necessary when dealing with huge images. Additionally you can specify chunk_size as param to download_image function, what aligns performance of stream vs non stream (for me stream=True and chunk_size=8192 resulted 2.3G image to be downloaded in 14 sec)</div><div class=""><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On 13. Oct 2022, at 23:24, Lucio Seki <<a href="mailto:lucioseki@gmail.com" class="">lucioseki@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="auto" class=""><div class="">Yes, I'm using tqdm to monitor the progress and speed.</div><div dir="auto" class="">I removed it, and it improved slightly (120kB/s -> 131kB/s) but not significantly :-/</div><div dir="auto" class=""><br class=""><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Thu, Oct 13, 2022, 16:54 Sean Mooney <<a href="mailto:smooney@redhat.com" class="">smooney@redhat.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Thu, 2022-10-13 at 16:21 -0300, Lucio Seki wrote:<br class="">
> Thanks Sean, that makes much easier to code!<br class="">
> <br class="">
> ```<br class="">
> ...<br class="">
> conn = openstack.connect(cloud_name)<br class="">
> <br class="">
> with open(path, 'wb') as image_file:<br class="">
> response = conn.image.download_image(image_name)<br class="">
> for chunk in tqdm(response.iter_content(), **tqdm_params):<br class="">
> image_file.write(chunk)<br class="">
> ```<br class="">
> <br class="">
> And it gave me some performance improvement (3kB/s -> 120kB/s).<br class="">
> ... though it would still take several days to download an image.<br class="">
> <br class="">
> Is there some tuning that I could apply?<br class="">
this is what nova does<br class="">
<a href="https://github.com/openstack/nova/blob/master/nova/image/glance.py#L344" rel="noreferrer noreferrer" target="_blank" class="">https://github.com/openstack/nova/blob/master/nova/image/glance.py#L344</a><br class="">
<br class="">
we get the image chunks by calling the data method on the glance client<br class="">
<a href="https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L373-L377" rel="noreferrer noreferrer" target="_blank" class="">https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L373-L377</a><br class="">
then bwe basiclly just loop over the chunks and write them to a file like you are<br class="">
<a href="https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L413-L437" rel="noreferrer noreferrer" target="_blank" class="">https://github.com/openstack/nova/blob/03d2715ed492350fa11908aea0fdd0265993e284/nova/image/glance.py#L413-L437</a><br class="">
we have some extra code for doing image verification but its basically the same as what you are doing<br class="">
we use eventlets to monkeypatch python io which can imporve performce but i woudl not expect it to be that dramatic<br class="">
and i dont think the glance clinet or opesntack client use eventlet so its sound liek something else is limiting the transfer speed.<br class="">
<br class="">
this is the glance client method we are invokeing <br class="">
<a href="https://github.com/openstack/python-glanceclient/blob/56186d6d5aa1a0c8fde99eeb535a650b0495925d/glanceclient/v2/images.py#L201-L271" rel="noreferrer noreferrer" target="_blank" class="">https://github.com/openstack/python-glanceclient/blob/56186d6d5aa1a0c8fde99eeb535a650b0495925d/glanceclient/v2/images.py#L201-L271</a><br class="">
<br class="">
<br class="">
im not sure what tqdm is by the way is it meusrign the transfer speed of something linke that?<br class="">
does the speed increase if you remvoe that?<br class="">
<a href="http://i.ie/" rel="noreferrer noreferrer" target="_blank" class="">i.ie</a> can you test this via a simple time script and see how much downloads say in up to 60 seconds by lookign at the file size?<br class="">
<br class="">
assuming its <a href="https://github.com/tqdm/tqdm" rel="noreferrer noreferrer" target="_blank" class="">https://github.com/tqdm/tqdm</a> perhaps the addtional io that woudl be doing to standard out is slowign it down?<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
> <br class="">
> On Thu, Oct 13, 2022, 14:18 Sean Mooney <<a href="mailto:smooney@redhat.com" target="_blank" rel="noreferrer" class="">smooney@redhat.com</a>> wrote:<br class="">
> <br class="">
> > On Thu, 2022-10-13 at 13:30 -0300, Lucio Seki wrote:<br class="">
> > > Hi glance experts,<br class="">
> > > <br class="">
> > > I'm using the following code to download a glance image:<br class="">
> > > <br class="">
> > > ```<br class="">
> > > from glanceapi import client<br class="">
> > > ...<br class="">
> > > glance = client.Client(GLANCE_API_VERSION, session=sess)<br class="">
> > > ...<br class="">
> > > with open(path, 'wb') as image_file:<br class="">
> > > data = glance.images.data(image_id)<br class="">
> > > for chunk in tqdm(data, unit='B', unit_scale=True,<br class="">
> > unit_divisor=1024):<br class="">
> > > image_file.write(chunk)<br class="">
> > > ```<br class="">
> > > <br class="">
> > > And I get a speed around 3kB/s. It would take months to download an<br class="">
> > image.<br class="">
> > > I'm using python3-glanceclient==3.6.0.<br class="">
> > > I even tried:<br class="">
> > > ```<br class="">
> > > for chunk in tqdm(data, unit='B', unit_scale=True,<br class="">
> > unit_divisor=1024):<br class="">
> > > pass<br class="">
> > > ```<br class="">
> > > to see if the bottleneck was the disk I/O, but didn't get any faster.<br class="">
> > > <br class="">
> > > In the same environment, when I use the glance CLI instead:<br class="">
> > > <br class="">
> > > ```<br class="">
> > > glance image-download --file $path $image_id<br class="">
> > > ```<br class="">
> > > I get hundreds of MB/s download speed, and it finishes in a few minutes.<br class="">
> > > <br class="">
> > > Is there anything I can do to improve the glanceclient performance?<br class="">
> > > I'm considering using subprocess.Popen(['glance', 'image-download', ...])<br class="">
> > > if nothing helps...<br class="">
> > have you considered using the openstacksdk instead<br class="">
> > <br class="">
> > the glanceclint is really only intendeted for other openstack service to<br class="">
> > use like<br class="">
> > nova or ironic.<br class="">
> > its not really ment to be used to write your onw code anymore.<br class="">
> > in the past it provided a programatic interface for interacting with glance<br class="">
> > but now you shoudl prefer the openstack sdk instead.<br class="">
> > <a href="https://github.com/openstack/openstacksdk" rel="noreferrer noreferrer" target="_blank" class="">https://github.com/openstack/openstacksdk</a><br class="">
> > <br class="">
> > > <br class="">
> > > Regards,<br class="">
> > > Lucio<br class="">
> > <br class="">
> > <br class="">
<br class="">
</blockquote></div></div></div>
</div></blockquote></div><br class=""></body></html>