Open Stack

On Friday, 26 May 2017 15:27:02 CEST Jeremy Stanley wrote:
> On 2017-04-28 14:39:51 +0200 (+0200), Luigi Toscano wrote:
> > The Sahara project has been providing pre-built images containing
> > the Hadoop/ Spark/$bigdata frameworks since the beginning of the
> > project, so that users can be immediately productive.
> > 
> > The generated qcow2 images have been living so far here:
> > http://sahara-files.mirantis.com/images/upstream/
> > 
> > As a team we were wondering whether we could store those images on
> > some shared and publicly accessible space on openstack.org (like
> > tarballs.openstack.org).
> > 
> > I guess that the main concern could be the disk usage. Currently
> > the space used for the older releases (from kilo to newton) is
> > around ~110GB. The estimate for Ocata is ~35GB and the number is
> > going to grow. Of course we can drop old images when a certain
> > release reaches its end-of- life (unless there is a place to store
> > some archived artifacts).
> > 
> > About the update frequency: the images are currenctly rebuilt with
> > with every commit in sahara-image-elements (and soon in sahara
> > with a different build method) by the tests. I don't think that we
> > would need to update the images in this stored space with every
> > commit, but at most once every month or, even better, when a new
> > release of sahara-image-elements is tagged.
> > 
> > Please note that we already store some artifacts on
> > tarballs.openstack.org, even if their size is not definitely not
> > the same of those disk images.
> > https://review.openstack.org/#/c/175395/
> > https://review.openstack.org/#/c/367271/
> > 
> > To summarize: would it be possible for us to use some shared
> > space, and if yes, which are the conditions?
> 
> Apologies for the delay in responding. We don't currently have
> sufficient free space to store the quantity of data you're talking
> about (some projects like Ironic, Trove and Kolla do or have in the
> past uploaded guest images there, but those are far fewer and much
> smaller than what you're requesting). We can see about extending the
> available free space for tarballs.openstack.org after we relocate it
> off the same server where we store our job logs, but we don't have a
> timeline for when that will be. I'm sorry I don't have better news
> on that front.

It wouldn't be a problem to wait a bit. Even if you don't have a timeline, do 
you know the approximate timing? Is it more 6 months, one year, or more?


> 
> On a separate note, what degree of security support is being
> provided for those images (as far as known vulnerabilities in
> non-OpenStack software aggregated within them)? There is still some
> concern expressed within the community around producing images of
> this nature for any purpose other than use within our CI system, in
> which case long-term archiving of release versions of those images
> is unnecessary. 

I'm aware and I've been following the discussion about container images on 
openstack-dev@ ("do we want to be publishing binary container images"). I 
understand (as I wrote there) that the decision made there could be relevant 
for this case too.
On one side the security issues are definitely important. On the other side 
the ready-to-use images can definitely help users to reduce the time needed to 
test the system.

It would be nice to have a clear way to mark the images as "unsupported, here 
be dragons"; there were some suggestions about this in the other thread, even 
if my understanding from the other discussion is that this is not considered 
possible or acceptable.




> If you need a periodic job to upload and replace
> pregenerated branch-tip snapshot images for consumption by other CI
> jobs, we should be able to work something out pretty easily (that's
> what the other projects I mentioned above have been doing).

This is not the use case described in my original request. Nevertheless, it 
could be useful for some of our scenario jobs. But wouldn't this be 
constrained by the lack of space as well?

-- 
Luigi