Open Stack

On 19 October 2017 at 13:37, Michał Jastrzębski <inc007 at gmail.com> wrote:
> On 19 October 2017 at 13:32, Joshua Harlow <harlowja at fastmail.com> wrote:
>> This reminded me of something I wanted to ask.
>>
>> Is it true to state that only way to get 'fully' shared-base layers is to
>> have `kolla-build` build all the projects (that a person/company/other may
>> use) in one invocation? (in part because of the jinja2 template generation
>> which would cause differences in dockerfiles?...)
>
> Well jinja2 should render same dockerfile no matter when you call it,
> so it should be fine. Alternatively you can run something like
> kolla-build nova --skip-parents  - this call will try to build all
> images with "nova" in them while not rebuilding openstack-base and
> base image.
>
>> I was pretty sure this was the case (unless things have changed), but just
>> wanting to check since that question seems (somewhat) on-topic...
>>
>> At godaddy we build individual projects using `kolla-build` (in part because
>> it makes it easy to rebuild + test + deply a single project with either an
>> update or a patch or ...) and I suspect others are doing this also (after
>> all the kolla-build command does take a regex of projects to build) - though
>> doing it in this way does seem like it would not reuse (all the layers
>> outside of the base operating system) layers 'optimally'?
>>
>> Thoughts?
>>
>> -Josh
>>
>> Sam Yaple wrote:
>>>
>>> docker_image wouldn't be the best place for that. Buf if you are looking
>>> for a quicker solution, kolla_docker was written specifically to be
>>> license compatible for openstack. its structure should make it easily
>>> adapted to delete an image. And you can copy it and cut it up thanks to
>>> the license.
>>>
>>> Are you pushing images with no shared base layers at all (300MB
>>> compressed image is no shared base layers)? With shared base layers a
>>> full image set of Kolla images should be much smaller than the numbers
>>> you posted.
>>>
>>> Thanks,
>>> SamYaple
>>>
>>> On Thu, Oct 19, 2017 at 11:03 AM, Gabriele Cerami <gcerami at redhat.com
>>> <mailto:gcerami at redhat.com>> wrote:
>>>
>>>     Hi,
>>>
>>>     our CI scripts are now automatically building, testing and pushing
>>>     approved openstack/RDO services images to public repositories in
>>>     dockerhub using ansible docker_image module.
>>>
>>>     Promotions have had some hiccups, but we're starting to regularly
>>> upload
>>>     new images every 4 hours.
>>>
>>>     When we'll get at full speed, we'll potentially have
>>>     - 3-4 different sets of images, one per release of openstack (counting
>>> a
>>>        EOL release grace period)
>>>     - 90-100 different services images per release
>>>     - 4-6 different versions of the same image ( keeping older promoted
>>>        images for a while )
>>>
>>>     At around 300MB per image a possible grand total is around 650GB of
>>>     space used.

That doesn't sound correct as images share a lot - full registry of
single type/distro (like centos source) is ~10gig

>>>     We don't know if this is acceptable usage of dockerhub space and for
>>>     this we already sent a similar email the to docker support to ask
>>>     specifically if the user would get penalty in any way (e.g. enforcing
>>>     quotas, rete limiting, blocking). We're still waiting for a reply.
>>>
>>>     In any case it's critical to keep the usage around the estimate, and
>>> to
>>>     achieve this we need a way to automatically delete the older images.
>>>     docker_image module does not provide this functionality, and we think
>>>     the only way is issuing direct calls to dockerhub API
>>>
>>>     https://docs.docker.com/registry/spec/api/#deleting-an-image
>>>     <https://docs.docker.com/registry/spec/api/#deleting-an-image>
>>>
>>>     docker_image module structure doesn't seem to encourage the addition
>>> of
>>>     such functionality directly in it, so we may be forced to use the uri
>>>     module.
>>>     With new images uploaded potentially every 4 hours, this will become a
>>>     problem to be solved within the next two weeks.
>>>
>>>     We'd appreciate any input for an existing, in progress and/or better
>>>     solution for bulk deletion, and issues that may arise with our space
>>>     usage in dockerhub
>>>
>>>     Thanks
>>>
>>>
>>> __________________________________________________________________________
>>>     OpenStack Development Mailing List (not for usage questions)
>>>     Unsubscribe:
>>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev