Open Stack

On 11/29/18 6:42 PM, Jiří Stránský wrote:
> On 28. 11. 18 18:29, Bogdan Dobrelya wrote:
>> On 11/28/18 6:02 PM, Jiří Stránský wrote:
>>> <snip>
>>>
>>>>
>>>> Reiterating again on previous points:
>>>>
>>>> -I'd be fine removing systemd. But lets do it properly and not via 'rpm
>>>> -ev --nodeps'.
>>>> -Puppet and Ruby *are* required for configuration. We can certainly put
>>>> them in a separate container outside of the runtime service containers
>>>> but doing so would actually cost you much more space/bandwidth for each
>>>> service container. As both of these have to get downloaded to each node
>>>> anyway in order to generate config files with our current mechanisms
>>>> I'm not sure this buys you anything.
>>>
>>> +1. I was actually under the impression that we concluded yesterday on
>>> IRC that this is the only thing that makes sense to seriously consider.
>>> But even then it's not a win-win -- we'd gain some security by leaner
>>> production images, but pay for it with space+bandwidth by duplicating
>>> image content (IOW we can help achieve one of the goals we had in mind
>>> by worsening the situation w/r/t the other goal we had in mind.)
>>>
>>> Personally i'm not sold yet but it's something that i'd consider if we
>>> got measurements of how much more space/bandwidth usage this would
>>> consume, and if we got some further details/examples about how serious
>>> are the security concerns if we leave config mgmt tools in runtime 
>>> images.
>>>
>>> IIRC the other options (that were brought forward so far) were already
>>> dismissed in yesterday's IRC discussion and on the reviews. Bin/lib bind
>>> mounting being too hacky and fragile, and nsenter not really solving the
>>> problem (because it allows us to switch to having different bins/libs
>>> available, but it does not allow merging the availability of bins/libs
>>> from two containers into a single context).
>>>
>>>>
>>>> We are going in circles here I think....
>>>
>>> +1. I think too much of the discussion focuses on "why it's bad to have
>>> config tools in runtime images", but IMO we all sorta agree that it
>>> would be better not to have them there, if it came at no cost.
>>>
>>> I think to move forward, it would be interesting to know: if we do this
>>> (i'll borrow Dan's drawing):
>>>
>>> |base container| --> |service container| --> |service container w/
>>> Puppet installed|
>>>
>>> How much more space and bandwidth would this consume per node (e.g.
>>> separately per controller, per compute). This could help with decision
>>> making.
>>
>> As I've already evaluated in the related bug, that is:
>>
>> puppet-* modules and manifests ~ 16MB
>> puppet with dependencies ~61MB
>> dependencies of the seemingly largest a dependency, systemd ~190MB
>>
>> that would be an extra layer size for each of the container images to be
>> downloaded/fetched into registries.
> 
> Thanks, i tried to do the math of the reduction vs. inflation in sizes 
> as follows. I think the crucial point here is the layering. If we do 
> this image layering:
> 
> |base| --> |+ service| --> |+ Puppet|
> 
> we'd drop ~267 MB from base image, but we'd be installing that to the 
> topmost level, per-component, right?

Given we detached systemd from puppet, cronie et al, that would be 
267-190MB, so the math below would be looking much better

> 
> In my basic deployment, undercloud seems to have 17 "components" (49 
> containers), overcloud controller 15 components (48 containers), and 
> overcloud compute 4 components (7 containers). Accounting for overlaps, 
> the total number of "components" used seems to be 19. (By "components" 
> here i mean whatever uses a different ConfigImage than other services. I 
> just eyeballed it but i think i'm not too far off the correct number.)
> 
> So we'd subtract 267 MB from base image and add that to 19 leaf images 
> used in this deployment. That means difference of +4.8 GB to the current 
> image sizes. My /var/lib/registry dir on undercloud with all the images 
> currently has 5.1 GB. We'd almost double that to 9.9 GB.
> 
> Going from 5.1 to 9.9 GB seems like a lot of extra traffic for the CDNs 
> (both external and e.g. internal within OpenStack Infra CI clouds).
> 
> And for internal traffic between local registry and overcloud nodes, it 
> gives +3.7 GB per controller and +800 MB per compute. That may not be so 
> critical but still feels like a considerable downside.
> 
> Another gut feeling is that this way of image layering would take longer 
> time to build and to run the modify-image Ansible role which we use in 
> CI, so that could endanger how our CI jobs fit into the time limit. We 
> could also probably measure this but i'm not sure if it's worth spending 
> the time.
> 
> All in all i'd argue we should be looking at different options still.
> 
>>
>> Given that we should decouple systemd from all/some of the dependencies
>> (an example topic for RDO [0]), that could save a 190MB. But it seems we
>> cannot break the love of puppet and systemd as it heavily relies on the
>> latter and changing packaging like that would higly likely affect
>> baremetal deployments with puppet and systemd co-operating.
> 
> Ack :/
> 
>>
>> Long story short, we cannot shoot both rabbits with a single shot, not
>> with puppet :) May be we could with ansible replacing puppet fully...
>> So splitting config and runtime images is the only choice yet to address
>> the raised security concerns. And let's forget about edge cases for now.
>> Tossing around a pair of extra bytes over 40,000 WAN-distributed
>> computes ain't gonna be our the biggest problem for sure.
>>
>> [0] https://review.rdoproject.org/r/#/q/topic:base-container-reduction
>>
>>>
>>>>
>>>> Dan
>>>>
>>>
>>> Thanks
>>>
>>> Jirka
>>>
>>> __________________________________________________________________________ 
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando