<div dir="ltr">Adrian,<div><br></div><div>flannel_network_cidr and flannel_network_subnetlen are two parameters needed for flannel. flannel_network_cidr is the network range for flannel overlay network. flannel_network is the size of subnet assigned to each minion. When flannel starts, it needs the two parameters. Flannel will allocate a subnet from flannel_network_cidr for each minion. THe subnets on different minions are different. The data of flannel is stored in etcd. BIP is equal to the subnet created by flannel. MTU depends on whether we use vxlan in flannel. </div><div><br></div><div>If we use one docker daemon, we need to start the docker daemon without BIP first, then run flannel and etcd to generate BIP. After that, we need to kill the previous docker daemon and start a new docker daemon with BIP, then run etcd and flannel on it.</div><div><br></div><div>Regards,</div><div>Wanghua</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Dec 19, 2015 at 2:19 AM, Adrian Otto <span dir="ltr"><<a href="mailto:adrian.otto@rackspace.com" target="_blank">adrian.otto@rackspace.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
Wanghua,
<div><br>
</div>
<div>I see. The circular dependency you described does sound like a formidable challenge. Having multiple docker daemons violates the principle of least surprise. I worry that when it comes time to perform troubleshooting, an engineer would be surprised
to find multiple dockers running at the same time within the same compute instance.</div>
<div><br>
</div>
<div>Perhaps there is a way to generate the BIP and MTU before the docker daemon is started, then use those while starting docker, and start both flannel and etcd containers so all containers on the instance can share a single docker daemon? Would
that work at all? I guess I’d need a better understanding of exactly how the BIP and MTU are generated before judging if this is a good idea.</div><span class="HOEnZb"><font color="#888888">
<div><br>
</div>
<div>Adrian</div></font></span><div><div class="h5">
<div><br>
<div>
<blockquote type="cite">
<div>On Dec 16, 2015, at 11:40 PM, 王华 <<a href="mailto:wanghua.humble@gmail.com" target="_blank">wanghua.humble@gmail.com</a>> wrote:</div>
<br>
<div>
<div dir="ltr">Adrian,
<div><br>
</div>
<div>When the docker daemon starts, it needs to know the bip and mtu which are generated by flannel. So flannel and etcd should start before docker daemon, but if flannel and etcd run in the same daemon, it introduces a circle. We need another docker
daemon which is dedicated to flannel and etcd.</div>
<div><br>
</div>
<div>Regards</div>
<div>wanghua</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Dec 14, 2015 at 11:45 AM, Steven Dake (stdake) <span dir="ltr">
<<a href="mailto:stdake@cisco.com" target="_blank">stdake@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;font-size:14px;font-family:Calibri,sans-serif">
<div>Adrian,</div>
<div><br>
</div>
<div>Its a real shame Atomic can't execute its mission - serve as a container operating system. If you need some guidance on image building find experienced developers on #kolla – we have extensive experience in producing containers for various runtime
environments focused around OpenStack.</div>
<div><br>
</div>
<div>Regards</div>
<div>-steve</div>
<div><br>
</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;border-width:1pt medium medium;border-style:solid none none;padding:3pt 0in 0in;border-top-color:rgb(181,196,223)">
<span style="font-weight:bold">From: </span>Adrian Otto <<a href="mailto:adrian.otto@rackspace.com" target="_blank">adrian.otto@rackspace.com</a>><span><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
</span><span style="font-weight:bold">Date: </span>Monday, December 7, 2015 at 1:16 PM<span><br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
</span>
<div>
<div><span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [magnum]storage for docker-bootstrap<br>
</div>
</div>
</div>
<div>
<div>
<div><br>
</div>
<div>
<div dir="auto">
<div>Until I see evidence to the contrary, I think adding some bootstrap complexity to simplify the process of bay node image management and customization is worth it. Think about where most users will focus customization efforts. My guess is that
it will be within these docker images. We should ask our team to keep things as simple as possible while working to containerize components where that makes sense. That may take some creativity and a few iterations to achieve. </div>
<div><br>
</div>
<div>We can pivot on this later if we try it and hate it.</div>
<div><br>
Thanks,</div>
<div><br>
<div>Adrian</div>
</div>
<div><br>
On Dec 7, 2015, at 1:57 AM, Kai Qiang Wu <<a href="mailto:wkqwu@cn.ibm.com" target="_blank">wkqwu@cn.ibm.com</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<p>HI Hua,<br>
<br>
>From my point of view, not everything needed to be put in container. Let's make the initial start (be simple)to work and then discussed other options if needed in IRC or weekly meeting.<br>
<br>
<br>
Thanks<br>
<br>
Best Wishes,<br>
--------------------------------------------------------------------------------<br>
Kai Qiang Wu (吴开强 Kennan）<br>
IBM China System and Technology Lab, Beijing<br>
<br>
E-mail: <a href="mailto:wkqwu@cn.ibm.com" target="_blank">wkqwu@cn.ibm.com</a><br>
Tel: 86-10-82451647<br>
Address: Building 28(Ring Building), ZhongGuanCun Software Park, <br>
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193<br>
--------------------------------------------------------------------------------<br>
Follow your heart. You are miracle! <br>
<br>
<graycol.gif><font color="#424282">王华 ---07/12/2015 10:10:38 am---Hi all, If we want to run etcd and flannel in container, we will introduce</font><br>
<br>
<font size="2" color="#5F5F5F">From: </font><font size="2">王华 <<a href="mailto:wanghua.humble@gmail.com" target="_blank">wanghua.humble@gmail.com</a>></font><br>
<font size="2" color="#5F5F5F">To: </font><font size="2">Egor Guz <<a href="mailto:EGuz@walmartlabs.com" target="_blank">EGuz@walmartlabs.com</a>></font><br>
<font size="2" color="#5F5F5F">Cc: </font><font size="2">"<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>></font><br>
<font size="2" color="#5F5F5F">Date: </font><font size="2">07/12/2015 10:10 am</font><br>
<font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [openstack-dev] [magnum]storage for docker-bootstrap</font><br>
</p>
<hr width="100%" size="2" align="left" noshade style="color:#8091a5">
<br>
<br>
<br>
Hi all, <br>
<br>
If we want to run etcd and flannel in container, we will introduce docker-bootstrap which makes setup become more complex as Egor pointed out. Should we pay for the price? <br>
<br>
<font size="4">On Sat, Nov 28, 2015 at 8:45 AM, Egor Guz <</font><a href="mailto:EGuz@walmartlabs.com" target="_blank"><u><font size="4" color="#0000FF">EGuz@walmartlabs.com</font></u></a><font size="4">> wrote:</font>
<ul>
<font size="4">Wanghua,<br>
<br>
I don’t think moving flannel to the container is good idea. This is setup great for dev environment, but become too complex from operator point of view (you add extra Docker daemon and need extra Cinder volume for this daemon, also<br>
keep in mind it makes sense to keep etcd data folder at Cinder storage as well because etcd is database). flannel has just there files without extra dependencies and it’s much easy to download it during cloud-init ;)<br>
<br>
I agree that we have pain with building Fedora Atomic images, but instead of simplify this process we should switch to another more “friendly” images (e.g. Fedora/CentOS/Ubuntu) which we can easy build with disk builder.<br>
Also we can fix CoreOS template (I believe people more asked about it instead of Atomic), but we may face similar to Atomic issues when we will try to integrate not CoreOS products (e.g. Calico or Weave)<br>
<br>
—<br>
Egor<br>
<br>
From: 王华 <</font><a href="mailto:wanghua.humble@gmail.com" target="_blank"><u><font size="4" color="#0000FF">wanghua.humble@gmail.com</font></u></a><font size="4"><mailto:</font><a href="mailto:wanghua.humble@gmail.com" target="_blank"><u><font size="4" color="#0000FF">wanghua.humble@gmail.com</font></u></a><font size="4">>><br>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4"><mailto:</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4">>><br>
Date: Thursday, November 26, 2015 at 00:15<br>
To: "OpenStack Development Mailing List (not for usage questions)" <</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4"><mailto:</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4">>><br>
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap<br>
<br>
Hi Hongbin,<br>
<br>
The docker in master node stores data in /dev/mapper/atomicos-docker--data and metadata in /dev/mapper/atomicos-docker--meta. /dev/mapper/atomicos-docker--data and /dev/mapper/atomicos-docker--meta are logic volumes. The docker in minion node store data in
the cinder volume, but /dev/mapper/atomicos-docker--meta and /dev/mapper/atomicos-docker--meta are not used. If we want to leverage Cinder volume for docker in master, should we drop /dev/mapper/atomicos-docker--meta and /dev/mapper/atomicos-docker--meta?
I think it is not necessary to allocate a Cinder volume. It is enough to allocate two logic volumes for docker, because only etcd, flannel, k8s run in the docker daemon which need not a large amount of storage.<br>
<br>
Best regards,<br>
Wanghua<br>
<br>
On Thu, Nov 26, 2015 at 12:40 AM, Hongbin Lu <</font><a href="mailto:hongbin.lu@huawei.com" target="_blank"><u><font size="4" color="#0000FF">hongbin.lu@huawei.com</font></u></a><font size="4"><mailto:</font><a href="mailto:hongbin.lu@huawei.com" target="_blank"><u><font size="4" color="#0000FF">hongbin.lu@huawei.com</font></u></a><font size="4">>>
wrote:<br>
Here is a bit more context.<br>
<br>
Currently, at k8s and swarm bay, some required binaries (i.e. etcd and flannel) are built into image and run at host. We are exploring the possibility to containerize some of these system components. The rationales are (i) it is infeasible to build custom packages
into an atomic image and (ii) it is infeasible to upgrade individual component. For example, if there is a bug in current version of flannel and we know the bug was fixed in the next version, we need to upgrade flannel by building a new image, which is a tedious
process.<br>
<br>
To containerize flannel, we need a second docker daemon, called docker-bootstrap [1]. In this setup, pods are running on the main docker daemon, and flannel and etcd are running on the second docker daemon. The reason is that flannel needs to manage the network
of the main docker daemon, so it needs to run on a separated daemon.<br>
<br>
Daneyon, I think it requires separated storage because it needs to run a separated docker daemon (unless there is a way to make two docker daemons share the same storage).<br>
<br>
Wanghua, is it possible to leverage Cinder volume for that. Leveraging external storage is always preferred [2].<br>
<br>
[1] </font><a href="http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html#bootstrap-docker" target="_blank"><u><font size="4" color="#0000FF">http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode.html#bootstrap-docker</font></u></a><font size="4"><br>
[2] </font><a href="http://www.projectatomic.io/docs/docker-storage-recommendation/" target="_blank"><u><font size="4" color="#0000FF">http://www.projectatomic.io/docs/docker-storage-recommendation/</font></u></a><font size="4"><br>
<br>
Best regards,<br>
Hongbin<br>
<br>
From: Daneyon Hansen (danehans) [mailto:</font><a href="mailto:danehans@cisco.com" target="_blank"><u><font size="4" color="#0000FF">danehans@cisco.com</font></u></a><font size="4"><mailto:</font><a href="mailto:danehans@cisco.com" target="_blank"><u><font size="4" color="#0000FF">danehans@cisco.com</font></u></a><font size="4">>]<br>
Sent: November-25-15 11:10 AM<br>
To: OpenStack Development Mailing List (not for usage questions)<br>
Subject: Re: [openstack-dev] [magnum]storage for docker-bootstrap<br>
<br>
<br>
<br>
From: 王华 <</font><a href="mailto:wanghua.humble@gmail.com" target="_blank"><u><font size="4" color="#0000FF">wanghua.humble@gmail.com</font></u></a><font size="4"><mailto:</font><a href="mailto:wanghua.humble@gmail.com" target="_blank"><u><font size="4" color="#0000FF">wanghua.humble@gmail.com</font></u></a><font size="4">>><br>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4"><mailto:</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4">>><br>
Date: Wednesday, November 25, 2015 at 5:00 AM<br>
To: "OpenStack Development Mailing List (not for usage questions)" <</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4"><mailto:</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4">>><br>
Subject: [openstack-dev] [magnum]storage for docker-bootstrap<br>
<br>
Hi all,<br>
<br>
I am working on containerizing etcd and flannel. But I met a problem. As described in [1], we need a docker-bootstrap. Docker and docker-bootstrap can not use the same storage, so we need some disk space for it.<br>
<br>
I reviewed [1] and I do not see where the bootstrap docker instance requires separate storage.<br>
<br>
The docker in master node stores data in /dev/mapper/atomicos-docker--data and metadata in /dev/mapper/atomicos-docker--meta. The disk space left is too same for docker-bootstrap. Even if the root_gb of the instance flavor is 20G, only 8G can be used in our
image. I want to make it bigger. One way is we can add the disk space left in the vda as vda3 into atomicos vg after the instance starts and we allocate two logic volumes for docker-bootstrap. Another way is when we create the image, we allocate two logic
volumes for docker-bootstrap. The second way has a advantage. It doesn't have to make filesystem when the instance is created which is time consuming.<br>
<br>
What is your opinion?<br>
<br>
Best Regards<br>
Wanghua<br>
<br>
[1] </font><a href="http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode/master.html" target="_blank"><u><font size="4" color="#0000FF">http://kubernetes.io/v1.1/docs/getting-started-guides/docker-multinode/master.html</font></u></a><font size="4"><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: </font><a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" target="_blank"><u><font size="4" color="#0000FF">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</font></u></a><font size="4"><</font><a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" target="_blank"><u><font size="4" color="#0000FF">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</font></u></a><font size="4">></font><u><font size="4" color="#0000FF"><br>
</font></u><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><u><font size="4" color="#0000FF">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></u></a><font size="4"><br>
<br>
</font></ul>
<tt>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br>
</tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br>
</tt><br>
<br>
<br>
</div>
</blockquote>
<blockquote type="cite">
<div><span>__________________________________________________________________________</span><br>
<span>OpenStack Development Mailing List (not for usage questions)</span><br>
<span>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe</span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></span><br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</span></div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</blockquote>
</div>
<br>
</div>
</div></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>