<div dir="ltr">Hi
<span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Muhammad,</span><div><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div>I am not sure what is the exact problem, but here is the list of things you might want to check:</div><div><br></div><div>1. Make sure the security group is open. This document explains how to find the security group(s) of the container: <a href="https://docs.openstack.org/zun/latest/admin/security-groups.html#find-container-s-security-groups">https://docs.openstack.org/zun/latest/admin/security-groups.html#find-container-s-security-groups</a> .</div><div><br></div><div>2. Check if you can ping the container from outside</div><div>$ NET_ID=$(openstack network show private | awk '/ id /{print $4}')<br></div><div><div>$ sudo ip netns | grep $NET_ID</div><div>qdhcp-6d688072-a0c3-4f1c-979e-2d1882564931</div></div><div><div>$ sudo ip netns exec qdhcp-6d688072-a0c3-4f1c-979e-2d1882564931 ping 10.0.0.9</div><div>PING 10.0.0.9 (10.0.0.9) 56(84) bytes of data.</div><div>64 bytes from <a href="http://10.0.0.9">10.0.0.9</a>: icmp_seq=1 ttl=64 time=0.845 ms</div><div>64 bytes from <a href="http://10.0.0.9">10.0.0.9</a>: icmp_seq=2 ttl=64 time=0.258 ms</div></div><div>...</div><div><br></div><div>3. Check if you can ping outside from the container</div><div><div>$ zun list</div><div>...</div><div>| 2c5d01ef-11f9-46f6-8ef2-da59914b6a10 | pi-24-container | nginx | Running | None | 10.0.0.9, fd66:a11d:c60:0:f816:3eff:fe9c:8f46 | [80] |</div><div>...</div></div><div>$ CONTAINER_ID=$(zun show pi-24-container | awk '/ uuid /{print $4}')</div><div><div>$ docker ps | grep $CONTAINER_ID</div><div>f9cd8aa9a911 nginx:latest "nginx -g 'daemon of…" 21 minutes ago Up 21 minutes zun-2c5d01ef-11f9-46f6-8ef2-da59914b6a10</div></div><div><div>$ docker inspect -f {{.State.Pid}} f9cd8aa9a911</div><div>15001</div></div><div><div>$ sudo nsenter -t 15001 -n ping 8.8.8.8</div><div>PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.</div><div>64 bytes from <a href="http://8.8.8.8">8.8.8.8</a>: icmp_seq=1 ttl=49 time=1.46 ms</div><div>64 bytes from <a href="http://8.8.8.8">8.8.8.8</a>: icmp_seq=2 ttl=49 time=0.957 ms</div></div><div><br></div><div>4. traceroute</div><div><div>$ sudo nsenter -t 15001 -n traceroute 8.8.8.8</div><div>traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets</div><div> 1 ip-10-0-0-1.ec2.internal (10.0.0.1) 3.173 ms 3.154 ms 3.139 ms</div><div> 2 ip-172-24-4-1.ec2.internal (172.24.4.1) 3.086 ms 3.074 ms 3.064 ms</div></div><div>...</div><div><br></div><div>5. ping & tcpdump on various interfaces. This is for checking where the traffic is blocked.</div><div>$ sudo ip netns exec qdhcp-6d688072-a0c3-4f1c-979e-2d1882564931 tcpdump -i tap2813b3ae-8d<br></div><div>...</div><div><br></div><div>See if you can find something by performing above steps. In any case, you might consider restarting the neutron processes which might fix everything magically. If no, I would need more details about your setup.</div><div><br></div><div>Best regards,</div><div>Hongbin </div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Jun 22, 2018 at 5:05 AM Usman Awais <<a href="mailto:usman.awais@gmail.com" target="_blank">usman.awais@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Hongbin,<div><br></div><div>Many thanks, got it running, that was awesom... :) The problem was unsynched time. I installed chrony and it started working. </div><div><br></div><div>Now I am running into another problem; the networking of the container. The container gets started, I can shell into it through appcontainer API, it even gets the correct IP address of my private network (named priv-net) in openstack, <span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">through DHCP</span>. But when I ping any address, even the address of the priv-net's gateway, it does nothing. I have following configuration</div><div><br></div><div>neutron-openvswitch-agent is running</div><div>neutron-ovs-cleanup is running<br></div><div>neutron-destroy-patch-ports is running<br></div><div>kuryr-libnetwork is running<br></div><div>docker is running<br></div><div>zun-compute is running<br></div><div><br></div><div>The eth0 network card has standard configuration of an OVSBridge.</div><div><br></div><div>When I create a new container it also creates taps and patch ports on the compute node. Now I am going to try to use kuryr script to test what happens with "bridged" and "host" networks.</div></div><div class="gmail_extra"><br clear="all"><div><div class="m_5580317170493731144m_-3858003578663069819gmail_signature" data-smartmail="gmail_signature">Muhammad Usman Awais<br><br><br></div></div>
<br><div class="gmail_quote">On Thu, Jun 21, 2018 at 1:14 PM, Hongbin Lu <span dir="ltr"><<a href="mailto:hongbin034@gmail.com" target="_blank">hongbin034@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>HI
<span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Muhammad,</span></div><div><br></div><div>Here is the code (run in controller node) that decides whether a service is up <a href="https://github.com/openstack/zun/blob/master/zun/api/servicegroup.py" target="_blank">https://github.com/openstack/zun/blob/master/zun/api/servicegroup.py</a> . There are several possibilities to cause a service to be 'down':</div><div>1. The service was being 'force_down' via API (e.g. explicitly issued a command like "appcontainer service forcedown")</div><div>2. The zun compute process is not doing the heartbeat for a certain period of time (CONF.service_down_time).</div><div>3. The zun compute process is doing the heartbeat properly but the time between controller node and compute node is out of sync.</div><div><br></div><div>In before, #3 is the common pitfall that people ran into. If it is not #3, you might want to check if the zun compute process is doing the heartbeat properly. Each zun compute process is running a periodic task to update its state in DB: <a href="https://github.com/openstack/zun/blob/master/zun/servicegroup/zun_service_periodic.py" target="_blank">https://github.com/openstack/zun/blob/master/zun/servicegroup/zun_service_periodic.py</a> . The call of '
<span style="color:rgb(36,41,46);font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;white-space:pre-wrap;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">report_state_up</span>
' will record this service is up in DB at current time. You might want to check if this periodic task is running properly, or if the current state is updated in the DB.<br></div><div><br></div><div>Above is my best guess. Please feel free to follow it up with me or other team members if you need further assistant for this issue.</div><div><br></div><div>Best regards,</div><div>Hongbin</div><br><div class="gmail_quote"><div><div class="m_5580317170493731144m_-3858003578663069819h5"><div dir="ltr">On Wed, Jun 20, 2018 at 9:14 AM Usman Awais <<a href="mailto:usman.awais@gmail.com" target="_blank">usman.awais@gmail.com</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="m_5580317170493731144m_-3858003578663069819h5"><div dir="ltr">Dear Zuners,<div><br></div><div>I have installed RDO pike. I stopped openstack-nova-compute service on one of the hosts, and installed a zun-compute service. Although all the services seems to be running ok on both controller and compute but when I do </div><div><br></div><div> openstack appcontainer service list</div><div><br></div><div>It gives me following</div><div><br></div><div>+----+--------------+-------------+-------+----------+-----------------+---------------------+-------------------+</div><div>| Id | Host | Binary | State | Disabled | Disabled Reason | Updated At | Availability Zone |</div><div>+----+--------------+-------------+-------+----------+-----------------+---------------------+-------------------+</div><div>| 1 | node1.os.lab | zun-compute | down | False | None | 2018-06-20 13:14:31 | nova |</div><div>+----+--------------+-------------+-------+----------+-----------------+---------------------+-------------------+</div><div><br></div><div>I have checked all ports in both directions they are open, including etcd ports and others. All services are running, only docker service has the warning message saying "failed to retrieve docker-runc version: exec: \"docker-runc\": executable file not found in $PATH". There is also a message at zun-compute "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/default_comparator.py:161: SAWarning: The IN-predicate on "container.uuid" was invoked with an empty sequence. This results in a contradiction, which nonetheless can be expensive to evaluate. Consider alternative strategies for improved performance."</div><div><br></div><div>Please guide...</div><div><br></div><div>Regards,<br clear="all"><div><div class="m_5580317170493731144m_-3858003578663069819m_1670455320156694095gmail-m_-8573622893539285047gmail_signature">Muhammad Usman Awais<br><br><br></div></div>
</div></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div>