Open Stack

Hi,

I’ve been testing out magnum (rocky) in my staging environment and I’ve been running into an issue I can’t really explain. Essentially, Magnum is able to provision all the VMs it needs according to the template. However, when it comes time to install kubernetes through cloud-init, cloud-init is unable to fetch the user-data. 

I must specify that ONLY the user-data is affected. Everything else that’s read from the metadata gets through without any issue. I can also do 'curl http://169.254.169.254/2009-04-04/' <http://169.254.169.254/2009-04-04/'> from the VM without any issue. When I try to curl http://169.254.169.254/2009-04-04/user-data <http://169.254.169.254/2009-04-04/user-data> , curl just hangs without any data showing up :

sudo curl -v http://169.254.169.254/2009-04-04/user-data
*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /2009-04-04/user-data HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.59.0
> Accept: */*

Tcpdump -vvv looks like this :

sudo tcpdump -vvv -i eth0 host 169.254.169.254
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:36:22.284195 IP (tos 0x0, ttl 64, id 11248, offset 0, flags [DF], proto TCP (6), length 60)
    kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [S], cksum 0x5e34 (incorrect -> 0x51a5), seq 2359174217, win 26730, options [mss 8910,sackOK,TS val 583688334 ecr 0,nop,wscale 7], length 0
20:36:22.284793 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    169.254.169.254.http > kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896: Flags [S.], cksum 0x9ed1 (correct), seq 2099498427, ack 2359174218, win 26694, options [mss 8910,sackOK,TS val 4068504964 ecr 583688334,nop,wscale 9], length 0
20:36:22.284875 IP (tos 0x0, ttl 64, id 11249, offset 0, flags [DF], proto TCP (6), length 52)
    kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [.], cksum 0x5e2c (incorrect -> 0x522e), seq 1, ack 1, win 209, options [nop,nop,TS val 583688335 ecr 4068504964], length 0
20:36:22.284962 IP (tos 0x0, ttl 64, id 11250, offset 0, flags [DF], proto TCP (6), length 151)
    kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [P.], cksum 0x5e8f (incorrect -> 0xb692), seq 1:100, ack 1, win 209, options [nop,nop,TS val 583688335 ecr 4068504964], length 99: HTTP, length: 99
        GET /2009-04-04/user-data HTTP/1.1
        Host: 169.254.169.254
        User-Agent: curl/7.59.0
        Accept: */*

20:36:22.325212 IP (tos 0x0, ttl 64, id 47160, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896: Flags [.], cksum 0x523e (correct), seq 1, ack 100, win 53, options [nop,nop,TS val 4068505005 ecr 583688335], length 0
20:36:33.546523 IP (tos 0x0, ttl 64, id 11251, offset 0, flags [DF], proto TCP (6), length 52)
    kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [F.], cksum 0x5e2c (incorrect -> 0x25a4), seq 100, ack 1, win 209, options [nop,nop,TS val 583699596 ecr 4068505005], length 0
20:36:33.586154 IP (tos 0x0, ttl 64, id 47169, offset 0, flags [DF], proto TCP (6), length 52)
    169.254.169.254.http > kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896: Flags [.], cksum 0x91fc (correct), seq 26695, ack 101, win 53, options [nop,nop,TS val 4068516266 ecr 583699596], length 0


So the instance does receive packets as a reply to the curl, but they’re empty.

I have tested on both Fedora-Atomic-27-20180212.2.x86_64.qcow2 and Fedora-AtomicHost-28-20180425.0.x86_64.qcow2 images and reproduced the issue both times. Nova-api and neutron-metadata logs show http 200 for each of my requests to nova. What could be blocking the user-data from reaching the instance in this case? Is there a way to set magnum-created intances to use configdrive instead?

I must also specify that user-data works on all regular instances. It’s only affecting instances created by magnum.

Best regards,

Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200428/8c52fdd1/attachment.html>