Hi, I’ve been testing out magnum (rocky) in my staging environment and I’ve been running into an issue I can’t really explain. Essentially, Magnum is able to provision all the VMs it needs according to the template. However, when it comes time to install kubernetes through cloud-init, cloud-init is unable to fetch the user-data. I must specify that ONLY the user-data is affected. Everything else that’s read from the metadata gets through without any issue. I can also do 'curl http://169.254.169.254/2009-04-04/' <http://169.254.169.254/2009-04-04/'> from the VM without any issue. When I try to curl http://169.254.169.254/2009-04-04/user-data <http://169.254.169.254/2009-04-04/user-data> , curl just hangs without any data showing up : sudo curl -v http://169.254.169.254/2009-04-04/user-data * Trying 169.254.169.254... * TCP_NODELAY set * Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
GET /2009-04-04/user-data HTTP/1.1 Host: 169.254.169.254 User-Agent: curl/7.59.0 Accept: */*
Tcpdump -vvv looks like this : sudo tcpdump -vvv -i eth0 host 169.254.169.254 tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 20:36:22.284195 IP (tos 0x0, ttl 64, id 11248, offset 0, flags [DF], proto TCP (6), length 60) kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [S], cksum 0x5e34 (incorrect -> 0x51a5), seq 2359174217, win 26730, options [mss 8910,sackOK,TS val 583688334 ecr 0,nop,wscale 7], length 0 20:36:22.284793 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 169.254.169.254.http > kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896: Flags [S.], cksum 0x9ed1 (correct), seq 2099498427, ack 2359174218, win 26694, options [mss 8910,sackOK,TS val 4068504964 ecr 583688334,nop,wscale 9], length 0 20:36:22.284875 IP (tos 0x0, ttl 64, id 11249, offset 0, flags [DF], proto TCP (6), length 52) kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [.], cksum 0x5e2c (incorrect -> 0x522e), seq 1, ack 1, win 209, options [nop,nop,TS val 583688335 ecr 4068504964], length 0 20:36:22.284962 IP (tos 0x0, ttl 64, id 11250, offset 0, flags [DF], proto TCP (6), length 151) kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [P.], cksum 0x5e8f (incorrect -> 0xb692), seq 1:100, ack 1, win 209, options [nop,nop,TS val 583688335 ecr 4068504964], length 99: HTTP, length: 99 GET /2009-04-04/user-data HTTP/1.1 Host: 169.254.169.254 User-Agent: curl/7.59.0 Accept: */* 20:36:22.325212 IP (tos 0x0, ttl 64, id 47160, offset 0, flags [DF], proto TCP (6), length 52) 169.254.169.254.http > kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896: Flags [.], cksum 0x523e (correct), seq 1, ack 100, win 53, options [nop,nop,TS val 4068505005 ecr 583688335], length 0 20:36:33.546523 IP (tos 0x0, ttl 64, id 11251, offset 0, flags [DF], proto TCP (6), length 52) kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896 > 169.254.169.254.http: Flags [F.], cksum 0x5e2c (incorrect -> 0x25a4), seq 100, ack 1, win 209, options [nop,nop,TS val 583699596 ecr 4068505005], length 0 20:36:33.586154 IP (tos 0x0, ttl 64, id 47169, offset 0, flags [DF], proto TCP (6), length 52) 169.254.169.254.http > kubernetes-cluster-f4anhakchpkj-minion-0.staging.planethoster.ne.56896: Flags [.], cksum 0x91fc (correct), seq 26695, ack 101, win 53, options [nop,nop,TS val 4068516266 ecr 583699596], length 0 So the instance does receive packets as a reply to the curl, but they’re empty. I have tested on both Fedora-Atomic-27-20180212.2.x86_64.qcow2 and Fedora-AtomicHost-28-20180425.0.x86_64.qcow2 images and reproduced the issue both times. Nova-api and neutron-metadata logs show http 200 for each of my requests to nova. What could be blocking the user-data from reaching the instance in this case? Is there a way to set magnum-created intances to use configdrive instead? I must also specify that user-data works on all regular instances. It’s only affecting instances created by magnum. Best regards, Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.