Open Stack

Hi,

we are currently working on a new environment for our school to teach
some basic knowledge about OpenStack.
Our goal is to run OpenStack training-labs for each student on VMs
running on ESXi.

Therefor we've enabled nested virtualization (vhv.enable=TRUE) for the
VMs in vSphere. Then we basically do following on Ubuntu 18.04.2:
```
echo 1 > /sys/module/kvm/parameters/ignore_msr
# to avoid following error while building the base image: "This
usually means the libguestfs appliance failed to start or crashed."

sudo apt-get install -y qemu-kvm libvirt-bin libguestfs-tools
bridge-utils virt-manager

virsh pool-create-as --name default --type dir --target /var/lib/libvirt/images

wget http://tarballs.openstack.org/training-labs/dist/labs-stable-rocky.tgz
tar xzf labs-stable-rocky.tgz
cd labs/
```
We build the base image ubuntu-18.04-server-amd64 with `./st.py
--build basedisk --provider kvm` and then the cluster `./st.py --build
cluster --provider kvm`.
Everything works fine till...:

stacktrain.log:
```
12256 13:08:19.992 stacktrain.core.autostart INFO Start
autostart/04_setup_nova_compute.sh
12256 13:08:19.994 stacktrain.core.ssh DEBUG vm_ssh: ssh -q -i
/opt/oslab/labs/lib/osbash-ssh-keys/osbash_key -o
UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o
ConnectTimeout=20 -o ControlPath=none -p 22 osbash at 192.168.122.208
bash autostart/04_setup_nova_compute.sh && rm -vf
autostart/04_setup_nova_compute.sh
12256 13:08:19.994 stacktrain.core.ssh DEBUG Writing live log for ssh
call at /opt/oslab/labs/log/021_04_setup_nova_compute.auto.
12256 13:08:56.551 stacktrain.core.ssh ERROR ssh returned status 255.
12256 13:08:56.552 stacktrain.core.autostart ERROR Script failure:
04_setup_nova_compute
11272 13:08:57.110 stacktrain.core.autostart ERROR Script failed. Exiting.
```

ssh.log:
```
ssh -q -i /opt/oslab/labs/lib/osbash-ssh-keys/osbash_key -o
UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o
ConnectTimeout=20 -o ControlPath=none -p 22 osbash at 192.168.122.208
mkdir -p autostart
ssh -q -i /opt/oslab/labs/lib/osbash-ssh-keys/osbash_key -o
UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o
ConnectTimeout=20 -o ControlPath=none -p 22 osbash at 192.168.122.208
bash autostart/04_setup_nova_compute.sh && rm -vf
autostart/04_setup_nova_compute.sh
```

For me it looks like both VMs (controller + compute1) get the same IP
address (with `./st.py --build nodes_only --provider kvm`):
...
Your cluster nodes:
INFO VM name: compute1
INFO SSH login: ssh osbash at 192.168.122.47
INFO            (password: osbash)
INFO VM name: controller
INFO SSH login: ssh osbash at 192.168.122.47
INFO            (password: osbash)
INFO Dashboard: Assuming horizon is on controller VM.
INFO            http://192.168.122.47/horizon/
INFO            User  : myuser (password: myuser_user_pass)
INFO            User  : admin (password: admin_user_secret)
INFO Network: mgmt
INFO          Network address: 10.0.0.0
INFO Network: provider
INFO          Network address: 203.0.113.0
```

>From within the VMs:
```
# controller:
osbash at osbash:~$ ifconfig ens3
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.47  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::5054:ff:fed9:b165  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:d9:b1:65  txqueuelen 1000  (Ethernet)
...
osbash at osbash:~$ netplan ip leases ens3
# This is private data. Do not parse.
ADDRESS=192.168.122.47
NETMASK=255.255.255.0
ROUTER=192.168.122.1
SERVER_ADDRESS=192.168.122.1
NEXT_SERVER=192.168.122.1
BROADCAST=192.168.122.255
T1=1800
T2=3150
LIFETIME=3600
DNS=192.168.122.1
HOSTNAME=osbash
CLIENTID=ffb55e67ff00020000ab11bba91ddef33599c5

osbash at osbash:~$ cat /etc/machine-id
b8c7057a4c084e69aa3743960d41d5ce

# compute1:
osbash at compute1:~$ ifconfig ens3
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.47  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::5054:ff:fe17:b640  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:17:b6:40  txqueuelen 1000  (Ethernet)
...
osbash at compute1:~$ netplan ip leases ens3
# This is private data. Do not parse.
ADDRESS=192.168.122.47
NETMASK=255.255.255.0
ROUTER=192.168.122.1
SERVER_ADDRESS=192.168.122.1
NEXT_SERVER=192.168.122.1
BROADCAST=192.168.122.255
T1=1800
T2=3150
LIFETIME=3600
DNS=192.168.122.1
HOSTNAME=compute1
CLIENTID=ffb55e67ff00020000ab11bba91ddef33599c5

osbash at compute1:~$ cat /etc/machine-id
b8c7057a4c084e69aa3743960d41d5ce
```

I believe the VMs share the same IP address because of the same
machine-id (which is used as DUID:
http://manpages.ubuntu.com/manpages/bionic/man5/networkd.conf.5.html)

Am I wrong?
Thanks for your help,
Julian