<div dir="ltr">Artur, Shura and Dima,<div><div>I want close the action item which I've got on the Fuel IRC meeting Nov 26th.</div><div>BP for investigation how we can(not) run HA tests on nodepool was created:</div><div><a href="https://blueprints.launchpad.net/fuel/+spec/qa-rnd-ha-on-nodepool">https://blueprints.launchpad.net/fuel/+spec/qa-rnd-ha-on-nodepool</a></div><div><br></div><div>Milestone target is "next" cause we are close to FF for 8.0 version, </div><div>but BP is open for comments and suggestions.</div></div><div><br></div><div><br></div><div>Nastya.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 25, 2015 at 4:55 PM, Artur Kaszuba <span dir="ltr"><<a href="mailto:akaszuba@mirantis.com" target="_blank">akaszuba@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi All<br>
<br>
W dniu 25.11.2015 o 14:00, Aleksandra Fedorova pisze:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
...<span class=""><br>
Thus the test layout as i see it should be the following:<br>
<br>
- one "manager vm" which will be responsible for the main test flow.<br>
On this vm we will:<br>
- build fuel-library package,<br>
- generate a repository,<br>
- get fuel-qa code,<br>
- run the actual test using based on job parameters,<br>
- collect test result and artifacts.<br>
- one vm with installed Fuel,<br>
- several bootstrapped nodes according to the test configuration -<br>
here probably we can switch from using nodes which are bootstrapped<br>
with Fuel to nodes which are just the most generic ones.<br>
<br>
As i got it from reading grenade job log [1], having manager vm is<br>
actually expected: there is a "main" node and list of subnodes which<br>
you can talk to from there. So this setup is not impossible, but<br>
requires support for requesting different types of nodes from<br>
nodepool. And we probably need to patch the Zuul_v3 spec in [2] to add<br>
this case to the format of request-accept calls?<br>
<br>
The other question here would be the network configuration: to<br>
function properly Fuel needs 5 (or 6?) different networks and we will<br>
need a way to configure them before we run the actual deployment. What<br>
are the options for configuring network on top of nodes provided by<br>
nodepool?<br>
<br>
<br>
</span></blockquote>
<br>
Some time ago i worked on similar solution, how to execute Fuel inside other openstack instances. During my tests i have found problem with network and passing vlans between openstack instances. This problem could also appear when we want to use nodepool as a source of virtual machines for deployment tests. I will try to show where my problem exist.<br>
<br>
As a underlying Openstack infrastructure was used Fuel 6.0 executed on HW nodes:<br>
- one controler node<br>
- 5 compute nodes<br>
- Neutron with GRE segmentation<br>
<br>
To reduce network issues all instances was executed on one compute node.<br>
<br>
Test assumes that we execute fuel master and slaves as a Openstack instances inside one dedicated tenant. Every test requires to create a few networks and instances, it was easier to manage networks and instances as a tenant group. At the end of the test we can just delete all tenant objects.<br>
<br>
Standard fuel installation requires to create few networks, i created:<br>
- admin<br>
- public<br>
- mgmt<br>
- storage<br>
<br>
In standard Fuel installations networks could be attached directly to interface or created as a VLAN on network interface. In this case i wanted to reduce problems with additional tagging/fragmentation and each of network was created as a separate neutron network with dedicated interface inside instance.<br>
This requires to create Neutron infrastructure with:<br>
- router connected to external network<br>
- 4 neutron networks<br>
- public and admin network connected to router<br>
<br>
Each of instance is connected to network by dedicated interface, it looks like this:<br>
<br>
+---------+<br>
| Neutron | +~~~~~~~~+<br>
+------| router |---|Internet|<br>
| +---------+ +~~~~~~~~+<br>
| |<br>
+---------+ +---------+ +---------+ +---------+<br>
| Neutron | | Neutron | | Neutron | | Neutron |<br>
| network | | network | | network | | network |<br>
| admin | | public | | mgmt | | storage |<br>
+---------+ +---------+ +---------+ +---------+<br>
| | | | |<br>
| +--------+ | | |<br>
| | | | |<br>
+-----------+ +------------------------+<br>
|Fuel Master| | Fuel slave X |<br>
+-----------+ +------------------------+<br>
<br>
In standard Openstack installation it is not possible to start DHCP server on instance, it is blocked by firewall. To solve this problem i changed neutron agent code to allow DHCP traffic.<br>
<br>
Shortcut from procedure used to install fuel inside openstack:<br>
- upload ISO Fuel 6.1 to glance<br>
- upload ISO iPXE to glance<br>
- create instance for fuel master server and install it<br>
- create 2 instances for fuel slave nodes and boot them from iPXE<br>
<br>
After those steps i got fuel installation with 2 nodes ready for deployment, like this:<br>
<br>
+------------------------------------------------------+<br>
| Hardware node |<br>
| +------------------------+ |<br>
| | Fuel slave 1 | |<br>
| | (KVM Instance) | |<br>
| +------------------------+ |<br>
| | | | | |<br>
| +--------+ | | | |<br>
| | | | | |<br>
| +---------+ +---------+ +---------+ +---------+ |<br>
| | Neutron | | Neutron | | Neutron | | Neutron | |<br>
| | network | | network | | network | | network | |<br>
| | admin | | public | | mgmt | | storage | |<br>
| | (OVS) | | (OVS) | | (OVS) | | (OVS) | |<br>
| +---------+ +---------+ +---------+ +---------+ |<br>
| | | | | | |<br>
| | +--------+ | | | |<br>
| | | | | | |<br>
| +-----------+ +------------------------+ |<br>
| |Fuel Master| | Fuel slave 2 | |<br>
| | (KVM) | | (KVM Instance) | |<br>
| +-----------+ +------------------------+ |<br>
| |<br>
+------------------------------------------------------+<br>
<br>
With this configuration it was possible to deploy new Openstack, but some type of configurations not work correctly. Problem appears when i deployed Openstack with VLAN separation, OSTF tests not passes.<br>
<br>
+------------------------------------------------------+<br>
| Hardware node |<br>
| +---------------------------------------+ |<br>
| | Fuel slave 1 +------------------+ | |<br>
| | (KVM Instance) |Openstack Instance| | |<br>
| | | TEST1 | | |<br>
| | | (KVM) | | |<br>
| | +------------------+ | |<br>
| | | | |<br>
| | +----------+ | |<br>
| | | OVS | | |<br>
| | | (VLAN) | | |<br>
| | +----------+ | |<br>
| | |tag:1000 | |<br>
| +---------------------------ETH---------+ |<br>
| | |<br>
| +----------+ |<br>
| | OVS | |<br>
| | (GRE) | |<br>
| +----------+ |<br>
| | |<br>
| +---------------------------ETH---------+ |<br>
| | Fuel slave 2 |tag:1000 | |<br>
| | (KVM Instance) +----------+ | |<br>
| | | OVS | | |<br>
| | | (VLAN) | | |<br>
| | +----------+ | |<br>
| | | | |<br>
| | +------------------+ | |<br>
| | |Openstack Instance| | |<br>
| | | TEST2 | | |<br>
| | | (KVM) | | |<br>
| | +------------------+ | |<br>
| +---------------------------------------+ |<br>
| |<br>
+------------------------------------------------------+<br>
<br>
Problem exists when we want to send packets between instances TEST1 and TEST2:<br>
- packet is sent from TEST1<br>
- OVS inside Fuel slave 1 needs to send this packet to other compute node inside correct tenant network, it will add VLAN tag and then will send it to the network interface<br>
- OVS on hardware node will get tagged frame on OVS interface which is not configured for tagging and OVS will drop it<br>
<br>
To start using nodepool instances in deployment tests we need to have sure that openstack used by nodepool allow to:<br>
- send responses from DHCP server installed on fuel master to slave instances, packets must pass by compute node firewall, in standard configuration responses from DHCP server are possible only from network node<br>
- we have solution which allow to pass tagged vlans between Openstack Instances, without this we cannot execute all kind of tests and scenarios<br>
<br>
Is it possible in actual Openstack infrastructure used by nodepool?<br>
Or maybe we could solve it in other way?<br>
<br>
--<br>
Artur Kaszuba<div class="HOEnZb"><div class="h5"><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>