Open Stack

HI Michael and thanks for your reply. Below please find my answers:

Yes the load balancer is healthy and the health check of the member 
endpoint in failing

Yes, none of the members have monitor_address or monitor_port configured 
to override the health check endpoint (below details of one of the members)

| Field               | Value
+---------------------+-------------------------------------------------------------------------------------------------------------------+
| address             | 192.168.1.99
| admin_state_up      | True
| created_at          | 2021-05-24T20:07:56
| id                  | 8d093783-79c9-4094-b3a2-8d31b1c4567f
| name                | 
member_0_k8s-c1-final-yrlns2q7qo2w-node-1_kube_service_f48f5572-0011-4b66-b9ab-63190d0c00b3_default_nginx-service
| operating_status    | ERROR
| project_id          | 35a0fa65de1741619709485c5f6d989b
| protocol_port       | 31765
| provisioning_status | ACTIVE
| subnet_id           | 19d73049-4bf7-455c-b278-3cba451102fb
| updated_at          | 2021-05-24T20:08:51
| weight              | 1
| monitor_port        | None
| monitor_address     | None
| backup              | False

The subnet ID shown in the above details for the member node are the 
same and the subnet ID assigned to this project as shown below:

| Field                | Value                                |
+----------------------+--------------------------------------+
| allocation_pools     | 192.168.1.10-192.168.1.254
| cidr                 | 192.168.1.0/24
| created_at           | 2021-03-24T20:14:03Z
| description          |
| dns_nameservers      | 8.8.4.4, 8.8.8.8
| dns_publish_fixed_ip | None
| enable_dhcp          | True
| gateway_ip           | 192.168.1.1
| host_routes          |
| id                   | 19d73049-4bf7-455c-b278-3cba451102fb
| ip_version           | 4
| ipv6_address_mode    | None
| ipv6_ra_mode         | None
| name                 | red-subnet-2
| network_id           | 9d2e17df-a93f-4709-941d-a6e8f98f5556
| prefix_length        | None
| project_id           | 35a0fa65de1741619709485c5f6d989b
| revision_number      | 1
| segment_id           | None
| service_types        |
| subnetpool_id        | None
| tags
| updated_at           | 2021-03-26T13:59:07Z

I can SSH on an instance unrelated to the issue and from within SSH into 
all 3 members without any issues. The members are part of a Kubernetes 
cluster (created using Magnum) and both subnet and network have been 
specified during the cluster creation.

the issue seems to be coming from the nodeport which is created by 
default and that should proxy the LoadBalancer requests to the clusterIP 
as I do not get any reply when querying the port with curl and since the 
health manager uses that port it is also failing

I do not believe that the issue is actually from Octavia but from Magnum 
from what I can see and the nodeport functionality that gets created to 
have the amphora instance reach the nodeports

Thanks in advance

On 25/05/2021 02:12, Michael Johnson wrote:
> Hi Luke,
>
>  From the snippet you provided, it looks like the load balancer is
> healthy and working as expected, but the health check of the member
> endpoint is failing.
>
> I also see that the health monitor is configured for a type of TCP and
> the members are configured for port 31765. This assumes that the
> members do not have "monitor_address" or "monitor_port" configured to
> override the health check endpoint.
>
> One thing to check is the subnet_id that was configured on the members
> in Octavia. The members (192.168.1.99:31765, etc.) must be reachable
> from that subnet_id. If no subnet_id was used when creating the
> members, it will use the VIP subnet of the load balancer.
>
> Sometimes users forget to specify the subnet_id, that a member should
> be reachable from, when creating the member on the load balancer.
>
> Michael
>
> On Mon, May 24, 2021 at 1:48 PM Luke Camilleri
> <luke.camilleri at zylacomputing.com> wrote:
>> I have configured magnum with a service type of LoadBalancer and it is
>> successfully deploying an external LoadBalancer via Octavia. The problem
>> is that I cannot reach the public IP address but can see the entries in
>> the haproxy.log on the amphora instance and the log shows <NOSRV> 0 SC--
>> at the end of each entry when it is being accessed (via a browser for
>> example).
>>
>> So the Octavia part seems to be fine, the config shows the correct LB
>> --> listener --> pool --> member and the nodeports that the service
>> should be listening on (I am assuming that the same nodeports are also
>> used for the healthchecks)
>>
>> The haproxy.cfg in the amphora instance shows the below in the pool
>> members section:
>>
>>      server 43834d2f-4e22-4065-b448-ddf0713f2ced 192.168.1.191:31765
>> weight 1 check inter 60s fall 3 rise 3
>>      server 3a733c48-24dd-426e-8394-699a908121ee 192.168.1.36:31765
>> weight 1 check inter 60s fall 3 rise 3
>>      server 8d093783-79c9-4094-b3a2-8d31b1c4567f 192.168.1.99:31765
>> weight 1 check inter 60s fall 3 rise 3
>>
>> and the status of the pool's members (# openstack loadbalancer member
>> list) is as follows:
>>
>> +------------------------------------------------------------+---------------------------+-----------------+-----------------------+---------------------------+
>> | id     | provisioning_status | address       |     protocol_port |
>> operating_status |
>>
>> 8d093783-79c9-4094-b3a2-8d31b1c4567f      ACTIVE     192.168.1.99
>> 31765                 ERROR
>> 43834d2f-4e22-4065-b448-ddf0713f2ced        ACTIVE     192.168.1.191
>> 31765                 ERROR
>> 3a733c48-24dd-426e-8394-699a908121ee     ACTIVE     192.168.1.36
>> 31765                 ERROR
>>
>>   From the below loadbalancer healthmonitor show command I can see that
>> the health checks are being done via TCP on the same port and can
>> confirm that the security group allows the nodeports range (ALLOW IPv4
>> 30000-32767/tcp from 0.0.0.0/0)
>>
>> +---------------------+--------------------------------------+
>> | Field                             | Value
>> +---------------------+--------------------------------------+
>> | provisioning_status     | ACTIVE
>> | type                            | TCP
>> | id                                | 86604638-27db-47d2-ad9c-0594564a44be
>> | operating_status        | ONLINE
>> +---------------------+--------------------------------------+
>>
>> $ kubectl describe service nginx-service
>> Name:                     nginx-service
>> Namespace:                default
>> Labels:                   <none>
>> Annotations:              <none>
>> Selector:                 app=nginx
>> Type:                     LoadBalancer
>> IP Families:              <none>
>> IP:                       10.254.255.232
>> IPs:                      <none>
>> LoadBalancer Ingress:     185.89.239.217
>> Port:                     <unset>  80/TCP
>> TargetPort:               8080/TCP
>> NodePort:                 <unset>  31765/TCP
>> Endpoints: 10.100.4.11:8080,10.100.4.12:8080,10.100.5.10:8080
>> Session Affinity:         None
>> External Traffic Policy:  Cluster
>> Events:
>>     Type    Reason                Age   From                Message
>>     ----    ------                ----  ----                -------
>>     Normal  EnsuringLoadBalancer  29m   service-controller  Ensuring load
>> balancer
>>     Normal  EnsuredLoadBalancer   28m   service-controller  Ensured load
>> balancer
>>
>> The Kubernetes deployment file:
>>
>> apiVersion: apps/v1
>> kind: Deployment
>> metadata:
>>     name: nginx-deployment
>>     labels:
>>       app: nginx
>> spec:
>>     replicas: 3
>>     selector:
>>       matchLabels:
>>         app: nginx
>>     template:
>>       metadata:
>>         labels:
>>           app: nginx
>>       spec:
>>         containers:
>>         - name: nginx
>>           image: nginx:latest
>>           ports:
>>           - containerPort: 8080
>> ---
>> apiVersion: v1
>> kind: Service
>> metadata:
>>     name: nginx-service
>> spec:
>>     selector:
>>       app: nginx
>>     type: LoadBalancer
>>     ports:
>>     - protocol: TCP
>>       port: 80
>>       targetPort: 8080
>>
>> Does ayone have any pointers as to why the amphora is not able to reach
>> the nodeports of the kubernetes workers?
>>
>> Thanks in advance
>>