I found the issue it was kolla-ansible role had some typo in jinja2 template 

https://bugs.launchpad.net/kolla-ansible/+bug/2107983 

On Tue, Apr 22, 2025 at 7:23 AM Sean Mooney <smooney@redhat.com> wrote:

On 22/04/2025 05:30, Michał Nasiadka wrote:
> Thank you Satish, can you please raise those bugs in Launchpad with
> more extensive logs and your configuration?
>
> It’s still not rc1, so you can expect some major bugs.
>
> Best regards
>
> Michał Nasiadka
> mnasiadka@gmail.com
>
>
> W dniu wt., 22 kwi 2025 o 05:40 Satish Patel <satish.txt@gmail.com>
> napisał(a):
>
>     Here is the output of debug mode related TASK [nova-cell : Waiting
>     for nova-compute services to register themselves]
>
>     TASK [nova-cell : Waiting for nova-compute services to register
>     themselves]
>     ******************************************************************************************************************************************
>     task path:
>     /opt/xenv/share/kolla-ansible/ansible/roles/nova-cell/tasks/wait_discover_computes.yml:8
>
>     The full traceback is:
>     Traceback (most recent call last):
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/executor/task_executor.py",
>     line 165, in run
>         res = self._execute()
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/executor/task_executor.py",
>     line 766, in _execute
>         if cond.evaluate_conditional(templar, vars_copy):
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/playbook/conditional.py",
>     line 59, in evaluate_conditional
>         return self.evaluate_conditional_with_result(templar, all_vars)[0]
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/playbook/conditional.py",
>     line 73, in evaluate_conditional_with_result
>         res = self._check_conditional(conditional, templar, all_vars)
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/playbook/conditional.py",
>     line 109, in _check_conditional
>         return templar.template(
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/template/__init__.py",
>     line 718, in template
>         result = self.do_template(
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/template/__init__.py",
>     line 964, in do_template
>         res = myenv.concat(rf)
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/template/native_helpers.py",
>     line 81, in ansible_eval_concat
>         head = list(islice(nodes, 2))
>       File "<template>", line 37, in root
>       File
>     "/opt/xenv/lib/python3.10/site-packages/ansible/template/__init__.py",
>     line 294, in wrapper
>         ret = func(*args, **kwargs)
>       File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
>         return _default_decoder.decode(s)
>       File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
>         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
>       File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
>         raise JSONDecodeError("Expecting value", s, err.value) from None
>
>     json.decoder.JSONDecodeError: Expecting value: line 1 column 1
>     (char 0)
>     fatal: [vsan2]: FAILED! => {
>         "msg": "Unexpected failure during module execution: Expecting
>     value: line 1 column 1 (char 0)",
>         "stdout": ""
>     }
>

are you trying to run this on top of xen hypervisor?

there are issue with newer kernels and old version of xen related to how
xen virtualization the kernel.

i ask because /opt/xenv/share/kolla-ansible looks odd.

>
>
>
>     On Mon, Apr 21, 2025 at 11:24 PM Satish Patel
>     <satish.txt@gmail.com> wrote:
>
>         Folks,
>
>         For the experiment, I am deploying epoxy in the lab and
>         getting some nova related issues. not sure if they are bugs or
>         just config tuning.
>
>         #BUG 1
>
>         placement_api container failed to start because of uWSIG
>         provider issue. I have changed to apache that fixed the issue.
>
>         placement_api_backend: "apache"
>         placement_wsgi_provider: "apache"
>

please include logs when reporting this but this is likely a kolla
config issue, it might be a placement bug but we deploy under uwsgi in
devstack jobs
so we know placement work with uwsgi but kolla and devstack might deploy
things differently.

>
>         # BUG 2 or Failed it this point.
>
>         TASK [nova-cell : Waiting for nova-compute services to
>         register themselves]
>         ******************************************************************************************************************************************
>         An exception occurred during task execution. To see the full
>         traceback, use -vvv. The error was:
>         json.decoder.JSONDecodeError: Expecting value: line 1 column 1
>         (char 0)
>         fatal: [vsan2]: FAILED! => {"msg": "Unexpected failure during
>         module execution: Expecting value: line 1 column 1 (char 0)",
>         "stdout": ""}
>
>
>         # BUG 3
>
>         inside the nova-computer container I didn't see any service or
>         process related privsep-helper --config-file
>         /etc/nova/nova.conf is this something new or missed by bug or
>         config?
>
>         (xenv) root@vsan2:/etc/kolla# docker exec -it -u0 nova_compute
>         bash
>         (nova-compute)[root@vsan2 /]# ps aux
>         USER         PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME
>         COMMAND
>         nova           1  0.0  0.0   2644   956 ?        Ss   08:28  
>         0:00 dumb-init --single-child -- kolla_start
>         nova           7  1.0  0.0 1883120 188576 ?      Sl   08:28  
>         0:15 /var/lib/kolla/venv/bin/python3
>         /var/lib/kolla/venv/bin/nova-compute
>         root         611  1.0  0.0   7636  4128 pts/2    Ss   08:52  
>         0:00 bash
>         root         627  0.0  0.0  10072  1580 pts/2    R+   08:52  
>         0:00 ps aux
>
This is normal. you will only see privsep after a call that need higher
privalages is executed, so unless you have created a new vms or
something like that

it wont have a privsep process intially on startup.