[kolla] Debugging with kolla-ansible
Heya, I'm chasing a bug at the moment, and have been able to recreate it with a stock kolla-ansible install. The next step is to add more debugging to the OpenStack code to try and chase down what's happening. Before I go off and do something wildly bonkers, does anyone have a nice way of overriding locally the container image that kolla is using for a given container? The best I've come up with at the moment is something like: - copy the contents of the container out to a directory on the host node - delete the docker container - create a new container which mimics the previous container (docker inspect and some muttering) and have that container mount the copied out stuff as a volume I considered just snapshotting the image being used by the current container, but I want a faster edit cycle than edit, snapshot, start provides. Thoughts? Michael
On Sun, Feb 3, 2019, 9:17 PM Michael Still <mikal@stillhq.com wrote:
Heya,
I'm chasing a bug at the moment, and have been able to recreate it with a stock kolla-ansible install. The next step is to add more debugging to the OpenStack code to try and chase down what's happening.
Before I go off and do something wildly bonkers, does anyone have a nice way of overriding locally the container image that kolla is using for a given container?
The best I've come up with at the moment is something like:
- copy the contents of the container out to a directory on the host node - delete the docker container - create a new container which mimics the previous container (docker inspect and some muttering) and have that container mount the copied out stuff as a volume
I considered just snapshotting the image being used by the current container, but I want a faster edit cycle than edit, snapshot, start provides.
Thoughts? Michael
Easiest way would be to deploy from a local registry. You can pull everything from docker hub and just use kolla-build to build and push the ones you're working on. Then just delete the image from wherever it's running, run a deploy with --tags of the project you're messing with, and it'll deploy the new image, or increment the docker tag when you push it and run upgrade. If I'm missing something and oversimplifying, let me know :). -Erik
That sounds interesting... So if I only wanted to redeploy say the ironic_neutron_agent container, how would I do that with a tag? Its not immediately obvious to me where the command line for docker comes from in the ansible. Is that just in ansible/roles/neutron/defaults/main.yml ? If so, I could tweak the container definition for the container I want to hack with the get its code from a volume, and then redeploy just that one container, yes? Thanks for your help! Michael On Mon, Feb 4, 2019 at 1:36 PM Erik McCormick <emccormick@cirrusseven.com> wrote:
On Sun, Feb 3, 2019, 9:17 PM Michael Still <mikal@stillhq.com wrote:
Heya,
I'm chasing a bug at the moment, and have been able to recreate it with a stock kolla-ansible install. The next step is to add more debugging to the OpenStack code to try and chase down what's happening.
Before I go off and do something wildly bonkers, does anyone have a nice way of overriding locally the container image that kolla is using for a given container?
The best I've come up with at the moment is something like:
- copy the contents of the container out to a directory on the host node - delete the docker container - create a new container which mimics the previous container (docker inspect and some muttering) and have that container mount the copied out stuff as a volume
I considered just snapshotting the image being used by the current container, but I want a faster edit cycle than edit, snapshot, start provides.
Thoughts? Michael
Easiest way would be to deploy from a local registry. You can pull everything from docker hub and just use kolla-build to build and push the ones you're working on.
Then just delete the image from wherever it's running, run a deploy with --tags of the project you're messing with, and it'll deploy the new image, or increment the docker tag when you push it and run upgrade.
If I'm missing something and oversimplifying, let me know :).
-Erik
Sorry for the delay. I didn't want to try and write this on my phone... On Sun, Feb 3, 2019, 9:39 PM Michael Still <mikal@stillhq.com wrote:
That sounds interesting... So if I only wanted to redeploy say the ironic_neutron_agent container, how would I do that with a tag?
To roll it out, update it's config, or upgrade to a new ticker tag, you'd just do kolla-ansible --tags neutron deploy | reconfigure | upgrade I don't think its granular enough to do just the agent and I'm not sure you'd want to anyway.
Its not immediately obvious to me where the command line for docker comes from in the ansible. Is that just in ansible/roles/neutron/defaults/main.yml ? If so, I could tweak the container definition for the container I want to hack with the get its code from a volume, and then redeploy just that one container, yes?
I suppose that's one way to go about quick hacks. You could add a new volume in that main.yml and then modify things in it. I think that would get messy though. There aren't any volume definitions explicitly for that container, so you'd have to add a whole section in there for it and I don't know what other side effects that might have. The slow but safe way to do it would be to point Kolla at your feature branch and rebuild the image each time you want to test a new patch set. In kolla-build.conf do something like: [neutron-base-plugin-networking-baremetal] type = url location = https://github.com/openstack/networking-baremetal.git reference = tonys-hacks then something like kolla-build --config-file /etc/kolla/kolla-build.conf --base centos --type source --push --registry localhost:5000 --logs-dir /tmp ironic-neutron-agent The really dirty but useful way to test small changes would be to just push them into the container with 'docker cp' and restart the container. Note that this will not work for config changes as those files get clobbered at startup, but for hacking the actual python bits, it'll do. Hope that's what you're looking for. If you drop by #opensatck-kolla during US daylight hours you might get more suggestions from Eduardo or one of the actual project devs. They probably have fancier methods. Cheers, Erik
Thanks for your help!
Michael
On Mon, Feb 4, 2019 at 1:36 PM Erik McCormick <emccormick@cirrusseven.com> wrote:
On Sun, Feb 3, 2019, 9:17 PM Michael Still <mikal@stillhq.com wrote:
Heya,
I'm chasing a bug at the moment, and have been able to recreate it with a stock kolla-ansible install. The next step is to add more debugging to the OpenStack code to try and chase down what's happening.
Before I go off and do something wildly bonkers, does anyone have a nice way of overriding locally the container image that kolla is using for a given container?
The best I've come up with at the moment is something like:
- copy the contents of the container out to a directory on the host node - delete the docker container - create a new container which mimics the previous container (docker inspect and some muttering) and have that container mount the copied out stuff as a volume
I considered just snapshotting the image being used by the current container, but I want a faster edit cycle than edit, snapshot, start provides.
Thoughts? Michael
Easiest way would be to deploy from a local registry. You can pull everything from docker hub and just use kolla-build to build and push the ones you're working on.
Then just delete the image from wherever it's running, run a deploy with --tags of the project you're messing with, and it'll deploy the new image, or increment the docker tag when you push it and run upgrade.
If I'm missing something and oversimplifying, let me know :).
-Erik
On Mon, Feb 4, 2019 at 4:01 PM Erik McCormick <emccormick@cirrusseven.com> wrote: [snip detailed helpful stuff] The really dirty but useful way to test small changes would be to just push
them into the container with 'docker cp' and restart the container. Note that this will not work for config changes as those files get clobbered at startup, but for hacking the actual python bits, it'll do.
This was news to me to be honest. I had assumed the container filesystem got reset on process restart, but you're right and that's not true. So, editing files in the container works for my current needs. Thanks heaps! Michael
Hi Michael, You could use a custom image and change the image definition in ansible, ie for define a different image for neutron_server you would add a variable in globals.yml like: neutron_server_image_full: "registry/repo/image_name:mytag:" If what you are debugin is openstack code, you could use kolla dev mode, where you can change git code locally and mount the code into the python path https://docs.openstack.org/kolla-ansible/latest/contributor/kolla-for-openst... Regards El lun., 4 feb. 2019 a las 7:16, Michael Still (<mikal@stillhq.com>) escribió:
On Mon, Feb 4, 2019 at 4:01 PM Erik McCormick <emccormick@cirrusseven.com> wrote:
[snip detailed helpful stuff]
The really dirty but useful way to test small changes would be to just
push them into the container with 'docker cp' and restart the container. Note that this will not work for config changes as those files get clobbered at startup, but for hacking the actual python bits, it'll do.
This was news to me to be honest. I had assumed the container filesystem got reset on process restart, but you're right and that's not true. So, editing files in the container works for my current needs.
Thanks heaps!
Michael
On Mon, 4 Feb 2019 at 08:25, Eduardo Gonzalez <dabarren@gmail.com> wrote:
Hi Michael,
You could use a custom image and change the image definition in ansible, ie for define a different image for neutron_server you would add a variable in globals.yml like:
neutron_server_image_full: "registry/repo/image_name:mytag:"
If what you are debugin is openstack code, you could use kolla dev mode, where you can change git code locally and mount the code into the python path https://docs.openstack.org/kolla-ansible/latest/contributor/kolla-for-openst...
Regards
Just a warning: I have recently had issues with dev mode because it does not do a pip install, but mounts the source code into the site-packages /<python>/<package> directory, if there are new source files these will not be included in the package's file manifest. Also this won't affect any files outside of site-packages/<python>/<package>. I just raised a bug [1] on this. What I often do when developing in a tight-ish loop on a single host is something like this: docker exec -it <container> pip install -e git+https://<repo>#<egg> docker restart <container> You have to be careful, since if the service doesn't start, the container will fail to start, and docker exec won't work. At that point you need to delete the container and redeploy. [1] https://bugs.launchpad.net/kolla-ansible/+bug/1814515 Mark
participants (4)
-
Eduardo Gonzalez
-
Erik McCormick
-
Mark Goddard
-
Michael Still