Hi I was trying the Software Deployment/Structured deployment of heat. I somehow can never get the signaling to work I see that authentication is happening but I don’t see a POST from the VM as a result stack is stuck in CREATE_IN_PROGRESS I see this message in my heat api cfn log which seems to suggest authentication is successful but it does not seem to POST. Have included debug output from VM and also the sample heat template I used. Don’t know if the template is correct as I referred some online examples to build it 2019-10-05 10:30:00.908 7 INFO heat.api.aws.ec2token [-] Checking AWS credentials.. 2019-10-05 10:30:00.909 7 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone. 2019-10-05 10:30:00.910 7 INFO heat.api.aws.ec2token [-] Authenticating with http://10.10.173.9:5000/v3/ec2tokens 2019-10-05 10:30:01.315 7 INFO heat.api.aws.ec2token [-] AWS authentication successful. 2019-10-05 10:30:02.326 7 INFO eventlet.wsgi.server [req-506f22c6-4062-4a84-8e85-40317a4099ed - adccd09df89e4b71b0a42f462679e75a-b1c6eb69-3877-466b-b00d-03dc051 - 0ecadd4762a34de1ac08508db4d3caa9 0ecadd4762a34de1ac08508db4d3caa9] 10.11.59.36,10.10.173.9 - - [05/Oct/2019 10:30:02] "GET /v1/?SignatureVersion=2&AWSAccessKeyId=f7874ac9898248edaae53511230534a4&StackName=test_stack&SignatureMethod=HmacSHA256&Signature=c03Q7Hb35q9tPPuYOv6YByn5YekF96p2s5zx36sX7x4%3D&Action=DescribeStackResource&LogicalResourceId=sig-vm-1 HTTP/1.1" 200 4669 1.418045 Some debugging output from my VM: [root@sig-vm-1 fedora]# sudo os-collect-config --force --one-time --debug /var/lib/os-collect-config/local-data not found. Skipping [2019-10-05 17:32:47,058] (os-refresh-config) [INFO] Starting phase pre-configure dib-run-parts Sat Oct 5 17:32:47 UTC 2019 ----------------------- PROFILING ----------------------- dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Target: pre-configure.d dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Script Seconds dib-run-parts Sat Oct 5 17:32:47 UTC 2019 --------------------------------------- ---------- dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 --------------------- END PROFILING --------------------- [2019-10-05 17:32:47,091] (os-refresh-config) [INFO] Completed phase pre-configure [2019-10-05 17:32:47,092] (os-refresh-config) [INFO] Starting phase configure dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Running /usr/libexec/os-refresh-config/configure.d/20-os-apply-config [2019/10/05 05:32:47 PM] [INFO] writing /var/run/heat-config/heat-config [2019/10/05 05:32:47 PM] [INFO] writing /etc/os-collect-config.conf [2019/10/05 05:32:47 PM] [INFO] success dib-run-parts Sat Oct 5 17:32:47 UTC 2019 20-os-apply-config completed dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Running /usr/libexec/os-refresh-config/configure.d/50-heat-config-docker-compose dib-run-parts Sat Oct 5 17:32:47 UTC 2019 50-heat-config-docker-compose completed dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Running /usr/libexec/os-refresh-config/configure.d/50-heat-config-kubelet dib-run-parts Sat Oct 5 17:32:47 UTC 2019 50-heat-config-kubelet completed dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Running /usr/libexec/os-refresh-config/configure.d/55-heat-config [2019-10-05 17:32:47,724] (heat-config) [ERROR] Skipping group Heat::Ungrouped with no hook script None [2019-10-05 17:32:47,724] (heat-config) [ERROR] Skipping group Heat::Ungrouped with no hook script None dib-run-parts Sat Oct 5 17:32:47 UTC 2019 55-heat-config completed dib-run-parts Sat Oct 5 17:32:47 UTC 2019 ----------------------- PROFILING ----------------------- dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Target: configure.d dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Script Seconds dib-run-parts Sat Oct 5 17:32:47 UTC 2019 --------------------------------------- ---------- dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 20-os-apply-config 0.345 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 50-heat-config-docker-compose 0.064 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 50-heat-config-kubelet 0.134 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 55-heat-config 0.065 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 dib-run-parts Sat Oct 5 17:32:47 UTC 2019 --------------------- END PROFILING --------------------- [2019-10-05 17:32:47,787] (os-refresh-config) [INFO] Completed phase configure [2019-10-05 17:32:47,787] (os-refresh-config) [INFO] Starting phase post-configure dib-run-parts Sat Oct 5 17:32:47 UTC 2019 Running /usr/libexec/os-refresh-config/post-configure.d/99-refresh-completed ++ os-apply-config --key completion-handle --type raw --key-default '' + HANDLE= ++ os-apply-config --key completion-signal --type raw --key-default '' + SIGNAL= ++ os-apply-config --key instance-id --type raw --key-default '' + ID=i-0000000d + '[' -n i-0000000d ']' + '[' -n '' ']' + '[' -n '' ']' ++ os-apply-config --key deployments --type raw --key-default '' ++ jq -r 'map(select(.group == "os-apply-config") | select(.inputs[].name == "deploy_signal_id") | .id + (.inputs | map(select(.name == "deploy_signal_id")) | .[].value)) | .[]' + DEPLOYMENTS= + DEPLOYED_DIR=/var/lib/os-apply-config-deployments/deployed + '[' '!' -d /var/lib/os-apply-config-deployments/deployed ']' dib-run-parts Sat Oct 5 17:32:49 UTC 2019 99-refresh-completed completed dib-run-parts Sat Oct 5 17:32:49 UTC 2019 ----------------------- PROFILING ----------------------- dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 Target: post-configure.d dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 Script Seconds dib-run-parts Sat Oct 5 17:32:49 UTC 2019 --------------------------------------- ---------- dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 99-refresh-completed 1.206 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 --------------------- END PROFILING --------------------- [2019-10-05 17:32:49,041] (os-refresh-config) [INFO] Completed phase post-configure [2019-10-05 17:32:49,042] (os-refresh-config) [INFO] Starting phase migration dib-run-parts Sat Oct 5 17:32:49 UTC 2019 ----------------------- PROFILING ----------------------- dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 Target: migration.d dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 Script Seconds dib-run-parts Sat Oct 5 17:32:49 UTC 2019 --------------------------------------- ---------- dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 dib-run-parts Sat Oct 5 17:32:49 UTC 2019 --------------------- END PROFILING --------------------- [2019-10-05 17:32:49,073] (os-refresh-config) [INFO] Completed phase migration onfig]# cat /var/run/heat-config/heat-config [{"inputs": [{"type": "String", "name": "foo", "value": "fu"}, {"type": "String", "name": "bar", "value": "barmy"}, {"type": "String", "name": "deploy_server_id", "value": "226ed96d-2335-436e-9707-95af73041e5f", "description": "ID of the server being deployed to"}, {"type": "String", "name": "deploy_action", "value": "CREATE", "description": "Name of the current action being deployed"}, {"type": "String", "name": "deploy_stack_id", "value": "test_stack/b1c6eb69-3877-466b-b00d-03dc051d1893", "description": "ID of the stack this deployment belongs to"}, {"type": "String", "name": "deploy_resource_name", "value": "other_deployment", "description": "Name of this deployment resource in the stack"}, {"type": "String", "name": "deploy_signal_transport", "value": "CFN_SIGNAL", "description": "How the server should signal to heat with the deployment output values."}, {"type": "String", "name": "deploy_signal_id", "value": "http://172.29.85.87:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Aadccd09df89e4b71b0a42f462679e75a%3Astacks/test_stack/b1c6eb69-3877-466b-b00d-03dc051d1893/resources/other_deployment?Timestamp=2019-10-05T01%3A11%3A46Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=28a09f5d996240b8b4a117ecb0e0142b&SignatureVersion=2&Signature=IqXbRf9MzJ%2FnzqM7CLNAsR3BiwmaaHyWQspegxYc3D8%3D", "description": "ID of signal to use for signaling output values"}, {"type": "String", "name": "deploy_signal_verb", "value": "POST", "description": "HTTP verb to use for signaling outputvalues"}], "group": "Heat::Ungrouped", "name": "test_stack-config-bmekpj67pq6p", "outputs": [], "creation_time": "2019-10-05T01:14:31Z", "options": {}, "config": {"config_value_foo": "fu", "config_value_bar": "barmy"}, "id": "5c404619-ce79-48cd-b001-00ac6ff4f4e8"}, {"inputs": [{"type": "String", "name": "foo", "value": "fooooo"}, {"type": "String", "name": "bar", "value": "baaaaa"}, {"type": "String", "name": "deploy_server_id", "value": "226ed96d-2335-436e-9707-95af73041e5f", "description": "ID of the server being deployed to"}, {"type": "String", "name": "deploy_action", "value": "CREATE", "description": "Name of the current action being deployed"}, {"type": "String", "name": "deploy_stack_id", "value": "test_stack/b1c6eb69-3877-466b-b00d-03dc051d1893", "description": "ID of the stack this deployment belongs to"}, {"type": "String", "name": "deploy_resource_name", "value": "deployment", "description": "Name of this deployment resource in the stack"}, {"type": "String", "name": "deploy_signal_transport", "value": "CFN_SIGNAL", "description": "How the server should signal to heat with the deployment output values."}, {"type": "String", "name": "deploy_signal_id", "value": "http://172.29.85.87:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3Aadccd09df89e4b71b0a42f462679e75a%3Astacks/test_stack/b1c6eb69-3877-466b-b00d-03dc051d1893/resources/deployment?Timestamp=2019-10-05T01%3A11%3A46Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=4c3d718796e0452ea94f2ce8dc6973ef&SignatureVersion=2&Signature=rxtSBNUSF%2FEXn9wvVK4XMU%2F1RzXVDGILtZr1hmkl7gg%3D", "description": "ID of signal to use for signaling output values"}, {"type": "String", "name": "deploy_signal_verb", "value": "POST", "description": "HTTP verb to use for signaling outputvalues"}], "group": "Heat::Ungrouped", "name": "test_stack-config-bmekpj67pq6p", "outputs": [], "creation_time": "2019-10-05T01:14:31Z", "options": {}, "config": {"config_value_foo": "fooooo", "config_value_bar": "baaaaa"}, "id": "f4dea0c1-73c9-4ce4-aa04-c76ef9b08859"}][root@sig-vm-1 heat-config]# [root@sig-vm-1 heat-config]# cat /etc/os-collect-config.conf [DEFAULT] command = os-refresh-config collectors = ec2 collectors = cfn collectors = local [cfn] metadata_url = http://172.29.85.87:8000/v1/ stack_name = test_stack secret_access_key = npa^GWsPtbRL7D*MYObOI*kV0i1yqKOG access_key_id = f7874ac9898248edaae53511230534a4 path = sig-vm-1.Metadata Here is my basic sample temple heat_template_version: 2013-05-23 description: > This template demonstrates how to use OS::Heat::StructuredDeployment to override substitute get_input placeholders defined in OS::Heat::StructuredConfig config. As there is no hook on the server to act on the configuration data, these deployment resource will perform no actual configuration. parameters: flavor: type: string default: 'a061cb6c-99e7-4bdb-93e4-f0037ee3e947' image: type: string default: 3be29d9f-2ce6-4b95-b80c-0dbca7acfdfe public_net_id: type: string default: 67ae0e17-6258-4fb6-8b9b-0f29f6adb9db private_net_id: type: string description: Private network id default: 995fc046-1c58-468a-b81c-e42c06fc8966 private_subnet_id: type: string description: Private subnet id default: 7598c805-3a9b-4c27-be5b-dca4d89f058c password: type: string description: SSH password default: lab123 resources: the_sg: type: OS::Neutron::SecurityGroup properties: name: the_sg description: Ping and SSH rules: - protocol: icmp - protocol: tcp port_range_min: 22 port_range_max: 22 config: type: OS::Heat::StructuredConfig properties: config: config_value_foo: {get_input: foo} config_value_bar: {get_input: bar} deployment: type: OS::Heat::StructuredDeployment properties: signal_transport: CFN_SIGNAL config: get_resource: config server: get_resource: sig-vm-1 input_values: foo: fooooo bar: baaaaa other_deployment: type: OS::Heat::StructuredDeployment properties: signal_transport: CFN_SIGNAL config: get_resource: config server: get_resource: sig-vm-1 input_values: foo: fu bar: barmy server1_port0: type: OS::Neutron::Port properties: network_id: { get_param: private_net_id } security_groups: - default fixed_ips: - subnet_id: { get_param: private_subnet_id } server1_public: type: OS::Neutron::FloatingIP properties: floating_network_id: { get_param: public_net_id } port_id: { get_resource: server1_port0 } sig-vm-1: type: OS::Nova::Server properties: name: sig-vm-1 image: { get_param: image } flavor: { get_param: flavor } networks: - port: { get_resource: server1_port0 } user_data_format: SOFTWARE_CONFIG user_data: get_resource: cloud_config cloud_config: type: OS::Heat::CloudConfig properties: cloud_config: password: { get_param: password } chpasswd: { expire: False } ssh_pwauth: True