Magnum Kubernetes openstack-cloud-controller-manager unable not resolve master node by DNS
Hi, i try to deploy a Kubernetes cluster with OpenStack Magnum but the openstack-cloud-controller-manager pod fails to resolve the master node hostname. Does magnum require further parameter to configure the DNS names of the master and minions? DNS resolution in the VMs works fine. Currently there is no Designate installed in the OpenStack setup. openstack coe cluster template create kubernetes-cluster-template1 \ --image Fedora-AtomicHost-29-20190429.0.x86_64 \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.kubernetes \ --flavor m1.kubernetes \ --coe kubernetes \ --volume-driver cinder \ --network-driver flannel \ --docker-volume-size 25 openstack coe cluster create kubernetes-cluster1 \ --cluster-template kubernetes-cluster-template1 \ --master-count 1 \ --node-count 2 \ --keypair mykey # kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE kube-system coredns-78df4bf8ff-mjp2c 0/1 Pending 0 36m <none> <none> <none> kube-system heapster-74f98f6489-tgtzl 0/1 Pending 0 36m <none> <none> <none> kube-system kube-dns-autoscaler-986c49747-wrvz4 0/1 Pending 0 36m <none> <none> <none> kube-system kubernetes-dashboard-54cb7b5997-sk5pj 0/1 Pending 0 36m <none> <none> <none> kube-system openstack-cloud-controller-manager-dgk64 0/1 CrashLoopBackOff 11 36m <none> kubernetes-cluster1-vulg5fz6hg2n-master-0 <none> # kubectl -n kube-system logs openstack-cloud-controller-manager-dgk64 Error from server: Get https://kubernetes-cluster1-vulg5fz6hg2n-master-0:10250/containerLogs/kube-s...: dial tcp: lookup kubernetes-cluster1-vulg5fz6hg2n-master-0 on 8.8.8.8:53: no such host BR Pawel
Try using the latest version, think there is an OCCM_TAG. Sent from my iPhone
On 7 May 2019, at 20:10, Pawel Konczalski <pawel.konczalski@everyware.ch> wrote:
Hi,
i try to deploy a Kubernetes cluster with OpenStack Magnum but the openstack-cloud-controller-manager pod fails to resolve the master node hostname.
Does magnum require further parameter to configure the DNS names of the master and minions? DNS resolution in the VMs works fine. Currently there is no Designate installed in the OpenStack setup.
openstack coe cluster template create kubernetes-cluster-template1 \ --image Fedora-AtomicHost-29-20190429.0.x86_64 \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.kubernetes \ --flavor m1.kubernetes \ --coe kubernetes \ --volume-driver cinder \ --network-driver flannel \ --docker-volume-size 25
openstack coe cluster create kubernetes-cluster1 \ --cluster-template kubernetes-cluster-template1 \ --master-count 1 \ --node-count 2 \ --keypair mykey
# kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE kube-system coredns-78df4bf8ff-mjp2c 0/1 Pending 0 36m <none> <none> <none> kube-system heapster-74f98f6489-tgtzl 0/1 Pending 0 36m <none> <none> <none> kube-system kube-dns-autoscaler-986c49747-wrvz4 0/1 Pending 0 36m <none> <none> <none> kube-system kubernetes-dashboard-54cb7b5997-sk5pj 0/1 Pending 0 36m <none> <none> <none> kube-system openstack-cloud-controller-manager-dgk64 0/1 CrashLoopBackOff 11 36m <none> kubernetes-cluster1-vulg5fz6hg2n-master-0 <none>
# kubectl -n kube-system logs openstack-cloud-controller-manager-dgk64 Error from server: Get https://kubernetes-cluster1-vulg5fz6hg2n-master-0:10250/containerLogs/kube-s...: dial tcp: lookup kubernetes-cluster1-vulg5fz6hg2n-master-0 on 8.8.8.8:53: no such host
BR
Pawel
Hi Bharat, i was able to deploy the Kubernetes cluster with Magnum after update / specify Kubernetes version with the "--labels kube_tag=v1.13.4" parameter. See: # kube_tag https://docs.openstack.org/magnum/latest/user/#kube-tag https://hub.docker.com/r/openstackmagnum/kubernetes-apiserver/tags/ # cloud_provider_tag https://docs.openstack.org/magnum/latest/user/#cloud-provider-tag https://hub.docker.com/r/k8scloudprovider/openstack-cloud-controller-manager... This may by related with this issue: https://github.com/kubernetes/cloud-provider-openstack/issues/280 # openstack coe cluster template create kubernetes-cluster-template \ --image "Fedora AtomicHost 29" \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.kubernetes \ --flavor m1.kubernetes \ --coe kubernetes \ --volume-driver cinder \ --network-driver flannel \ --docker-volume-size 25 \ --public \ --labels kube_tag=v1.13.4,cloud_provider_tag=1.13.1 # openstack coe cluster create kubernetes-cluster \ --cluster-template kubernetes-cluster-template \ --master-count 1 \ --node-count 2 \ --keypair mykey # kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-dcc6d487d-hxpgq 1/1 Running 0 7h55m 10.100.9.2 kubernetes-cluster7-sysemevhbq4i-minion-1 <none> <none> kube-system coredns-dcc6d487d-nkb9p 1/1 Running 0 7h57m 10.100.78.4 kubernetes-cluster7-sysemevhbq4i-minion-0 <none> <none> kube-system heapster-796547984d-6wgwp 1/1 Running 0 7h57m 10.100.78.2 kubernetes-cluster7-sysemevhbq4i-minion-0 <none> <none> kube-system kube-dns-autoscaler-7865df57cd-ln4cc 1/1 Running 0 7h57m 10.100.78.3 kubernetes-cluster7-sysemevhbq4i-minion-0 <none> <none> kube-system kubernetes-dashboard-f5496d66d-tdbvv 1/1 Running 0 7h57m 10.100.78.5 kubernetes-cluster7-sysemevhbq4i-minion-0 <none> <none> kube-system openstack-cloud-controller-manager-9s5wh 1/1 Running 3 7h57m 10.0.0.10 kubernetes-cluster7-sysemevhbq4i-master-0 <none> <none> Thank you Pawel Am 08.05.19 um 8:40 vorm. schrieb Bharat Kunwar:
Try using the latest version, think there is an OCCM_TAG.
Sent from my iPhone
On 7 May 2019, at 20:10, Pawel Konczalski <pawel.konczalski@everyware.ch> wrote:
Hi,
i try to deploy a Kubernetes cluster with OpenStack Magnum but the openstack-cloud-controller-manager pod fails to resolve the master node hostname.
Does magnum require further parameter to configure the DNS names of the master and minions? DNS resolution in the VMs works fine. Currently there is no Designate installed in the OpenStack setup.
openstack coe cluster template create kubernetes-cluster-template1 \ --image Fedora-AtomicHost-29-20190429.0.x86_64 \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.kubernetes \ --flavor m1.kubernetes \ --coe kubernetes \ --volume-driver cinder \ --network-driver flannel \ --docker-volume-size 25
openstack coe cluster create kubernetes-cluster1 \ --cluster-template kubernetes-cluster-template1 \ --master-count 1 \ --node-count 2 \ --keypair mykey
# kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE kube-system coredns-78df4bf8ff-mjp2c 0/1 Pending 0 36m <none> <none> <none> kube-system heapster-74f98f6489-tgtzl 0/1 Pending 0 36m <none> <none> <none> kube-system kube-dns-autoscaler-986c49747-wrvz4 0/1 Pending 0 36m <none> <none> <none> kube-system kubernetes-dashboard-54cb7b5997-sk5pj 0/1 Pending 0 36m <none> <none> <none> kube-system openstack-cloud-controller-manager-dgk64 0/1 CrashLoopBackOff 11 36m <none> kubernetes-cluster1-vulg5fz6hg2n-master-0 <none>
# kubectl -n kube-system logs openstack-cloud-controller-manager-dgk64 Error from server: Get https://kubernetes-cluster1-vulg5fz6hg2n-master-0:10250/containerLogs/kube-s...: dial tcp: lookup kubernetes-cluster1-vulg5fz6hg2n-master-0 on 8.8.8.8:53: no such host
BR
Pawel
Hi together, i just try to (re)create a Kubernetes Magnum cluster on a OpenStack installation where those already was working but get recently flowing error in Docker logs: Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0 # openstack coe cluster template create kubernetes-cluster-template \ --image "Fedora AtomicHost 29" \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-flavor m1.kubernetes \ --flavor m1.kubernetes \ --coe kubernetes \ --volume-driver cinder \ --network-driver flannel \ --docker-volume-size 25 \ --public \ --labels kube_tag=v1.13.4,cloud_provider_tag=1.13.1 # openstack coe cluster create kubernetes-cluster10 \ --cluster-template kubernetes-cluster-template \ --master-count 1 \ --node-count 2 \ --keypair mykey [root@pko-k8s-cluster10-jpojymhgkhdc-master-0 ~]# journalctl -u docker.service -- Logs begin at Wed 2019-07-31 10:02:52 UTC, end at Wed 2019-07-31 10:59:56 UTC. -- Jul 31 10:03:10 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Starting Docker Application Container Engine... Jul 31 10:03:10 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:10.966720103Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.242863167Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243328235Z" level=warning msg="Your kernel does not support cgroup rt period" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243346111Z" level=warning msg="Your kernel does not support cgroup rt runtime" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243358575Z" level=warning msg="Your kernel does not support cgroup blkio weight" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243369525Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.243880444Z" level=info msg="Loading containers: start." Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.340931401Z" level=info msg="Firewalld running: false" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.423128319Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.497740931Z" level=info msg="Loading containers: done." Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.813539429Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.822435917Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.823644437Z" level=info msg="Daemon has completed initialization" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.825054441Z" level=info msg="Docker daemon" commit="1185cfd/1.13.1" graphdriver=overlay2 version=1.13.1 Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:11.903581154Z" level=info msg="API listen on /var/run/docker.sock" Jul 31 10:03:11 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Started Docker Application Container Engine. Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.886226816Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.887560748Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.995141066Z" level=error msg="Handler for GET /v1.26/containers/etcd/json returned error: No such container: etcd" Jul 31 10:03:13 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:13.995439344Z" level=error msg="Handler for GET /v1.26/containers/etcd/json returned error: No such container: etcd" Jul 31 10:03:52 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Stopping Docker Application Container Engine... Jul 31 10:03:52 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[1235]: time="2019-07-31T10:03:52.843904648Z" level=info msg="Processing signal 'terminated'" Jul 31 10:03:52 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Stopped Docker Application Container Engine. Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Starting Docker Application Container Engine... Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:32.665652883Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found" Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:32.841139565Z" level=info msg="devmapper: Creating filesystem xfs on device docker-253:0-8716235-base" Jul 31 10:07:32 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:32.985841976Z" level=info msg="devmapper: Successfully created filesystem xfs on device docker-253:0-8716235-base" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.045750870Z" level=info msg="Graph migration to content-addressability took 0.00 seconds" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046156219Z" level=warning msg="Your kernel does not support cgroup rt period" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046176360Z" level=warning msg="Your kernel does not support cgroup rt runtime" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046189025Z" level=warning msg="Your kernel does not support cgroup blkio weight" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046199775Z" level=warning msg="Your kernel does not support cgroup blkio weight_device" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.046703801Z" level=info msg="Loading containers: start." Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.053599598Z" level=info msg="Firewalld running: false" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.212814678Z" level=info msg="Loading containers: done." Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244296246Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244734418Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244925171Z" level=info msg="Daemon has completed initialization" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.244951281Z" level=info msg="Docker daemon" commit="1185cfd/1.13.1" graphdriver=devicemapper version=1.13.1 Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal systemd[1]: Started Docker Application Container Engine. Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.263178384Z" level=info msg="API listen on /var/run/docker.sock" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.663856788Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.664293204Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.679268641Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.679675879Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.744362224Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.744862040Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.761317130Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.761846808Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.779548407Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.779851911Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.794963363Z" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version 1.0.0-rc2\nspec: 1.0.0-rc2-dev\n" Jul 31 10:07:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:07:33.795546041Z" level=warning msg="failed to retrieve docker-init version: unknown output format: tini version 0.18.0\n" Jul 31 10:12:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:12:33.784829232Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:12:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:12:33.785054762Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Jul 31 10:17:33 pko-k8s-cluster10-jpojymhgkhdc-master-0.novalocal dockerd-current[3603]: time="2019-07-31T10:17:33.788586476Z" level=error msg="Handler for GET /v1.26/images/gcr.io/google_containers/pause:3.0/json returned error: No such image: gcr.io/google_containers/pause:3.0" Did somebody know if Google registry has changed recently? BR Pawel
participants (2)
-
Bharat Kunwar
-
Pawel Konczalski