The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

...

On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine

Thanks Ignazio

Thanks, cluster_user_trust is right in my conf. Now I am going to play wth heat_container_agent labels. Many thanks. I'll send updates. Ignazio Il giorno lun 2 dic 2019 alle ore 16:29 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

Yes you can specify comma separated `labels` when you create a cluster template/cluster from the dashboard.

e.g. heat_container_agent_tag=train-stable,kube_tag=v1.14.8.

Also make sure cluster_user_trust=True inside your magnum.conf.

On 2 Dec 2019, at 15:26, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio

Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

...

On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine

Thanks Ignazio

Hello again, It seems I am facing the bug 1701381 (socket broken pipe) but it is related to ocata :-( Ignazio Il giorno lun 2 dic 2019 alle ore 16:49 Ignazio Cassano < ignaziocassano@gmail.com> ha scritto:

Hello, the stack went fine with paramenter you suggested....many many thanks.

I have got further issues :

1) /var/log/magnum on controllers in emply.

Logs are sent on /var/log/messages

2) in var log messages I have a lot of errors like the following:

ec 2 16:47:15 tst2-osctrl01 magnum-api: Traceback (most recent call last): Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 568, in process_request Dec 2 16:47:15 tst2-osctrl01 magnum-api: self.finish_request(request, client_address) Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request Dec 2 16:47:15 tst2-osctrl01 magnum-api: self.RequestHandlerClass(request, client_address, self) Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 651, in __init__ Dec 2 16:47:15 tst2-osctrl01 magnum-api: self.finish() Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/SocketServer.py", line 710, in finish Dec 2 16:47:15 tst2-osctrl01 magnum-api: self.wfile.close() Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/socket.py", line 279, in close Dec 2 16:47:15 tst2-osctrl01 magnum-api: self.flush() Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib64/python2.7/socket.py", line 303, in flush Dec 2 16:47:15 tst2-osctrl01 magnum-api: self._sock.sendall(view[write_offset:write_offset+buffer_size]) Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 401, in sendall Dec 2 16:47:15 tst2-osctrl01 magnum-api: tail = self.send(data, flags) Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 395, in send Dec 2 16:47:15 tst2-osctrl01 magnum-api: return self._send_loop(self.fd.send, data, flags) Dec 2 16:47:15 tst2-osctrl01 magnum-api: File "/usr/lib/python2.7/site-packages/eventlet/greenio/base.py", line 382, in _send_loop Dec 2 16:47:15 tst2-osctrl01 magnum-api: return send_method(data, *args) Dec 2 16:47:15 tst2-osctrl01 magnum-api: error: [Errno 32] Broken pipe

I deplpyed magnum on 3 controllers with a pacemaker cluster and haproxy. I can use dashboard and command line for magnum so I do not understand why I got the above errors.

Regards Ignazio

Il giorno lun 2 dic 2019 alle ore 16:29 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Yes you can specify comma separated `labels` when you create a cluster template/cluster from the dashboard.

e.g. heat_container_agent_tag=train-stable,kube_tag=v1.14.8.

Also make sure cluster_user_trust=True inside your magnum.conf.

On 2 Dec 2019, at 15:26, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio

Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar < bharat@stackhpc.com> ha scritto:

...

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

...

On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine

Thanks Ignazio

Hello Bharat, we wrote today on irc. On rdo stein testing repo the new openstack magnum version has ben released at 2:27 Pm. I tested it but master kubernetes node reports errors while is trying to tag nodes. If you want I could send you more details . Best Regards Ignazio Il Lun 2 Dic 2019, 16:29 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

Yes you can specify comma separated `labels` when you create a cluster template/cluster from the dashboard.

e.g. heat_container_agent_tag=train-stable,kube_tag=v1.14.8.

Also make sure cluster_user_trust=True inside your magnum.conf.

On 2 Dec 2019, at 15:26, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio

Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

...

On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine

Thanks Ignazio

Hello Bharat, the following are last lines of the master node /var/log/cloud-init-output.log file: ystemctl enable heat-container-agent + systemctl start heat-container-agent starting services activating service etcd activating service docker activating service kube-apiserver activating service kube-controller-manager activating service kube-scheduler activating service kubelet activating service kube-proxy Trying to label master node with node-role.kubernetes.io/master="" Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" Regards Ignazio Il giorno ven 6 dic 2019 alle ore 22:54 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

More details is good.

Cheers

Bharat

On 6 Dec 2019, at 17:46, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 Hello Bharat, we wrote today on irc. On rdo stein testing repo the new openstack magnum version has ben released at 2:27 Pm. I tested it but master kubernetes node reports errors while is trying to tag nodes. If you want I could send you more details . Best Regards Ignazio

Il Lun 2 Dic 2019, 16:29 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Yes you can specify comma separated `labels` when you create a cluster template/cluster from the dashboard.

e.g. heat_container_agent_tag=train-stable,kube_tag=v1.14.8.

Also make sure cluster_user_trust=True inside your magnum.conf.

On 2 Dec 2019, at 15:26, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio

Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar < bharat@stackhpc.com> ha scritto:

...

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

...

On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine

Thanks Ignazio

No, I don't. I'll apply this patch asap. Thanks Ignazio Il Dom 8 Dic 2019, 00:06 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

Do you have this change in the version of magnum you installed? If you do, it should definitely work!

https://review.opendev.org/#/c/695632/

Sent from my iPhone

On 7 Dec 2019, at 17:35, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 Hello Bharat, the following are last lines of the master node /var/log/cloud-init-output.log file: ystemctl enable heat-container-agent + systemctl start heat-container-agent starting services activating service etcd activating service docker activating service kube-apiserver activating service kube-controller-manager activating service kube-scheduler activating service kubelet activating service kube-proxy Trying to label master node with node-role.kubernetes.io/master="" Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master=""

Regards Ignazio

Il giorno ven 6 dic 2019 alle ore 22:54 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

More details is good.

Cheers

Bharat

On 6 Dec 2019, at 17:46, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 Hello Bharat, we wrote today on irc. On rdo stein testing repo the new openstack magnum version has ben released at 2:27 Pm. I tested it but master kubernetes node reports errors while is trying to tag nodes. If you want I could send you more details . Best Regards Ignazio

Il Lun 2 Dic 2019, 16:29 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Yes you can specify comma separated `labels` when you create a cluster template/cluster from the dashboard.

e.g. heat_container_agent_tag=train-stable,kube_tag=v1.14.8.

Also make sure cluster_user_trust=True inside your magnum.conf.

On 2 Dec 2019, at 15:26, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio

Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar < bharat@stackhpc.com> ha scritto:

...

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

...

On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine

Thanks Ignazio

Yes, you are right, I think they releases patched packages today Ignazio Il giorno dom 8 dic 2019 alle ore 13:37 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

It should have been installed with stein 8.2.0. It sounds like you haven’t installed the right version of the package.

On 8 Dec 2019, at 10:55, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 No, I don't. I'll apply this patch asap. Thanks Ignazio

Il Dom 8 Dic 2019, 00:06 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Do you have this change in the version of magnum you installed? If you do, it should definitely work!

https://review.opendev.org/#/c/695632/

Sent from my iPhone

On 7 Dec 2019, at 17:35, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 Hello Bharat, the following are last lines of the master node /var/log/cloud-init-output.log file: ystemctl enable heat-container-agent + systemctl start heat-container-agent starting services activating service etcd activating service docker activating service kube-apiserver activating service kube-controller-manager activating service kube-scheduler activating service kubelet activating service kube-proxy Trying to label master node with node-role.kubernetes.io/master="" Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master=""

Regards Ignazio

Il giorno ven 6 dic 2019 alle ore 22:54 Bharat Kunwar < bharat@stackhpc.com> ha scritto:

...

More details is good.

Cheers

Bharat

On 6 Dec 2019, at 17:46, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 Hello Bharat, we wrote today on irc. On rdo stein testing repo the new openstack magnum version has ben released at 2:27 Pm. I tested it but master kubernetes node reports errors while is trying to tag nodes. If you want I could send you more details . Best Regards Ignazio

Il Lun 2 Dic 2019, 16:29 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Yes you can specify comma separated `labels` when you create a cluster template/cluster from the dashboard.

e.g. heat_container_agent_tag=train-stable,kube_tag=v1.14.8.

Also make sure cluster_user_trust=True inside your magnum.conf.

On 2 Dec 2019, at 15:26, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio

Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar < bharat@stackhpc.com> ha scritto:

...

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

...

On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine

Thanks Ignazio

Hello, applyng pathes you sent me, only master node is tagged while minion are not tagged: [centos@kubectl ~]$ kubectl get nodes NAME STATUS ROLES AGE VERSION kube-vkfsogbuz4we-master-0 Ready master 10m v1.14.8 kube-vkfsogbuz4we-minion-0 Ready <none> 10m v1.14.8 [centos@kubectl ~]$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-865bd969f-cldrp 0/1 Pending 0 10m heapster-7bf5794cc7-hkhcl 0/1 Pending 0 10m k8s-keystone-auth-pdcxt 1/1 Running 0 10m kube-dns-autoscaler-6c6b4b7cf7-pwxtl 0/1 Pending 0 10m kube-flannel-ds-amd64-k5r97 1/1 Running 0 10m kube-flannel-ds-amd64-s2vrm 1/1 Running 0 10m kubernetes-dashboard-d48c76949-wvgr4 0/1 Pending 0 10m openstack-cloud-controller-manager-sxxkd 1/1 Running 0 10m Ignazio Il giorno dom 8 dic 2019 alle ore 16:20 Ignazio Cassano < ignaziocassano@gmail.com> ha scritto:

New rpm packages contains only the path for the second file related to the bug you sent me, any case I am going to try patching by hand

Il giorno dom 8 dic 2019 alle ore 13:37 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

It should have been installed with stein 8.2.0. It sounds like you haven’t installed the right version of the package.

On 8 Dec 2019, at 10:55, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 No, I don't. I'll apply this patch asap. Thanks Ignazio

Il Dom 8 Dic 2019, 00:06 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Do you have this change in the version of magnum you installed? If you do, it should definitely work!

https://review.opendev.org/#/c/695632/

Sent from my iPhone

On 7 Dec 2019, at 17:35, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 Hello Bharat, the following are last lines of the master node /var/log/cloud-init-output.log file: ystemctl enable heat-container-agent + systemctl start heat-container-agent starting services activating service etcd activating service docker activating service kube-apiserver activating service kube-controller-manager activating service kube-scheduler activating service kubelet activating service kube-proxy Trying to label master node with node-role.kubernetes.io/master="" Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master="" error: resource(s) were provided, but no name, label selector, or --all flag specified Trying to label master node with node-role.kubernetes.io/master=""

Regards Ignazio

Il giorno ven 6 dic 2019 alle ore 22:54 Bharat Kunwar < bharat@stackhpc.com> ha scritto:

...

More details is good.

Cheers

Bharat

On 6 Dec 2019, at 17:46, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

 Hello Bharat, we wrote today on irc. On rdo stein testing repo the new openstack magnum version has ben released at 2:27 Pm. I tested it but master kubernetes node reports errors while is trying to tag nodes. If you want I could send you more details . Best Regards Ignazio

Il Lun 2 Dic 2019, 16:29 Bharat Kunwar <bharat@stackhpc.com> ha scritto:

...

Yes you can specify comma separated `labels` when you create a cluster template/cluster from the dashboard.

e.g. heat_container_agent_tag=train-stable,kube_tag=v1.14.8.

Also make sure cluster_user_trust=True inside your magnum.conf.

On 2 Dec 2019, at 15:26, Ignazio Cassano <ignaziocassano@gmail.com> wrote:

The network plugin is flannel . I do not kow what is kube_tag . I read that with it I can specify the kubernetes version, right ? I am using openstack dashboard for creating my cluster: where I can specify heat_container_agent_tag: train-stable in the dashboard ? Thanks Ignazio

Il giorno lun 2 dic 2019 alle ore 16:11 Bharat Kunwar < bharat@stackhpc.com> ha scritto:

...

Hi Ignazio,

What version of kube_tag are you trying to run? What network plugin? I suggest using `heat_container_agent_tag: train-stable` as its easier to debug.

Best

bharat

> On 2 Dec 2019, at 15:08, Ignazio Cassano <ignaziocassano@gmail.com> wrote: > > hello, I've just installed stein with magnum on centos 7. I am creating a kubernetes cluster but it is running since 22 minutes and I think it will not work because the heat stack is waiting OSSoftwareDeployment completes > I presume something is not working on heat with magnum. I ran several stack with Software Deployment without magnum, and they worked fine > > Thanks > Ignazio