Open Stack

Hi Gerry,

Thanks for the reply.
Attached the outputs for the commands you asked for.

I see a python traceback while reading container metadata. But I've
configured replica count as 2 for container rings too (all 3 types of rings
have been similarly configured). And it's able to read the other replica.

I wonder why this error then?


On Fri, Jul 4, 2014 at 4:00 PM, Drudy, Gerry <Gerry.Drudy at hp.com> wrote:

>  I too would have expected the PUT to succeed given the config you
> describe.  Perhaps the proxy log entries for the PUT might indicate why it
> is failing?
>
>
>
> If you can’t figure out what is going wrong from the proxy server logs can
> you post the o/p of the following 2 commands:
>
>
>
> swift-ring-builder /etc/swift/object.builder
>
> swift-get-nodes /etc/swift/object.ring.gz blah blah blah
>
>
>
> Gerry.
>
>
>
> *From:* Shyam Prasad N [mailto:nspmangalore at gmail.com]
> *Sent:* 04 July 2014 10:45
> *To:* openstack at lists.openstack.org
> *Subject:* [Openstack] Replication count..
>
>
>
> Hi,
>
> I have a swift setup with 2 disks each on 2 nodes. The swift rings have
> been setup with replication count of 2, with each node with unique zone id.
> After creation of a few objects, I pulled out one of the disks from node1.
> After some more object creations, I pulled out the remaining swift disk on
> node1. Now, my GETs are successful. But PUT requests are failing with
> "Service Unavailable" error.
>
> I'm confused with this behaviour. I thought swift tries to PUT replicas
> for an object into different zones on a best effort basis. Right now, there
> are still two disks active on the other zone. Shouldn't it put both the
> replicas on the available disks now, rather than returning the above error?
> Or is my understanding about this wrong?
>
> Thanks in advance for the replies.
>
>
> --
> -Shyam
>



-- 
-Shyam
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140704/40bbe4db/attachment.html>
-------------- next part --------------
~# grep -R 'txa3cac3a103d54f4d84bed-0053b67a56' /var/log/swift/*
/var/log/swift/proxy.error:Jul  4 09:58:29 ip-10-141-173-100 proxy-server: ERROR with Container server 10.141.173.100:6011/xvdb re: Trying to HEAD /v1/AUTH_test/container: 
Traceback (most recent call last):
  File "/home/ubuntu/swift/swift/proxy/controllers/base.py", line 765, in _get_source_and_node
    possible_source = conn.getresponse()
  File "/home/ubuntu/swift/swift/common/bufferedhttp.py", line 123, in getresponse
    response = HTTPConnection.getresponse(self)
  File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 407, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
    raise BadStatusLine(line)
BadStatusLine: '' (txn: txa3cac3a103d54f4d84bed-0053b67a56)
/var/log/swift/proxy.error:Jul  4 10:08:29 ip-10-141-173-100 proxy-server: ERROR with Object server 10.141.173.100:6010/xvdb re: Expect: 100-continue on /AUTH_test/container/test_object2: Timeout (600s) (txn: txa3cac3a103d54f4d84bed-0053b67a56) (client_ip: 127.0.0.1)
/var/log/swift/proxy.error:Jul  4 10:08:29 ip-10-141-173-100 proxy-server: Handoff requested (1) (txn: txa3cac3a103d54f4d84bed-0053b67a56) (client_ip: 127.0.0.1)
/var/log/swift/proxy.error:Jul  4 10:08:32 ip-10-141-173-100 proxy-server: Object PUT returning 503 for [408, 201] (txn: txa3cac3a103d54f4d84bed-0053b67a56) (client_ip: 127.0.0.1)
/var/log/swift/proxy.log:Jul  4 09:56:38 ip-10-141-173-100 proxy-server: - - 04/Jul/2014/09/56/38 HEAD /v1/AUTH_test HTTP/1.0 204 - Swift - - - - txa3cac3a103d54f4d84bed-0053b67a56 - 0.0062 GET_INFO - 1404467798.633363962 1404467798.639591932
/var/log/swift/proxy.log:Jul  4 09:58:29 ip-10-141-173-100 proxy-server: - - 04/Jul/2014/09/58/29 HEAD /v1/AUTH_test/container HTTP/1.0 204 - Swift - - - - txa3cac3a103d54f4d84bed-0053b67a56 - 110.6712 CQ - 1404467798.641381025 1404467909.312623024
/var/log/swift/proxy.log:Jul  4 10:08:32 ip-10-141-173-100 proxy-server: 127.0.0.1 127.0.0.1 04/Jul/2014/10/08/32 PUT /v1/AUTH_test/container/test_object2 HTTP/1.0 503 - curl/7.22.0%20%28x86_64-pc-linux-gnu%29%20libcurl/7.22.0%20OpenSSL/1.0.1%20zlib/1.2.3.4%20libidn/1.23%20librtmp/2.3 AUTH_tkc6611d395e884b1497d0285ba9d6b0cb 3 118 - txa3cac3a103d54f4d84bed-0053b67a56 - 713.8247 - - 1404467798.630681038 1404468512.455360889

~# swift-ring-builder /etc/swift/object.builder
/etc/swift/object.builder, build version 6
1024 partitions, 2.000000 replicas, 1 regions, 2 zones, 4 devices, 0.00 balance
The minimum number of hours before a partition can be reassigned is 1
Devices:    id  region  zone      ip address  port  replication ip  replication port      name weight partitions balance meta
             0       1     1  10.141.173.100  6010  10.141.173.100              6010      xvdb   1.00        512    0.00 
             1       1     1  10.141.173.100  6020  10.141.173.100              6020      xvdc   1.00        512    0.00 
             2       1     2   10.238.160.21  6010   10.238.160.21              6010      xvdb   1.00        512    0.00 
             3       1     2   10.238.160.21  6020   10.238.160.21              6020      xvdc   1.00        512    0.00 

~# swift-get-nodes /etc/swift/object.ring.gz AUTH_test container test_object2

Account 	AUTH_test
Container	container
Object   	test_object2


Partition	277
Hash       	456bc75d7ffa08359cbe41e4aa7f904b

Server:Port Device	10.238.160.21:6010 xvdb
Server:Port Device	10.141.173.100:6010 xvdb
Server:Port Device	10.238.160.21:6020 xvdc	 [Handoff]
Server:Port Device	10.141.173.100:6020 xvdc	 [Handoff]


curl -I -XHEAD "http://10.238.160.21:6010/xvdb/277/AUTH_test/container/test_object2"
curl -I -XHEAD "http://10.141.173.100:6010/xvdb/277/AUTH_test/container/test_object2"
curl -I -XHEAD "http://10.238.160.21:6020/xvdc/277/AUTH_test/container/test_object2" # [Handoff]
curl -I -XHEAD "http://10.141.173.100:6020/xvdc/277/AUTH_test/container/test_object2" # [Handoff]


Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 10.238.160.21 "ls -lah ${DEVICE:-/srv/node}/xvdb/objects/277/04b/456bc75d7ffa08359cbe41e4aa7f904b/"
ssh 10.141.173.100 "ls -lah ${DEVICE:-/srv/node}/xvdb/objects/277/04b/456bc75d7ffa08359cbe41e4aa7f904b/"
ssh 10.238.160.21 "ls -lah ${DEVICE:-/srv/node}/xvdc/objects/277/04b/456bc75d7ffa08359cbe41e4aa7f904b/" # [Handoff]
ssh 10.141.173.100 "ls -lah ${DEVICE:-/srv/node}/xvdc/objects/277/04b/456bc75d7ffa08359cbe41e4aa7f904b/" # [Handoff]