Open Stack

On 3/24/15 2:52 PM, Shrinand Javadekar wrote:
> Hi,
>
> I wrote a small microbenchmark for measuring the performance of
> extended attributes in XFS. In the experiment, I wrote 100K files,
> each with extended attributes. In one experiment, XFS was formatted
> with the default inode size of 256 bytes. In the other experiment, it
> was formatted with an inode size of 2048 bytes.
>
> Here's the actual code: http://pastie.org/private/2gnr3j9p5lxzzufdypkyzq
>
> I ran this using cProfile. The results suggest that time required for
> "xattr._xattr.fsetxattr" when the inode is 256 bytes is ~4 times that
> when the inode is 2048 bytes.
>
> When the inode is 256 bytes:
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>     100000    3.933    0.000    3.933    0.000 {xattr._xattr.fsetxattr}
>     100000    4.344    0.000    4.344    0.000 {xattr._xattr.fsetxattr}
>     100000    4.624    0.000    4.624    0.000 {xattr._xattr.fsetxattr}
>
> When the inode is 2048 bytes:
>
>     ncalls  tottime  percall  cumtime  percall filename:lineno(function)
>     100000    1.239    0.000    1.239    0.000 {xattr._xattr.fsetxattr}
>     100000    1.102    0.000    1.102    0.000 {xattr._xattr.fsetxattr}
>     100000    1.067    0.000    1.067    0.000 {xattr._xattr.fsetxattr}
>
> I will delve into more elaborate experiments with Swift and XFS
> filesystems formatted with different inode sizes. However, before
> that, I wanted to check if the above values are plausible and/or get
> suggestions around this microbenchmark.

The benchmark makes sense. The code (reproduced below in case that paste 
goes away someday [1]) is setting the xattr "user.swift.metadata" (19 
bytes) to a value of length 228, for a total of 247 bytes.

If the extended attributes will fit into the inode, then XFS will store 
them there. However, if the extended attributes do not fit into the 
inode, then the extended attributes are stored outside the inode in one 
or more blocks. See [2] for more details.

It's important to note that *all* the extended attributes are stored in 
the *same place*. If everything fits in the inode, then all extended 
attributes live in the inode. If the extended attributes, taken 
together, exceed the available inode space by just 1 byte, then all the 
extended attributes are kicked out of the inode and off to an extents block.

Thus, with a 256-byte inode and a 247-byte payload, there's not room in 
the inode with all the other stuff that lives in there for the extended 
attributes, so they spill to extents. With a 2 KiB inode, there's more 
than enough room.

> If these values make sense, will this translate into improved Swift performance?

Maybe(TM). It depends on your workload. When you make inodes bigger, 
fewer of them fit in the kernel's buffer cache, possibly resulting in 
more work. On the other hand, when you make them smaller, then you 
always get xattrs spilled to extents.

Also, it's affected by what your users do. Are your users storing lots 
of metadata (X-Object-Meta-*) on their objects? Are you using middleware 
that stores lots of sysmeta (X-Object-Sysmeta-*) on your objects? Are 
you running SELinux, which uses lots of xattrs for its own purposes?

Further, I think the benchmarking script is too naive to really get at 
the key improvement here: avoiding seeks. If the xattrs live in the 
inode, then we get them essentially for free when we open() the file. If 
the xattrs live in extents, then at least one random IO is required to 
fetch them. The improvement shown here is on the order of 30 
microseconds, while an extra disk seek is more like 10-20 *milliseconds* 
(on spinning platters). If you really want to measure the performance 
gain, you'll need to drop the buffer cache each time and fsync() when 
you're done.


[1]
============
#!/usr/bin/python

import xattr
prefix = "/srv/node/r1/"

for i in range (100000):
    fd = open(prefix + "file-" + str(i), 'w')
    fd.write("Some text")
    xattr.setxattr(fd, "user.swift.metadata", 
"U.Content-Lengthq.U.166U.nameq.U'/AUTH_admin/container-test/testxattr.pyq.U.X-Object-Meta-MtimeU.1426996406.379914q.U.ETagq.U 
bab3674eed1a8725793f2d2de21f50a9q.U.X-Timestampq.U.1426996448.35449U.Content-Typeq.U.text/x-pythonq.u.")
    fd.flush()
    fd.close()

============


[2] 
http://xfs.org/docs/xfsdocs-xml-dev/XFS_Filesystem_Structure/tmp/en-US/html/Extended_Attributes.html