Consider this a diary post. Perhaps the war-story is useful.
A further update on my Ceph cluster. I was running BTRFS v0.19 and the performance was horrible. This is most likely a very early version of BTRFS and I am running on very under spec hardware. While the future of BTRFS is bright, it is not for the nodes I have running now. I’ve reverted all my OSDs on USB keys back to XFS. That gave an instant 2-3 times speed up on writes.
I found a good deal on a second hand server with a generous case, 8 gigs of RAM, dual gig Ethernet ports and a decent enough CPU. It already has 3 spinning disks on-board and room for plenty more. I need to rearrange my office space to fit it in so that’ll take a couple of weeks.
I also have the budget to upgrade my desktop machine. The parts that come free after that upgrade will be built into another ceph node.
The other neat thing was creating an init.d script to automatically find and mount the lvm volumes, startup the OSDs then mount the Ceph filestystem. I needed something that performs this task quite late (read: last) in the boot process so that all the USB devices have had a chance to wake up.
So, once some work and deadlines are cleared then there'll be exciting things happening with my ceph cluster.
