The BFC Computing Weblog

A note I wrote when asked about the future of ZFS and current best-practices for storage:

"My current line of thinking is that ZFS has the required reliability, OpenSolaris is where it actually works now.

FreeBSD 9 should have a good version of it (they can't get the zpool up to v23 in FreeBSD 8 based on FreeBSD's major-version compatibility requirements), and Linux is also going to get it. Either of those two are good destinations. Both are currently flakey though, and OpenSolaris is solid.

The ZFS storage layer has been ported to linux by one of the National Labs - they're still working on the POSIX layer. I suspect that'll be ready in a year or so.

I suspect Nexenta will wind up switching kernels itself and still calling itself Nexenta in a year or two. ZFS is endian- and platform agnostic, so moving a pool from one OS to another is a 'zpool export storage' on the source OS, re-install the base OS, and 'zpool import storage' and you're good to go.

Linux's btrfs might even be a decent in 2-3 years, but it's really really early still.

There's also the Illumos project which is making an Oracle-free OpenSolaris which stands a chance. Nexenta is funding those guys - at least they'll finally be able to do a complete Open Source build. That's probably transitional, though - why compete with FreeBSD on drivers? "Because OpenSolaris is much faster on disk access" is a fair answer, so they could also wind up supporting a more narrow set of hardware on an ongoing basis, until FreeBSD is that fast (they can figure out how OpenSolaris beats them pretty easily, especially with lots of people wanting it to be where they wind up.)

FreeBSD is license-compatible with CDDL, otherwise Linux would be the presumptive destination. Oracle could still chose to dual-license and settle the matter that quickly, but that's seeming less and less likely. Bill Moore, co-lead on ZFS has already left Oracle and joined the Nexenta board, so I suspect that's where ZFS will evolve, not at Oracle. If Jeff Bonwick does the same, it's pretty much settled."

Just a quick note for the search engines to find - peth0 can go missing from ifconfig if there is a GATEWAY= entry in ifcfg-eth0 (anaconda puts it here) and presumably -eth1, etc.. Put the default gateway in /etc/syconfig/network instead and use route-eth1 files instead to specify gateways.

Reboot for xend to do its setup correctly (please commment if there's a way to do this without reboot that works...) and instead of 'no peth0' you'll find one now exists. Also, your xenbr0 will be set up properly.

If you're seeing this, this article isn't quite done yet. Still setting things up right.

Fedora client, Nexenta Server.

On the linux side, start rpcidmapd and set it to start on boot.

service rpcidmapd start
          chkconfig --levels 345 rpcidmapd on
          

Wherever your DNS is, make sure your forward and reverse are set up correctly. No, really, make sure.

$ host my.linux.host.fqdn
          my.linux.host.fqdn has address 1.2.3.4
          $ host 1.2.3.4
          4.3.2.1.in-addr.arpa domain name pointer my.linux.host.fqdn.
          

Make sure dnsdomainname returns correctly on the linux host. You need to have my.linux.host.fqdn first on the line with 127.0.0.1 in /etc/hosts. This sets the NFSv4 domain name. Restart rpcidmapd if you needed to fix this. If this is wrong your files will all show as nobody:nobody on the mount (at this point everybody in the mailing lsit archives gives up and goes back to the crummy NFSv3). Make sure linux's dnsdomainname matches the output of Solaris's:

cat /var/run/nfs4_domain
          

Now, share under zfs:

zfs set sharenfs=rw=my.linux.host.fqdn,root=my.linux.host.fqdn pool/vol/subvol
          

mount under linux:

mount -t nfs4 solarismachine:/vol/subvol /mnt/localmount/ -o rw,intr,hard,proto=tcp,port=2049
          

Then set up a root nfs mount by:

blah, blah, blah, todo, todo, todo
          

I've got Jolicloud on the wife's netbook, and it's a nice easy-to-use distro.

Trouble is, it's based on Jaunty, which has old mythtv packages. These won't connect to our MythTV 0.23 backend in the TV room.

There is hope, though, the Avendard repo has newer packages compiled for Jaunty, but they're a bit tricky to install.

The process roughly:

Create a file:

/etc/apt/sources.list.d/avenard.list
          

with the lines:

deb http://www.avenard.org/files/ubuntu-repos jaunty release
          deb http://www.avenard.org/files/ubuntu-repos jaunty testing
          

and run the commands:

wget http://www.avenard.org/files/ubuntu-repos/ubuntu-repos.key && sudo apt-key add ubuntu-repos.key && rm ubuntu-repos.key
          apt-get update
          

to pull in the new repo. Now, remember this is dpkg/apt, so we can't just go installing mythtv first as the dependency resolution needs a bit of help.

First do:

sudo apt-get update
          sudo apt-get install nvidia-glx-185 nvidia-185-libvdpau nvidia-185-kernel-source
          sudo apt-get install libvdpau1
          

Now do:

sudo apt-get dist-upgrade mythtv-frontend mythvideo
          

and whichever other modules you need.

Then run:

sudo dpkg-reconfigure mythtv-common
          

to set your backend password.

Finally, install nfs and autofs to be able to mount your storage directory:

sudo apt-get install autofs nfs-common
          

and then edit:

/etc/auto.master
          

uncomment the /net entry, save, and run:

/etc/init.d/autofs restart
          

Then symlink to however you have your backend storage configured, e.g.:

ln -s /net/192.168.1.10/storage/ /storage
          

Now, launch mythfrontend from the Jolicloud Sound & Video group, where it will ask you for sudo access to add your user to the mythtv group and logout. Do it.

Log back in again, launch MythTV again, and go into 'Setup' and configure the storage directories for your media and/or recordings. Set parental controls as needed, they're front-end specific. Change the theme if needed, and set your painter to OpenGL if appropriate.

Those being done, you should be good to go to exit and start MythFrontend from the menu and just use it normally. SD MPEG-2 DVD video streams over 802.11g seems to work fine here in an ASUS 1000HE netbook. And now you have the world's most complex second(third,fourth) television.

Update: Jolicloud support writes via Twitter: "Adding third-party repositories could compromise your configuration. We won't be able to provide you support. ^CD" I suspect if you're reading this you can handle your own support, but be forewarned if you count on Jolicloud support. Personally, I'd rather see them engaging and supporting their community, but I understand about resource constraints.

They don't make flash drives like they used to. I've seen several pfSense installs fail recently due to drives flaking out and the wear-leveling not working as advertised.

Of course, you make regular backups of your config file, but in case you forgot, we can probably rescue your config file off of a disk image. Re-installing pfSense doesn't take too long, but rebuilding a working config file can take many hours, so rescuing is preferential.

This script has worked for me with dozens of file versions, but one can imagine scenarios with fragmented files where it would fail. There's nothing fancy going on here (this was hacked up with a client standing in my office with a broken pfSense box), but it might prove useful in a pinch.

          #!/usr/bin/perl -w
          use strict;
          use warnings FATAL=>'all';
          
          =comment
          pfsense_extract.pl - extract pfSense configs from an input stream
          (c) 2010 BFC Computing, LLC.  Licensed under the same terms as pfSense.
          
          This is useful for taking an image file of a damaged pfSense install
          and pulling out config files.  If you can mount the image normally, you
          should do that first.
          
          Due to the nature of the filesystem, there are often many copies of a 
          config file in a disk image, from each time it was saved.  You will 
          find a bunch of output files named: pfsense-config-1.xml, pfsense-config-2.xml,
          etc.  You can then use tools like diff to find out which the right one was.
          
          Example: 
             dd if=/dev/sdg of=broken_pfsense_image.dd bs=2M conv=sync,noerror
             strings broken_pfsense_image.dd | perl pfsense_extract.pl
          
          Processing a 1GB image as per the example takes about 20 seconds on a standard 
          2GHz desktop machine.
          =cut
          
          my $BASENAME='pfsense-config-X.xml';
          my $counter = 0;
          my ($outfile);
          my $do_output = 0;
          
          while (<>) {
          
              chomp;
          
              if ($_ eq '<pfsense>') {
          	$counter++;
          	my $filename = $BASENAME;
          	$filename =~ s/X/$counter/;
          	open($outfile,">$filename");
          	$do_output = 1;
              }
          
              if ($do_output) {
          	print $outfile $_ . "\n";
              }
          
              if ($_ eq '</pfsense>') {
          	close $outfile;
          	$do_output = 0;
              }
              
          }
          

There's a problem when running two X-displays with MythTV - some events on the non-Myth screen will steal focus and then the MythTV controls will no longer respond. This thread describes the problem well, but is now closed for comments.

Since then, mouse-switchscreen has been written, and solves the problem correctly. It's possible to bind the program to a hotkey.

In the end, I found it better to just run one display at a time since I couldn't prevent the focus stealing.

Redhat recently released a set of virtualized I/O devices for KVM, the kernel virtual machine. This short post will outline a method of converting a Windows Vista install (on KVM) to the new drivers using Virt-Manager. It has been tested on Fedora 11.

1. Make sure Vista VM is up to date on patches and the disk is error free.

2. Download drivers from Redhat network or here.

3. Mount the .iso file as a CD-ROM device.

Now you might think you can use the ‘Add Hardware Wizard’ here and add the drivers, add the hardware, and be good. I did. I wound up with an unbootable disk. Apparently Vista’s autodetection is required in this process. So…

1. Add a new network device of type ‘virtio’. Vista will do its “you’ve got hardware” routine and run you through all of its wizards. When it asks you for drivers, point it at the i386/2008 directory on the driver disc image. Yes, Yes, OK, Yes, Really, Continue, etc.

2. Shutdown the VM and remove the old ethernet controller. Boot up Vista and make sure the network works. You can conceivably skip this step for now if you want to make troubleshooting harder.

3. Add a new Storage controller. Leave the existing one as-is for now. You’ll have to pick a disk image you’re not using right now, or make a new one. Anything is fine, we’re not going to ever use it inside Vista. Do the driver dance again.

4. Shutdown Windows. Remove the storage controllers, and add a new one, type ‘virtio’, with your normal hard drive image. Take care of the old ethernet controller here too, if you ignored my previous advice.

5. Boot Windows normally. It should now be coming up on VirtIO disk and network drivers. If you get a bluescreen or a plea to use the RepairCD, something went wrong. Use the repair CD to restore to a previous restore-point and try again.

If anybody knows where to find a sound driver, please leave a comment!

For folks who are running the current development, or soon-to-be-just-released Fedora 11, you might find Firefox to be very crashy. It's not because it's the semi-controversial 3.5b4 version (which is excellent), it's because of a buggy library.

I'm running it with the Tree Style Tab and NoScript extensions, and can get a crash half the time when Session Restore is running, and almost all the time when I allow a site in NoScript.

If you run firefox from the console, so you get the debug messages, you'll see:

cairo-ft-font.c:554: _cairo_ft_unscaled_font_lock_face: Assertion `!unscaled->from_face' failed 

when the crash happens. I tracked this down through the Mozilla and Freedesktop bug systems to a problem with the Cairo graphics engine improperly disposing of fonts which it didn't own, for which a fix was incorporated last December. However, the version of Cairo shipping in Fedora 11 is older than that.

So, I applied the simple patch, fixed up the .spec, and put up some new RPM's for i386 and an SRPM for hackers and x86_64 users to build (rpmbuild --rebuild cairo-1.8.6-3.fc11.src.rpm).

I haven't tried cross-compiling from i386 to x86_64 before, and --target=x86_64 doesn't work, so if anybody can tell me how to do that short of learning mock, please leave a comment and I'll put up RPM's for that too.

The Redhat bug is here. Hopefully it gets accepted soon.

I recently had the power supply fail on my SOHO server, which was a mongrel of old parts, far too many USB cables, and was pretty darn slow. It was also very expensive to run, having a Pentium IV in it, the worst of Intel's line.

My goals for a new server were:

• quiet
• energy efficient
• virtualization support
• lots of storage
• easy to take backups offsite
• rackmount
• budget-friendly.

After poking around NewEgg for a while (I think I enjoy shopping there a bit too much) I came up with a list of parts (after reading many of the helpful reviews), and I have to say I couldn't be happier with the system.

It's almost inaudible, runs at about 105W under normal load, has seven hard drives in it, of various capacities, fits in my rack, has a hot-swap drive for off-site backups, and runs Fedora 10 like a charm. The case is especially nice to work inside, and is of higher quality than you'd expect for the price.

I'm acually using the 2.66GHz version of the Core2Duo, but they don't seem to make that anymore - 3.0GHz seems to be the low-end. It's worth noting here that most of the commercial server builders try to force you into the Xeon line with a rackmount server and those are both more expensive and more power hungry than the Core2Duo and Core2Quad lines. Get what you really need, keeping in mind that virtualizing multiple systems onto one is a huge energy win.

Additionally, I got a cooler from BestBuy (surprisingly their in-stock cooler is the nicest I've found) and used Arctic Silver 5 thermal compound to bond the CPU. Plus a bunch of SATA cables I have in a box (they seem to spontaneously generate in there). The whole package comes in under $1200 even if you have to buy every part. Compare at fifty percent more to purchase pre-assembled.

Here's the parts list:

• 1 x ARK 4U-500-CA Black 4U Rackmount Case - Retail
• 4 x Athena Power 6" SATA II Y cable Model CABLE-YSATA290 - Retail
• 1 x ASUS P5N7A-VM LGA 775 NVIDIA GeForce 9300/nForce 730i HDMI Micro ATX Intel Motherboard - Retail
• 1 x Rosewill RG430-2 430W 80Plus Certified,ATX12V v2.3/EPS12V v2.91, Active-PFC Power Supply, UL,FCC,CE,TUV,ROHS - Retail
• 1 x ICY DOCK MB671SK-BB Tray-less 3.5" SATA I & II Mobile Rack Removable Hard Drive Kit - Retail
• 1 x  Intel Core 2 Duo E8400 Wolfdale 3.0GHz 6MB L2 Cache LGA 775 65W Dual-Core Processor - Retail
• 2 x Kingston 4GB (2 x 2GB) 240-Pin DDR2 SDRAM DDR2 800 (PC2 6400) Dual Channel Kit Desktop Memory Model KVR800D2N5K2/4G - Retail
• 4 x Seagate Barracuda 7200.11 ST31500341AS 1.5TB 7200 RPM SATA 3.0Gb/s 3.5" Internal Hard Drive (bare drive) - OEM
• 2 x MASSCOOL FD08025B1M3/4 80mm Case Fan - Retail
• 1 x Antec 761345-75120-9 120mm Case Fan - Retail
• 1 x Rosewill RCR-IC001 40-in-1 USB 2.0 3.5" Internal Card Reader w/ USB port / Extra silver face plate - Retail
• 1 x LG 22X DVD±R DVD Burner with LightScribe Black SATA Model GH22LS30 - OEM
• 1 x SYBA SD-SA2PEX-2IR PCI Express SATA II Controller Card - Retail

The secondary SATA controller is only needed if you're going over the number of drives the motherboard supports, and likewise the power splitters. If you were buying all new 1.5TB drives you'd likely not need this. Obviously the memory card reader is only if you need it. But who wants a floppy drive anymore?

Happy building!

This is a neat enhancement to postfix for reducing spam by attacking its economics: making sure it speaks SMTP properly.

A spammer gets paid by the message delivered. So, it's in his interest to flood them out as quickly as possible. Because of this, they rarely implement mailers which negotiate the SMTP connection politely - they simply open the TCP connection and start sending.

When an SMTP client doesn't respect the proper-back-and forth postfix expects, it'll flag it as unauthorized 'pipelining' - for example when multiple messages are sent in succession, but which would otherwise be OK.

We can take advantage of this by forcing the issue, and increasing the odds a spammer will make this mistake by waiting just a second between establishing the TCP connection and telling the spammer we're ready to take mail. A loaded mail server may behave this way anyway, so it's not outside the norm and the resource consumption is minimal, but it attacks the economics of spamming.

In your main.cf file, you would add to smtpd_client_restrictions something like this:

          smtpd_client_restrictions =
                  permit_sasl_authenticated,
                  permit_mynetworks,
                  check_client_access hash:/etc/postfix/access-client,
                  sleep 1,
                  reject_unauth_pipelining
          
          

We accept all of our own users' connections (interactive ones, perhaps) right away, and if the sender is totally unknown to us, we wait for just a second. Then we reject any unauthorized pipelining. The log will show something like this:

May 6 10:49:00 mailhub postfix/smtpd[8965]: NOQUEUE: reject: RCPT from unknown[10.1.2.3]: 403 4.5.0 spamtarget@example.com: Recipient address rejected: Improper use of SMTP command pipelining

when the spammer attempts to just send.

It's worth noting that this method may not scale to very large installations, as those one second delays may be too much. But for the average-sized postfix install, it can make yet another dent in the spam deluge. Where it does consume 'too many' resources, one must weight the cost of computing resources vs. the time cost of dealing with yet another spam.