The BFC Computing Weblog

Rescuing a Broken pfSense Install

Bill McGonigle — Wed, 03 Mar 2010 13:54:00 -0500

They don't make flash drives like they used to. I've seen several pfSense installs fail recently due to drives flaking out and the wear-leveling not working as advertised.

Of course, you make regular backups of your config file, but in case you forgot, we can probably rescue your config file off of a disk image. Re-installing pfSense doesn't take too long, but rebuilding a working config file can take many hours, so rescuing is preferential.

This script has worked for me with dozens of file versions, but one can imagine scenarios with fragmented files where it would fail. There's nothing fancy going on here (this was hacked up with a client standing in my office with a broken pfSense box), but it might prove useful in a pinch.

#!/usr/bin/perl -w
use strict;
use warnings FATAL=>'all';

=comment
pfsense_extract.pl - extract pfSense configs from an input stream
(c) 2010 BFC Computing, LLC.  Licensed under the same terms as pfSense.

This is useful for taking an image file of a damaged pfSense install
and pulling out config files.  If you can mount the image normally, you
should do that first.

Due to the nature of the filesystem, there are often many copies of a 
config file in a disk image, from each time it was saved.  You will 
find a bunch of output files named: pfsense-config-1.xml, pfsense-config-2.xml,
etc.  You can then use tools like diff to find out which the right one was.

Example: 
   dd if=/dev/sdg of=broken_pfsense_image.dd bs=2M conv=sync,noerror
   strings broken_pfsense_image.dd | perl pfsense_extract.pl

Processing a 1GB image as per the example takes about 20 seconds on a standard 
2GHz desktop machine.
=cut

my $BASENAME='pfsense-config-X.xml';
my $counter = 0;
my ($outfile);
my $do_output = 0;

while (<>) {

    chomp;

    if ($_ eq '<pfsense>') {
	$counter++;
	my $filename = $BASENAME;
	$filename =~ s/X/$counter/;
	open($outfile,">$filename");
	$do_output = 1;
    }

    if ($do_output) {
	print $outfile $_ . "\n";
    }

    if ($_ eq '</pfsense>') {
	close $outfile;
	$do_output = 0;
    }
    
}

Dual Screen vs. MythTV vs. Mouse Focus

Bill McGonigle — Thu, 24 Sep 2009 00:35:00 -0400

There's a problem when running two X-displays with MythTV - some events on the non-Myth screen will steal focus and then the MythTV controls will no longer respond. This thread describes the problem well, but is now closed for comments.

Since then, mouse-switchscreen has been written, and solves the problem correctly. It's possible to bind the program to a hotkey.

In the end, I found it better to just run one display at a time since I couldn't prevent the focus stealing.

Converting a Windows Vista KVM Virtual Machine to Redhat VirtIO Drivers

Bill McGonigle — Mon, 14 Sep 2009 16:23:00 -0400

Redhat recently released a set of virtualized I/O devices for KVM, the kernel virtual machine. This short post will outline a method of converting a Windows Vista install (on KVM) to the new drivers using Virt-Manager. It has been tested on Fedora 11.

Make sure Vista VM is up to date on patches and the disk is error free.
Download drivers from Redhat network or here.
Mount the .iso file as a CD-ROM device.

Now you might think you can use the ‘Add Hardware Wizard’ here and add the drivers, add the hardware, and be good. I did. I wound up with an unbootable disk. Apparently Vista’s autodetection is required in this process. So…

Add a new network device of type ‘virtio’. Vista will do its “you’ve got hardware” routine and run you through all of its wizards. When it asks you for drivers, point it at the i386/2008 directory on the driver disc image. Yes, Yes, OK, Yes, Really, Continue, etc.
Shutdown the VM and remove the old ethernet controller. Boot up Vista and make sure the network works. You can conceivably skip this step for now if you want to make troubleshooting harder.
Add a new Storage controller. Leave the existing one as-is for now. You’ll have to pick a disk image you’re not using right now, or make a new one. Anything is fine, we’re not going to ever use it inside Vista. Do the driver dance again.
Shutdown Windows. Remove the storage controllers, and add a new one, type ‘virtio’, with your normal hard drive image. Take care of the old ethernet controller here too, if you ignored my previous advice.
Boot Windows normally. It should now be coming up on VirtIO disk and network drivers. If you get a bluescreen or a plea to use the RepairCD, something went wrong. Use the repair CD to restore to a previous restore-point and try again.

If anybody knows where to find a sound driver, please leave a comment!

Firefox Crashes on Fedora 11

Bill McGonigle — Thu, 28 May 2009 21:38:00 -0400

For folks who are running the current development, or soon-to-be-just-released Fedora 11, you might find Firefox to be very crashy. It's not because it's the semi-controversial 3.5b4 version (which is excellent), it's because of a buggy library.

I'm running it with the Tree Style Tab and NoScript extensions, and can get a crash half the time when Session Restore is running, and almost all the time when I allow a site in NoScript.

If you run firefox from the console, so you get the debug messages, you'll see:

cairo-ft-font.c:554: _cairo_ft_unscaled_font_lock_face: Assertion `!unscaled->from_face' failed

when the crash happens. I tracked this down through the Mozilla and Freedesktop bug systems to a problem with the Cairo graphics engine improperly disposing of fonts which it didn't own, for which a fix was incorporated last December. However, the version of Cairo shipping in Fedora 11 is older than that.

So, I applied the simple patch, fixed up the .spec, and put up some new RPM's for i386 and an SRPM for hackers and x86_64 users to build (rpmbuild --rebuild cairo-1.8.6-3.fc11.src.rpm).

I haven't tried cross-compiling from i386 to x86_64 before, and --target=x86_64 doesn't work, so if anybody can tell me how to do that short of learning mock, please leave a comment and I'll put up RPM's for that too.

The Redhat bug is here. Hopefully it gets accepted soon.

Quiet Rackmount Server w/ Lots of Storage

Bill McGonigle — Wed, 20 May 2009 16:43:00 -0400

I recently had the power supply fail on my SOHO server, which was a mongrel of old parts, far too many USB cables, and was pretty darn slow. It was also very expensive to run, having a Pentium IV in it, the worst of Intel's line.

My goals for a new server were:

quiet
energy efficient
virtualization support
lots of storage
easy to take backups offsite
rackmount
budget-friendly.

After poking around NewEgg for a while (I think I enjoy shopping there a bit too much) I came up with a list of parts (after reading many of the helpful reviews), and I have to say I couldn't be happier with the system.

It's almost inaudible, runs at about 105W under normal load, has seven hard drives in it, of various capacities, fits in my rack, has a hot-swap drive for off-site backups, and runs Fedora 10 like a charm. The case is especially nice to work inside, and is of higher quality than you'd expect for the price.

I'm acually using the 2.66GHz version of the Core2Duo, but they don't seem to make that anymore - 3.0GHz seems to be the low-end. It's worth noting here that most of the commercial server builders try to force you into the Xeon line with a rackmount server and those are both more expensive and more power hungry than the Core2Duo and Core2Quad lines. Get what you really need, keeping in mind that virtualizing multiple systems onto one is a huge energy win.

Additionally, I got a cooler from BestBuy (surprisingly their in-stock cooler is the nicest I've found) and used Arctic Silver 5 thermal compound to bond the CPU. Plus a bunch of SATA cables I have in a box (they seem to spontaneously generate in there). The whole package comes in under $1200 even if you have to buy every part. Compare at fifty percent more to purchase pre-assembled.

Here's the parts list:

The secondary SATA controller is only needed if you're going over the number of drives the motherboard supports, and likewise the power splitters. If you were buying all new 1.5TB drives you'd likely not need this. Obviously the memory card reader is only if you need it. But who wants a floppy drive anymore?

Happy building!

Reducing Spam with SMTP Validation on Postfix

Bill McGonigle — Wed, 06 May 2009 10:50:00 -0400

This is a neat enhancement to postfix for reducing spam by attacking its economics: making sure it speaks SMTP properly.

A spammer gets paid by the message delivered. So, it's in his interest to flood them out as quickly as possible. Because of this, they rarely implement mailers which negotiate the SMTP connection politely - they simply open the TCP connection and start sending.

When an SMTP client doesn't respect the proper-back-and forth postfix expects, it'll flag it as unauthorized 'pipelining' - for example when multiple messages are sent in succession, but which would otherwise be OK.

We can take advantage of this by forcing the issue, and increasing the odds a spammer will make this mistake by waiting just a second between establishing the TCP connection and telling the spammer we're ready to take mail. A loaded mail server may behave this way anyway, so it's not outside the norm and the resource consumption is minimal, but it attacks the economics of spamming.

In your main.cf file, you would add to smtpd_client_restrictions something like this:

smtpd_client_restrictions =
        permit_sasl_authenticated,
        permit_mynetworks,
        check_client_access hash:/etc/postfix/access-client,
        sleep 1,
        reject_unauth_pipelining

We accept all of our own users' connections (interactive ones, perhaps) right away, and if the sender is totally unknown to us, we wait for just a second. Then we reject any unauthorized pipelining. The log will show something like this:

May 6 10:49:00 mailhub postfix/smtpd[8965]: NOQUEUE: reject: RCPT from unknown[10.1.2.3]: 403 4.5.0 spamtarget@example.com: Recipient address rejected: Improper use of SMTP command pipelining

when the spammer attempts to just send.

It's worth noting that this method may not scale to very large installations, as those one second delays may be too much. But for the average-sized postfix install, it can make yet another dent in the spam deluge. Where it does consume 'too many' resources, one must weight the cost of computing resources vs. the time cost of dealing with yet another spam.

Mac OS X Keychain Export Tool

Bill McGonigle — Mon, 02 Mar 2009 23:34:00 -0500

A Mac user might want to export his Keychain passwords and notes for several reasons - using a third-party password manager on Mac OS X, creating a time-resistant backup of passwords, printouts of passwords for the safe-deposit box or attorney, or switching to another operating system.

There's no easy way to do this. Keychain Access only allows you to export certificates, and Apple recommends backing up the Keychain database files, which accomplishes none of the above goals and promotes lock-in.

The keychain code is itself open source, but I couldn't find it compiled for another platform anywhere. I assume that enough of the OSX toolchain is required to make this infeasible, though likely not impossible. Still, it's not there.

Fortunately, I ran across an Applescript that uses Keychain Scripting to create a text file from a user's login Keychain. Unfortunately, it didn't do a bunch of things I thought were required for moving my passwords to a Linux machine, so here's the delta:

version 2009030201:

handle all keychains
handle all key types
handle comments and descriptions
handle errors
trim dangling whitespace
write to tab delimited format
unlock all keychains first, so the mad tapping won't hit 'cancel'
add username to filename
replace carriage returns/newlines in text fields with spaces
use unix line endings in output file

and some general code cleanup. I'm assuming the sample code is in the public domain and releasing this version under GPLv2+. Please improve this and comment here when you do or send changes back. If you own the original code and feel this is improperly licensed, let me know ASAP.

I've run this out of Script Editor - the advantage there is it's easy; the disadvantage is double-confirming every keychain access, one for Script Editor, one for Keychain Scripting. Terribly time consuming. I suspect if you compile this it'll eliminate the first half.

I've set this to open all the keychains first. Otherwise when hitting "allow, allow, allow" you might hit 'cancel' if it asks to unlock a keychain. If your keychain is big enough you might not get through the whole thing before the keychain unlock times out, so be careful.

Your minutes of tapping on the mouse button like a human waiting for a treat will be rewarded with a ~/Desktop/Passwords-yourusername file. It'll be easy to then process with other scripts, importable into databases or spreadsheets for further manipulation. I'll leave it up to you to be smart and not leave this password file sitting around in some unencrypted/unprotected location for any longer than absolutely necessary. If it gets stolen you're probably up a creek, right? So, be careful, only aim at what you intend to kill.

Download KeychainExport.

Portable Computer States

Bill McGonigle — Tue, 17 Feb 2009 12:15:00 -0500

Here's a technology idea: combine a solid-state flash drive, a synchronization engine, advanced virtual memory techniques, and a portable hardware abstraction layer to create a portable computer state device.

The idea would be like this: you have a small hardware device that you bring with your anywhere. When you plug it into one of your computers, it would synchronize the filesystem states, restore memory images, and resume your computing environment the way you left it at the last location.

It's roughly equivalent to the idea of network computers, except you don't need the ubiquitous ultra-high-speed Internet that doesn't really exist (when wireless gigabit is pervasive, this would become passe).

Current reasons this can't work, using linux as the obvious OS to start with, include the lack of an abstract HAL (root drive, home drive, etc) and the lack of virtual-memory restore on a per-process basis. Lots of the other parts exist already.

Initial limitations would probably be a restriction to the same hardware architecture (x86, AMD64, ARM, etc), inability to deal with filesystem changes greater than the capacity of the SSD, and an inability to restore stateful network connections (an IP proxy might work around the last one).

One company has made an approach at this experience by running the environment directly on the portable device, but this forfeits local resources and demands power draws unachievable on an external bus (for simple connectivity). That approach may gain viability over time, though, but not yet.

Would you, gentle reader, use such a device?

Preventing Streaming Video Freezes with TCP Buffer Size Adjustments

Bill McGonigle — Mon, 16 Feb 2009 23:11:00 -0500

I've been using streaming video solutions more, since we pared back our satellite TV package at home and have been saving the Netflix allotment for the kids.

But streaming performance has been lacking for me. I've frequently experienced [buffering] and [recalculating bandwidth] messages, and having streams just stop dead and freeze up.

I wondered if Comcast might be playing games, since they have a history of doing so and this competes with one of their other products, so I decided to check out Google's tools that measure this possibility. Comcast is clean.

But the tools did help me find the actual cause: my TCP buffer receive size was set too low. Their network diagnostic tool revealed that 80% of the time my system was responsible for causing the delay, and only 20% of the time was the network at fault. After rejecting many forum suggestions I found as bizarre, I came across a decent O'Reilly article, which linked to an LBNL site with this recommended setting (for Mac OS X):


sysctl -w net.inet.tcp.win_scale_factor=8
sysctl -w kern.ipc.maxsockbuf=16777216
sysctl -w net.inet.tcp.sendspace=8388608
sysctl -w net.inet.tcp.recvspace=8388608

This increases the send and receive buffers to 8MB each and adjusts the kernel ipc buffer to accommodate. The first line is obsolete as of at least 10.4.11, which my video streaming system is on.

The last three lines above are a good way to test, and for permanence, create a file, /etc/sysctl.conf , with just the parameters, like this:


kern.ipc.maxsockbuf=16777216
net.inet.tcp.sendspace=8388608
net.inet.tcp.recvspace=8388608

After setting that, my videos are all streaming without errors and the Google test shows that now the network is my delay 80% of the time and my client side none.

Recent OSX (10.5.x) and Linux (2.6.17) have TCP buffer autotuning which might, in some cases, make the above unnecessary. The Linux version only sets 4MB buffers, though, which may or may not be enough depending on your bandwidth delay product. Some experimentation may be in order, look up the proper variables for your kernel version, the above is only tested on xnu 8.11.1.

Intel BIOS ISO image with SATA CD-ROM Drive

Bill McGonigle — Tue, 10 Feb 2009 00:22:00 -0500

Intel thoughtfully has some ISO images of their BIOS flash upgrades, so you don't need to worry about finding the right flash software for your operating system and then timidly hoping that all works OK. You burn the image to a CD and reboot, then it flashes for you (using a FreeDOS/ISOLINUX system).

However, if you have a SATA CD-ROM drive, the device driver in FreeDOS doesn't support that. There is a SATA-compatible FreeDOS driver, but rather than rebuild Intel's ISO, there's an easier solution - make the BIOS emulate an IDE drive.

Go into BIOS Setup (F2 at boot), then Advanced ... Drive Configuration, and set 'Configure SATA as' to 'IDE' (mine was AHCI) and ATA/IDE Mode to 'Legacy'.

Reboot, allow the flash to succeed, then switch your BIOS settings back.

There's nothing wrong with this method, but Intel should highlight it on their download page.