The BFC Computing Weblog

They don't make flash drives like they used to. I've seen several pfSense installs fail recently due to drives flaking out and the wear-leveling not working as advertised.

Of course, you make regular backups of your config file, but in case you forgot, we can probably rescue your config file off of a disk image. Re-installing pfSense doesn't take too long, but rebuilding a working config file can take many hours, so rescuing is preferential.

This script has worked for me with dozens of file versions, but one can imagine scenarios with fragmented files where it would fail. There's nothing fancy going on here (this was hacked up with a client standing in my office with a broken pfSense box), but it might prove useful in a pinch.

          #!/usr/bin/perl -w
          use strict;
          use warnings FATAL=>'all';
          
          =comment
          pfsense_extract.pl - extract pfSense configs from an input stream
          (c) 2010 BFC Computing, LLC.  Licensed under the same terms as pfSense.
          
          This is useful for taking an image file of a damaged pfSense install
          and pulling out config files.  If you can mount the image normally, you
          should do that first.
          
          Due to the nature of the filesystem, there are often many copies of a 
          config file in a disk image, from each time it was saved.  You will 
          find a bunch of output files named: pfsense-config-1.xml, pfsense-config-2.xml,
          etc.  You can then use tools like diff to find out which the right one was.
          
          Example: 
             dd if=/dev/sdg of=broken_pfsense_image.dd bs=2M conv=sync,noerror
             strings broken_pfsense_image.dd | perl pfsense_extract.pl
          
          Processing a 1GB image as per the example takes about 20 seconds on a standard 
          2GHz desktop machine.
          =cut
          
          my $BASENAME='pfsense-config-X.xml';
          my $counter = 0;
          my ($outfile);
          my $do_output = 0;
          
          while (<>) {
          
              chomp;
          
              if ($_ eq '<pfsense>') {
          	$counter++;
          	my $filename = $BASENAME;
          	$filename =~ s/X/$counter/;
          	open($outfile,">$filename");
          	$do_output = 1;
              }
          
              if ($do_output) {
          	print $outfile $_ . "\n";
              }
          
              if ($_ eq '</pfsense>') {
          	close $outfile;
          	$do_output = 0;
              }
              
          }
          

A Mac user might want to export his Keychain passwords and notes for several reasons - using a third-party password manager on Mac OS X, creating a time-resistant backup of passwords, printouts of passwords for the safe-deposit box or attorney, or switching to another operating system.

There's no easy way to do this. Keychain Access only allows you to export certificates, and Apple recommends backing up the Keychain database files, which accomplishes none of the above goals and promotes lock-in.

The keychain code is itself open source, but I couldn't find it compiled for another platform anywhere. I assume that enough of the OSX toolchain is required to make this infeasible, though likely not impossible. Still, it's not there.

Fortunately, I ran across an Applescript that uses Keychain Scripting to create a text file from a user's login Keychain. Unfortunately, it didn't do a bunch of things I thought were required for moving my passwords to a Linux machine, so here's the delta:

version 2009030201:

• handle all keychains
• handle all key types
• handle comments and descriptions
• handle errors
• trim dangling whitespace
• write to tab delimited format
• unlock all keychains first, so the mad tapping won't hit 'cancel'
• add username to filename
• replace carriage returns/newlines in text fields with spaces
• use unix line endings in output file

and some general code cleanup. I'm assuming the sample code is in the public domain and releasing this version under GPLv2+. Please improve this and comment here when you do or send changes back. If you own the original code and feel this is improperly licensed, let me know ASAP.

I've run this out of Script Editor - the advantage there is it's easy; the disadvantage is double-confirming every keychain access, one for Script Editor, one for Keychain Scripting. Terribly time consuming. I suspect if you compile this it'll eliminate the first half.

I've set this to open all the keychains first. Otherwise when hitting "allow, allow, allow" you might hit 'cancel' if it asks to unlock a keychain. If your keychain is big enough you might not get through the whole thing before the keychain unlock times out, so be careful.

Your minutes of tapping on the mouse button like a human waiting for a treat will be rewarded with a ~/Desktop/Passwords-yourusername file. It'll be easy to then process with other scripts, importable into databases or spreadsheets for further manipulation. I'll leave it up to you to be smart and not leave this password file sitting around in some unencrypted/unprotected location for any longer than absolutely necessary. If it gets stolen you're probably up a creek, right? So, be careful, only aim at what you intend to kill.

Download KeychainExport.

Here's a technology idea: combine a solid-state flash drive, a synchronization engine, advanced virtual memory techniques, and a portable hardware abstraction layer to create a portable computer state device.

The idea would be like this: you have a small hardware device that you bring with your anywhere. When you plug it into one of your computers, it would synchronize the filesystem states, restore memory images, and resume your computing environment the way you left it at the last location.

It's roughly equivalent to the idea of network computers, except you don't need the ubiquitous ultra-high-speed Internet that doesn't really exist (when wireless gigabit is pervasive, this would become passe).

Current reasons this can't work, using linux as the obvious OS to start with, include the lack of an abstract HAL (root drive, home drive, etc) and the lack of virtual-memory restore on a per-process basis. Lots of the other parts exist already.

Initial limitations would probably be a restriction to the same hardware architecture (x86, AMD64, ARM, etc), inability to deal with filesystem changes greater than the capacity of the SSD, and an inability to restore stateful network connections (an IP proxy might work around the last one).

One company has made an approach at this experience by running the environment directly on the portable device, but this forfeits local resources and demands power draws unachievable on an external bus (for simple connectivity). That approach may gain viability over time, though, but not yet.

Would you, gentle reader, use such a device?

Apple's Snow Leopard (10.6) operating system is due out in the next quarter according to slides shown recently at the LISA conference. It adds a small handful of features but it's mainly an architecture, performance, and bugfix release. Leopard (10.5) is pretty buggy and Apple readily admits it's not what an OS should be. So they're coming out with an update less than a year and a half since the last one, which is by most counts what Leopard should have been. This isn't really disputed, even Apple's name isn't for a new cat, this is the one with all the 'marks cleaned off'.

OK, so it's great that Apple's getting everything squared away so quickly, right? Yeah, it is if you've got recent hardware.

But what if you have a computer that was purchased in, say, the first half of 2006? It's going to have a PowerPC processor in it, and Snow Leopard doesn't support PowerPC. OK, so then you can run Leopard, which does support PowerPC. But, wait, Leopard is buggy, that's why they're fixing it.

OK, so you can run Tiger (10.4). Well, no, if you're going to be connected to a network you'd be foolish to do that; Apple only issues security updates for the current and previous versions of its OS, and with 10.6, 10.4 will go by the wayside. Within months there will be public exploits for your 10.4 machine available and the time to your machine being compromised is just a roll of the dice.

"Wait," you may be saying, "my machine is less than three years old and it's now unsupported?" "It's still under AppleCare warranty and I can't even get security updates?"

Yep, and there we see the tactical brilliance behind splitting the Leopard and Snow Leopard releases - Apple gets to book its revenue early on a not-ready OS, beat Microsoft to the market, and save a ton of money really only supporting one majoor version of its operating system. So, this doesn't really work out well for you? Just buy a new Mac, they're probably not going to do this again in three more years. Right?

This may be a dangerous gamble for Apple in a recessionary economic period, so perhaps they'll do the right thing and simultaneously keep their customer base. If not, Ubuntu 8/PPC isn't eligible for a commercial support contract but it'll run on your Mac and its security updates will be current for another two years. At that point your machine will be five years old and you can keep it around with debian or netbsd or if we're coming out of the downturn get yourself a brand new machine. By then you'll be so used to Ubuntu you'll have broad purchase options.

This blog is on a shiny (ok, flat black, really) new server, with much faster everything, and it’s a half-U low-power para-virtualized beta product I’m working on (more on that to come, now that I have room for pictures!). Just typing this it’s immediately apparent that there’s way more snappy (live preview with round-trips to the server) but there’s bound to be a bug or two somewhere. Please let me know if you see anything amiss.

As mentioned before, all BFC Computing servers are named for real-life heroes, and this one is no exception. stevens.bfccomputing.com is named for Brenda Stevens, a grandmother from Deerfield, N.H., who was killed when a tornado struck her home on July 24th and the building collapsed onto her. As the tornado destroyed her home she held onto her stepson’s baby boy, whom she was babysitting, long enough to keep him from sustaining any more than minor injuries. Mrs. Stevens didn’t survive, but her grandson lives due to her ultimate sacrifice.

GNOME has a good page describing some of the concerns with linking GPL’ed software against the OpenSSL libraries.

Debian believes that as long as OpenSSL and the GPL’ed software are delivered on the same medium or in the same distribution, the GPL’s ‘accompanying clause’ is invoked:

There is some disagreement about what this exception means when the both the GPL program and the GPL incompatible library are shipped as part of the operating system. The “unless that component itself accompanies the executable” clause suggests that the special exception does not cover this case and that it is a violation of the GPL. Debian takes this position on the issue.

There are three reasons I think this is an incorrect interpretation.

• First, the likely scenario envisioned in the GPL is the distribution of a program, by its authors, with another product bundled with it. This is very common, actually, for commercial code and it would be a way to achieve an end-run around the GPL in open-source. A distribution aggregating disparate authors’ work isn’t an attempt at an end-run around the GPL, so the purpose of the accompanying clause isn’t achieved.
• Second, making a distinction between physical media and the Internet has never been an essential part of the GPL (nor should it be), yet a distribution or metadistribution that references an Internet source for OpenSSL rather than including the bits on the media would satisfy the incorrect interpretation of the accompanying clause, but would not benefit the GPL or Free Software in any way. So, this interpretation fails a practical usefulness test.
• Third, and perhaps most importantly, so much of a typical distribution falls outside the purview of the GPL that it’s not sensible to try to apply the terms of a GPL on the scale of a distribution. The GPL is intended, and terribly useful for, software packages that are discrete in their purpose and construction, and its language is properly interpreted in that context. By making broader claims about its reach, the GPL itself stands to be weakened, for if one license can enforce terms on another by mere aggregation the entire concept of licensing fails to be coherent. Only by saying, “The GPL applies to this distro,” can one make the incorrect claim, and so that argument fails a test of common sense.

This will sync any attached iPods if iTunes is running:

          #!/bin/sh
          osascript <<_END_OF_UPDATE
          tell application "Finder"
                  set processList to get name of every process
                  if processList contains "iTunes" then
                          tell application "iTunes" to update (every source whose kind is iPod)
                  end if
          end tell
          _END_OF_UPDATE
          

It can be called from a cronjob to keep an iPod updated. iTunes ought to just do this automatically.

From Fedora Weekly News:

Daniel P. Berrange laid it out there. “There is pretty much zero chance that Fedora 10 will include a Xen Dom0 host. While upstream Xen developers are making good progress on porting Dom0 to paravirt_ops, there is simply too little time for this to be ready for Fedora 10. So if you need to use Fedora 10 as a host, then KVM is your only viable option at this time. If you can wait for Fedora 11 (or use RHEL-5 / CentOS-5) then Xen may be an option for you.”

The basic issue is that the the virtualization groups are getting together on a standard kernel interface, paravirt_ops, and the Xen folks aren’t ready yet. Fedora 10 has a hard deadline, and the Xen group isn’t likely to make it in time.

Why Fedora 10 is important, is that Fedora 8 will stop getting security updates once Fedora 10 is released, per policy. When Fedora 9 came out, the Fedora Project told Xen users to hold off on Fedora 9, that Fedora 10 would have the Xen pieces. Had it told users at that time stop using Fedora they might have had a reasonable opportunity to plan for a chance, but at this point they have only weeks to get their systems off onto another operating system.

While RHEL/CentOS have always been billed as the ‘stable’ platform, many community-minded systems administrators run Fedora as a way to help find problems and improve the codebase. In return, they get more up-to-date packages and an expectation of being able to upgrade to new releases as they’re available.

Now, for the first time I can recall, Fedora has dropped its end of the bargain, for any folks who are using Xen virtualization. I know I don’t deploy any new servers these days that don’t use virtualization, and I doubt I’m highly unusual in that regard. Fedora thus stands to lose a great number of users, i.e. testers, trust and goodwill. After all, if one of the two major kernel flavors can get the axe, just about anything else can too. It raises the question of what Fedora provides, as a distribution. Sure, we understand that the upstream kernel isn’t ready, but is Fedora willing to merely have its feature set dictated by outside parties? This is as much a function of Fedora’s release-by-date rather than release-when-ready policy. They want to release approximately every six months, come hell or high water, and while momentum is desirable in a vacuum, sometimes the community might deserve some consideration as well.

The current expectation is that the Dom0 bits will be in kernel-2.6.28. By all expectations, this release will come about 90 days after 2.6.27 is released, or approximately mid-January, if not sooner. One would hope since the Xen kernel no longer requires a separate RPM package that when Fedora adopts 2.6.28 as its primary kernel (early Feb ‘09, perhaps) that Xen Dom0 support will re-appear.

So, to arrive at a detente, the most practical approach would be to extend Fedora 8 security updates until such a time as a Xen-Dom0 kernel is integrated into Fedora. Without argument, this will consume precious development hours among the Fedora development community. And Fedora can legitimately argue that it’s ‘for experimental use only’ and plausibly get away with not doing so. However, the practical reality of choosing this path is to lose community-sourced testing hours orders of magnitude larger than would be saved by continuing the updates. And since RHEL 6 will require a stable Xen base for its release, Fedora 10 with Xen is going to be very important to have well shaken-out for RHT.

On an RPM-based system, yum-utils provides a utility called ‘package-cleanup’. It has two useful options:

–orphans shows RPM packages that do not belong to any currently-configured repositories, and:

–leaves shows RPM packages for which there are no dependencies; that is removing them won’t trigger the removal of other packages. By default it’s concerned with libraries, but –all removes that restriction.

So, ideally you’d like to run:

package-cleanup –orphans –leaves –all

to get a list of all the packages you might want to consider for cleanup, say before or after an upgrade. But package-cleanup doesn’t support that.

So, here’s a little perl script, called leavesorphans.pl on my system that will run package-cleanup twice and print for you the intersection of the two sets:

#!/usr/bin/perl -w
          use strict;
          use warnings FATAL=>'all';
          
          use Data::Dumper;
          
          my @orphans = `package-cleanup --orphans`;
          my @leaves = `package-cleanup --leaves --all`;
          
          my (%orphans,%leaves);
          foreach my $orphan (@orphans) {
              $orphans{$orphan} = 1;
          }
          foreach my $leaf (@leaves) {
              $leaves{$leaf} = 1;
          }
          
          my (@matches);
          foreach my $orphan (keys %orphans) {
              foreach my $leaf (keys %leaves) {
                  if ($orphan eq $leaf) {
                      push (@matches,$orphan);
                      delete $leaves{$leaf};
                  }
              }
          }
          
          
          foreach my $match (@matches) {
              if ($match !~  m/Setting up yum/) {
                  print $match;
              }
          }
          
          

I recently ran it and found a few packages that were lingering on my system since Fedora Core 4, just wasting system resources. If all of your proper packages belong to a repository you can simply pipe the output of the command to xargs rpm -e. I’m not quite that slick, so I manually reviewed the list and kept the packages I had installed by hand.

Fraser Speirs, former iPhone developer, had his application rejected by Apple on grounds that it might compete with iTunes. Unfortunately for Speirs and every other developer out there, you have no way of knowing if an app will be allowed by Apple until the last step in the development process, unless the app already exists in the Store. This raises the risk for investing in an iPhone app tremendously, meaning few businesses will make the investment, especially if their application is cutting edge. How would you like to invest $200K in an iPhone development project to have it turned back by a fickle screener?

But, I mean, who could have seen this coming in a completely closed and proprietary development environment?

Rule #3 of business - never have your business completely dependent on another business. A well-diversified, well-capitalized business might be able to take this risk, but the majority won’t. Android and Maemo are waiting.