Java shell tips

In which jar file does the class, CLASSNAME, reside?

for i in lib/*.jar; do jar tf $i | grep CLASSNAME >/dev/null && echo $i; done

To recursively search through all jars in the current directory...

find . -name "*.jar" | while read i; do jar tf $i | grep CLASSNAME >/dev/null && echo $i; done

And here's a handy little Python script that recursively "explodes" J2EE artifacts into their constituent components. If they're already exploded, it'll "implode" them. This is a great tool for quickly updating deployment descriptors at the command line.

#!/usr/bin/env python
#
# Each file passed on the command line that is a valid j2ee archive
# (war, jar, ear, sar) will be either exploded if archived or archived
# if exploded.

import sys,os
from zipfile import is_zipfile,ZipFile
from os.path import isdir,exists,realpath,dirname,basename,join
from shutil import rmtree

# The result of this try block is a function named 'mktempdir' that
# makes a temporary directory and returns its path.  Python 2.3
# provides tempfile.mkdtemp, which does exactly what we need, but we
# must implement it ourself in older versions.
try:
    from tempfile import mkdtemp as mktempdir
except ImportError:
    from tempfile import mktemp
    def mktempdir():
        path = mktemp()
        os.mkdir(path)
        return path

# The result of this try block is a function called 'move', which
# moves a file or directory, possibly across filesystems.  Python 2.3
# provides shutil.move, but older versions do not, so we revert to a
# system call.  The os.rename function will not work across
# filesystems.
try:
    from shutil import move
except ImportError:
    def move(src,dest):
        os.system("mv "+src+" "+dest)
        
# The presence of one of these indicates a J2EE artifact.
DEPLOYMENT_DESCRIPTORS = ("META-INF/application.xml",     # ear
                          "WEB-INF/web.xml",              # war
                          "META-INF/ejb-jar.xml",         # jar
                          "META-INF/jboss-service.xml")   # sar

def explode(file):
    '''Recursively unpack the contents of a J2EE file into a directory of same name.'''
    file = realpath(file)
    tmp = mktempdir()
    os.system("cd "+tmp+"; jar xMf "+file)
    [explode(join(tmp,f)) for f in os.listdir(tmp) if is_j2ee_zip(join(tmp,f))]
    os.remove(file)
    move(tmp,file)
    print "Exploded "+file

def implode(dir):
    '''Recursively package the contents of a directory into a J2EE file with same name.'''
    dir = realpath(dir)
    [implode(join(dir,f)) for f in os.listdir(dir) if is_j2ee_dir(join(dir,f))]
    tmp = mktempdir()+"/"+basename(dir)
    os.system("cd "+dir+"; jar cMf "+tmp+" *")
    rmtree(dir)
    move(tmp,dir)
    os.rmdir(dirname(tmp))
    print "Imploded "+dir

def is_j2ee_dir(path):
    '''Return True if directory contents are structured as a valid J2EE artifact.'''
    if isdir(path):
        for dd in DEPLOYMENT_DESCRIPTORS:
            if exists(join(path,dd)): return True

def is_j2ee_zip(path):
    '''Return True if path refers to a valid J2EE artifact.'''
    if is_zipfile(path):
        files = ZipFile(path).namelist()
        for dd in DEPLOYMENT_DESCRIPTORS:
            if dd in files: return True

for path in sys.argv[1:]:
    if is_j2ee_zip(path):
        explode(path)
    elif is_j2ee_dir(path):
        implode(path)

Stupidest Guitar Chord: D# minor

I love the Beatles' song, "I'm Only Sleeping", especially the very first chord, played with a beautiful, exaggerated "up-stroke" for emphasis. Strictly speaking, listening to the recording, the chord is a D# minor, but a guitar just ain't made to play that! For a guitarist, a D#m is a pretty stupid chord.

Why? Because E minor sounds so much better! The open strings allow it to ring long and full, using the lowest note possible in standard tuning. And it's easier to play! And it's only a half-step away! A mere semi-tone!

So either George Martin altered the speed of the recording, or maybe the guitars were tuned down a half step. But either way, nobody was playing a D#m on their guitar that day. The Beatles weren't stupid.

So you shouldn't be, either. But it's a pain to detune your guitar just to play along, right? Thankfully, there's an easier way: Audacity, a free audio editor for Linux, can do magic for the budding guitarist. It can change the pitch of a song without changing its tempo. Conversely, it can slow the tempo without changing the pitch, which is really handy for figuring out solos. What was impossible with analog recordings is now amazingly simple with digital audio.

Nobody should be forced to play D# minor.

Remote Monitoring with Ruby and sshfs

At work I've been doing some performance analysis as of late. Specifically, I've been combing JBoss server.log files recording the time it takes various transactions to complete. Admittedly, I'm fighting a little with Werner Heisenberg because configuring the app to produce the log messages on which I depend will affect performance, but my interest is in relative rather than overall performance, i.e. I'm looking for potential bottlenecks.

Still, I don't want to take up any more resources from the app server's CPU than absolutely necessary. The Ruby script that processes the log files obviously makes much use of regular expressions. Here's an example of a Request object from my script.

  Request.new("Typical request") { |request|
    request.expect /Processing Request/
    request.expect /No existing Session found/
    request.reject /ERROR/
    request.reject /reaping required/
    request.expect /Processing message took \d+ millis/
  } 

Here's an example of a line from the server.log:

2007-05-09 16:28:51,511 INFO  [UDPListener] (Thread-16) UDPListener thread running

The Ruby script organizes the log messages by thread id. If it finds matches for each expected regular expression in the correct order without finding any it should reject, it records the total time for the Request. When finished, it reports various statistics (average, mean, stddev, etc) for each type of Request it found in the log.

Ideally, I would run that script on a log after a particular load test was performed, but hey that's no fun! I'd rather see how the server is doing in realtime, monitoring its performance while it's running. Here comes sshfs to the rescue!

With sshfs, I can mount the file system of the app server on a remote machine running my Ruby script. The app server doesn't need Samba or NFS installed, only ssh, and what respectable app server wouldn't have sshd running?

With the server.log mounted via sshfs, far fewer CPU resources are taken up by the analysis script, no more than if you were "tailing the log" yourself while the app server was running.

Installing sshfs is very simple on modern Debian-based distros:

  $ sudo apt-get install sshfs

Depending on the distro, you may need to add your user to the 'fuse' group or modprobe fuse, or chmod /dev/fuse, or some other such minor detail, but once installed, remote, realtime log file analysis is a breeze:

  $ sshfs appsrvr:/ /mnt/appsrvr
  $ cd /mnt/appsrvr/usr/local/jboss/server/default/log
  $ tail -f server.log | statistics.rb

So you use local CPU resources to parse a remote log file. Cool, huh?

From Gentoo+Postfix+Courier to Debian+Exim+Dovecot

Ok, let's say -- hypothetically, of course -- you chose Gentoo for your home email server a few years ago. You found one of Gentoo's great docs on setting up Postfix, Courier and SpamAssassin. You accepted all the reasonable default options during the install, and things seemed to work great.

Over time, you began to take the little email server for granted. Oh sure, you'd occasionally login and "emerge" some security updates, but the dang thing just worked. Why bother it?

Of course, entropy is a cold bitch, and "for granted taking" turned to neglect. After a while, you began to forget all the arcane portage commands, having to use rpms at work and happily experimenting with Ubuntu's adopted dpkg on your desktop. In the beginning, spam was under control, but it's a constant battle, and forgetting your distro's packaging commands won't exactly help you in that fight.

And then terror struck: the Gentoo maintainers decided to release a portage update that was incompatible with the version running on your little email server. And at that precise version did your server remain for the rest of its life. The prospect of recompiling your entire system was just too much to bear. You decided it was a good time to look for another distro, maybe in a couple weeks or so, when you're not so busy...

Well, a couple of weeks turned into a couple of years. And over that time, you came to realize the benefits of the maildir storage format over the traditional mbox one, the latter being one of those reasonable defaults mentioned above. More importantly, you began to appreciate apt/dpkg as "portage without all the compiling", and you became aware of Debian's reputation for server stability.

One day when the spam/ham ratio was particularly high, you decided the time had finally come to switch the ol' home server from Gentoo to Debian. You decided you'd build a new box, transfer the family email over and convert it to the Maildir format. Just because you're a total dumbass-glutton-for-punishment, you also decided to switch MTA's (Postfix to Exim) and IMAP daemons (Courier to Dovecot). "Why the hell not?" you wondered.

Um... I'm not sure I can keep up the hypothetical perspective any longer. I'm thinking you were seeing right through it, anyway. Sorry.

So you -- I mean, I -- found a cheap Dell P3 on Ebay for $33+s/h, installed Debian Etch, and then fired up aptitude...

Exim

By default, Debian installs exim4-daemon-light, which is fine for most purposes, but exim4-daemon-heavy is required for integration with most spam/virus detection packages and, despite its name, it's really not that heavy.

  # aptitude install exim4-daemon-heavy

Configuring Exim is easy enough, though it took me a while to grok the split vs non-split formats. I wasted a lot of time googling for docs/howtos: I wish I had just started with /usr/share/doc/exim4/README.Debian.gz like the maintainers suggest. I don't know why Google results seem so much more glamorous than the packaged docs.

I went through the normal routine:

  # dpkg-reconfigure exim4-config

I chose non-split, mail sent by smarthost, received via SMTP, and accepted mail for *.internal.domain;internal.domain;external.domain. Your -- my? -- domain names will differ, of course. I chose to relay mail for my local network, set the name of the outgoing smarthost, hid the local mail names making only my external domain visible, and made sure to set Maildir as my delivery format.

I then created user accounts for all my family members, and sent them some test mail from a box configured to relay outgoing mail to my new server. In addition to testing the above config, it also caused the Maildir structures to be created in the users' home directories. Easy-peazy-lemon-squeazy.

Something else cool: I can add entries to /etc/aliases and exim will pick them up immediately without restarting or rerunning newaliases or some such. I do that for extended relatives so I don't have to remember their real email addresses, only their alias, e.g. unclebob@crossleys.org. Let me know if you want one.

Dovecot

So delivery is working. Now I need an IMAP server for retrieval.

  # aptitude install dovecot-imapd

Configuring Dovecot was simple: I just needed to enable plaintext logins (I'm not exposed to the bad ol' Internet) and the imap protocol. Here's a diff of my changes to /etc/dovecot/dovecot.conf:

  21c21
  < protocols =
  ---
  > protocols = imap
  46c46
  < #disable_plaintext_auth = yes
  ---
  > disable_plaintext_auth = no

After saving those changes, a restart is required:

  # /etc/init.d/dovecot restart

imapsync

Now comes the hard part: how to move my email from the old server to the new one? After googling a bit, I came across a utility called mb2md, but its interface is a little weird, seemed to expect files to be in certain places, and seemed more geared toward a single user rather than a group. Somewhat discouraged, I restarted my search, and came across imapsync. This discovery simplified my approach: instead of copying mbox files (both spooled and saved) from the old server to the new and *then* converting them, I could simply encapsulate the format differences behind the IMAP interface, i.e. imapsync allows me to copy email from one IMAP server to another, irrespective of the underlying formats of each! Neat-Oh!

Installing imapsync is easy:

  # aptitude install imapsync

Using it takes some practice. Fortunately it provides a --dry option for just that purpose. Here's an example of my Courier-to-Dovecot imapsync incantations for a particular user. I needed to invoke it twice: once for the main inbox, and again for the categories of saved mail I kept beneath the ~/mail directory on the old server:

  imapsync --host1 oldbox --user1 jim --password1 oldpass \
           --host2 newbox --user2 jim --password2 newpass \
           --folder INBOX
  imapsync --host1 oldbox --user1 jim --password1 oldpass \
           --host2 newbox --user2 jim --password2 newpass \
           --include "^mail/.+"

The nice thing about imapsync is that it's a true sync. You can run it multiple times and it only copies over what's changed. This allowed me to be pretty sloppy about when I actually updated my firewall to send SMTP requests to the new server.

ClamAV

Ok, so we've (we, as in me, not you) accomplished a lot. The only thing remaining is to reject viruses and spam. ClamAV will handle the viruses:

  # aptitude install clamav-daemon

To integrate with Exim, two changes to /etc/exim4/exim4.conf.template were required. I set av_scanner to this:

  av_scanner = clamd:/var/run/clamav/clamd.ctl

And uncommented these lines (presumably put there by exim4-daemon-heavy):

  deny
    malware = *
    message = This message was detected as possible malware ($malware_name).

After running update-exim4.conf, restarting exim and sending a fake virus (as described here), I noticed some "lstat() failed" errors in the mainlog. Smells like a perms problem. To fix that, I did this:

  # adduser clamav Debian-exim
  # /etc/init.d/clamav-daemon restart

SpamAssassin, er... greylistd

At this point in our story, I expected to tell you how easy it was to integrate SpamAssassin with our (my) new setup, but frankly I haven't done it yet.

Instead, I discovered greylisting. This has eliminated 95% of my spam. For this reason, I'm holding off on SpamAssassin integration for the time being.

The greylistd package provides a convenient script that completely idiot-proofs exim4 integration:

  # aptitude install greylistd
  # greylistd-setup-exim4 add

Simple, huh?

Conclusion

That's it. We're done. Whew!

Now I have a stable home email server that I can actually update and try to secure. Sweet.

Linux Podcasts

Since I got my new Meizu, I've been listening to a lot of podcasts, especially those dedicated to Linux. There are a surprising number of them out there, and I've listened to quite a few, but I wanted to share my favorites.

• Linux Reality - Chess Griffin's generous, informative and practical podcast, presumably for newbies but really for anyone interested in Linux
• LugRadio - can't understand them half the time, but usually very funny
• JaK Attack - kind of hit and miss, but nice to hear a female voice
• ITConversations Programming series - not active any longer, and focused on software development rather than linux, but great interviews with industry heavyweights like Paul Graham, Kent Beck, etc
• Linux Action Show - often silly and self-conscious, but not boring and usually informative
• Run Your Own Server - smart cats, more geared toward BSD than Linux, though

Charter DNS still sucks

I set up dnsmasq on the Debian box I had been running bind9 on. It was infinitely easier to configure than a "real" DNS server -- just edit /etc/hosts -- and it perfectly suited the host-naming requirements for my admittedly tiny home network.

As with all things computer, of course, I still had hurdles to overcome, and they all involved difficulties with the commercial operating systems in my network, specifically Win XP and Mac OS X. All the Linux boxes were blissfully easy to setup and predictably functional and even adaptable to the weirdness required by the commercial ones.

As far as the Mac goes, it wouldn't resolve my local domain name, apparently because it ended in .local. I used that suffix because I read somewhere that it was an illegal TLD (top level domain, e.g. .com, .org, .edu, etc). Apparently, Mac's lookupd process eats those names because of the file /etc/resolver/local. I could fix the problem by adding a file for my domain to /etc/resolver, but I decided it was easier to use the suffix .home instead. I'm pretty sure it's an illegal TLD, too.

The WinXP box was sad for another reason. I had initially tried configuring dnsmasq as a DHCP daemon, which was cool, because then I could control all my static IP's in /etc/dnsmasq.conf instead of having to configure each host on the network. But alas, that broke my work VPN client on the WinXP box. I didn't spend any time trying to figure out why, though.

So now my router still acts as the DHCP server, even though the work laptop is the only thing that uses it. The other hosts are configured with static IP's that are replicated in the /etc/hosts file on the host running dnsmasq. Suboptimal, but functional.

Charter DNS sucks

For the most part, I've been happy with my Charter cable service, with two glaring exceptions:

• They won't provide the NFL Network
• Their DNS servers are unreliable, often down, and slow when they're up

I can't do much about the first issue -- other than relocate -- because the cable industry is rife with un-American anti-competition policies, but I stumbled across a VERY EASY, i.e. lazy, solution to the second: install a "caching forwarding nameserver". The performance improvement is ridiculous.

It so happens that Debian's default configuration of DNS (bind9) is to behave just that way: cache-ily and foward-ily. I realize there are other lighter-weight solutions, e.g. nscd, dnsmasq, or pdnsd, but it was just so bone-stupid-simple to install Debian on a spare PC (or VM, of course) and check the "DNS Server" option on the installation's package selection screen. And what better use for the $33 PC I just bought off ebay, right? Incidentally, the shipping cost was only 15 cents less than the bid price, which technically makes it a $65.85 PC, but hey, whenever server TCO is less than $100, I'm happy.

Of course, I now must configure the other computers in my home network to use my new nameserver. For my Ubuntu laptop, I edit /etc/dhcp3/dhclient.conf, and add (uncomment) this line:

  prepend domain-name-servers THE_IP_ADDRESS

Of course, you'll substitute THE_IP_ADDRESS with the real IP address of your $65.85 PC. This results in THE_IP_ADDRESS of my nameserver showing up ahead of Charter's nameservers in /etc/resolv.conf when the laptop obtains its DHCP address at boot. Alternatively, I could use my $65.85 PC as my DHCP server so that no per-client config is necessary at all. If only I wasn't so lazy...

Did I mention how ridiculously fast "the Internet" seems now? All the perceived slowness was due to Charter's name servers, which suck, btw. I mentioned that, right?

Maxtor OneTouch, HFS+ to ext3

I bought a Maxtor OneTouch 300GB external USB/Firewire drive a couple of years ago, mostly to save my iMovie experiments on my wife's Mac, but also to serve as a general backup device for my other Linux boxen. Because I had no box other than the Mac with a firewire port, I hooked it up there and -- due to the normal fear and ignorance computer manufacturers have come to expect from their users -- I ran the supplied installation CD.

That turned out to be a mistake.

The Mac formatted the file system on the drive as HFS+, of course. This was fine, as long as the drive was connected to the Mac. But when I plugged it into a Linux box, it was none-too-happy with HFS+ and would only mount the thing read-only.

So today, I finally got around to reformatting the drive as ext3. This turned out to be way easier than I thought it would. Actually, the hardest part was finding places to temporarily store the files that were on there during the reformatting. Once that was done, I simply plugged the thing into my Ubuntu Edgy laptop, fired up gparted, deleted the old partitions -- for some reason, there were three of them -- and then I created a single ext3 partition.

Using, I assume, some combination of "udev/hotplug/hal/gnome" magic, Ubuntu recognized the drive when I power-cycled it and automounted it (/dev/sdb) to /media/usbdisk. Not liking that name, I realized gparted hadn't prompted me for a volume label, so I then umounted it and ran:

  $ e2label /dev/sdb1 maxtor

I guess I'm in the right group to allow me to do that. Yanking and reinserting the USB cable now caused the ghosts in the machine to mount the drive at /media/maxtor. Perfect!

But alas, I couldn't write to it. :-(

  $ sudo chmod 777 /media/maxtor

There, all better. Now I'm off to move all my files back to it, and hook it up to a "real" file server. In doing so, I'll no longer benefit from all the udev automounting hocus pocus, because I'm pretty sure that's a gnome-ish feature. If I'm not logged into gnome, nothing shows up in /media when I connect the drive, though /var/log/messages does show the device being detected as /dev/sdb. I'm pretty sure I'll need to put something like the following in /etc/fstab:

  LABEL=maxtor  /mnt/maxtor  auto  defaults  0 0

I'll let you know if that doesn't work out.

My New Meizu

My wife bought me a MiniPlayer for Christmas, a portable media player made by Meizu, a Chinese manufacturer. I love the darn thing.

My wife has owned two iPods, an original and a Nano. I hate the darn things. I think they're unreliable and flaky. Oh, and overpriced.

But I have a large music library, and my new job has a long commute, plenty enough time to listen to some cool, geeky podcasts. She asked if I wanted one for Christmas. I said sure, but only if it works with Linux and plays ogg files. So she found the Meizu, which does that and much more, including videos, which I wasn't expecting. So far, I've been very impressed. There are plenty of reviews on the web, and YouTube has some demonstration videos.

If you're looking for a great DAP for Linux, I heartily recommend it. Amarok works well with it, and you can use mencoder to format your videos to play on it. It ships with the Windows-only VirtualDub, but the following mencoder incantation has worked great for me. SRC is the filename to be converted, and TGT is the name of the output file, sans extension.

CMD="mencoder $SRC \
-quiet \
-vc ffmpeg12,mpeg12, \
-ofps 20 \
-vf scale=320:240:0:0:0.00:0.75,expand=320:240,rotate=1 \
-ovc xvid \
-xvidencopts profile=dxnhtntsc:cartoon:zones=0,w,1.0:bitrate=384:pass=1 \
-oac mp3lame \
-lameopts cbr:br=128 \
-info name=$TGT \
-o $TGT.avi"