Remote Monitoring with Ruby and sshfs

At work I've been doing some performance analysis as of late. Specifically, I've been combing JBoss server.log files recording the time it takes various transactions to complete. Admittedly, I'm fighting a little with Werner Heisenberg because configuring the app to produce the log messages on which I depend will affect performance, but my interest is in relative rather than overall performance, i.e. I'm looking for potential bottlenecks.

Still, I don't want to take up any more resources from the app server's CPU than absolutely necessary. The Ruby script that processes the log files obviously makes much use of regular expressions. Here's an example of a Request object from my script.

  Request.new("Typical request") { |request|
    request.expect /Processing Request/
    request.expect /No existing Session found/
    request.reject /ERROR/
    request.reject /reaping required/
    request.expect /Processing message took \d+ millis/
  } 

Here's an example of a line from the server.log:

2007-05-09 16:28:51,511 INFO  [UDPListener] (Thread-16) UDPListener thread running

The Ruby script organizes the log messages by thread id. If it finds matches for each expected regular expression in the correct order without finding any it should reject, it records the total time for the Request. When finished, it reports various statistics (average, mean, stddev, etc) for each type of Request it found in the log.

Ideally, I would run that script on a log after a particular load test was performed, but hey that's no fun! I'd rather see how the server is doing in realtime, monitoring its performance while it's running. Here comes sshfs to the rescue!

With sshfs, I can mount the file system of the app server on a remote machine running my Ruby script. The app server doesn't need Samba or NFS installed, only ssh, and what respectable app server wouldn't have sshd running?

With the server.log mounted via sshfs, far fewer CPU resources are taken up by the analysis script, no more than if you were "tailing the log" yourself while the app server was running.

Installing sshfs is very simple on modern Debian-based distros:

  $ sudo apt-get install sshfs

Depending on the distro, you may need to add your user to the 'fuse' group or modprobe fuse, or chmod /dev/fuse, or some other such minor detail, but once installed, remote, realtime log file analysis is a breeze:

  $ sshfs appsrvr:/ /mnt/appsrvr
  $ cd /mnt/appsrvr/usr/local/jboss/server/default/log
  $ tail -f server.log | statistics.rb

So you use local CPU resources to parse a remote log file. Cool, huh?