Thanks to my friend, Sean, I noticed yesterday that my last blog entry was posted exactly one year ago today. Well, I'm nothing if not punctual, and I'm committed to blogging at least once a year, whether I have anything to say or not.
So on the off-chance you're wondering what I've been doing this past year...
Well... I learned to juggle... a little... a few tricks, only 3 balls.
It's a little depressing when I think about it.
Thanks again, Sean. ;-)
Sunday, February 21, 2010
Saturday, February 21, 2009
A little lithp
After almost 20 years of fairly hardcore Emacs love, I finally wrote my first Lisp macro. And now I want to smear my good fortune all over you, dear reader, by recounting my experience in more depth.
How lucky for you!
About Lisp...
Created in 1958, Lisp is the 2nd oldest (not to mention sexiest) programming language in widespread use today. Its syntax is simple: lists of tokens contained within parentheses. In Lisp, code and data are represented the same way: as lists of tokens, duh. This deceptively powerful quality is what allows "Lispers" to create new syntax, i.e. Domain Specific Languages, within Lisp itself.
Lisp is basically what the "executable XML geniuses" -- yes, I'm looking at you Ant, Spring, J2EE, VoiceXML, and countless others -- should have used! XML is what you use when your chosen language for manipulating data can't adequately represent it.
My Macro
Ok, enough with the history. Here's the macro:
The trick to macros is the backquote character: `. Combined with the comma character, they allow you to selectively evaluate elements of a quoted list. In my macro, the list whose first token is defun is quoted (by a backquote) and my "selectively evaluated elements" are fun-name, connect-string, and buf-name.
Here's how I invoke my macro:
Um, hello?
Your stolid reaction belies the awesomeness of my macro!
Maybe I should've started with my reasons for creating the macro in the first place.
But perhaps it would be enlightening to see what I would've had to code to achieve the same result without using a macro:
See how verbose and redundant that is? It's positively VERBOSIDUNDANT!!! And my naming/calling conventions are sprinkled all over the place. To change any one would require multiple redundant changes. And that's not DRY, and that's not good.
How lucky for you!
About Lisp...
Created in 1958, Lisp is the 2nd oldest (not to mention sexiest) programming language in widespread use today. Its syntax is simple: lists of tokens contained within parentheses. In Lisp, code and data are represented the same way: as lists of tokens, duh. This deceptively powerful quality is what allows "Lispers" to create new syntax, i.e. Domain Specific Languages, within Lisp itself.
Lisp is basically what the "executable XML geniuses" -- yes, I'm looking at you Ant, Spring, J2EE, VoiceXML, and countless others -- should have used! XML is what you use when your chosen language for manipulating data can't adequately represent it.
My Macro
Ok, enough with the history. Here's the macro:
Lisp macros are like functions, except that instead of computing a value, they compute another Lisp expression that will compute the value. In this case, my macro creates a defun expression, which defines a function. Simple, huh?
(require 'sqlplus)
(defmacro def-oracle (name connect-string)
(let* ((buf-name (concat "oracle-" (symbol-name name)))
(fun-name (intern buf-name)))
`(defun ,fun-name ()
(interactive)
(sqlplus ,connect-string ,buf-name))))
The trick to macros is the backquote character: `. Combined with the comma character, they allow you to selectively evaluate elements of a quoted list. In my macro, the list whose first token is defun is quoted (by a backquote) and my "selectively evaluated elements" are fun-name, connect-string, and buf-name.
Here's how I invoke my macro:
(def-oracle dev "user/password@dev.host:port/sid")After those lines execute, three new functions are created:
(def-oracle qa "user/password@qa.host:port/sid")
(def-oracle prod "user/password@prod.host:port/sid")
- oracle-dev
- oracle-qa
- oracle-prod
Um, hello?
Your stolid reaction belies the awesomeness of my macro!
Maybe I should've started with my reasons for creating the macro in the first place.
- Create a handy function that "wraps" the Emacs sqlplus function for each of the databases to which I must often connect
- Don't prompt me for the dang connection credentials every time I invoke the handy wrapper function
- Enforce a consistent naming convention for the various sqlplus buffers I have open so that I may, for example, easily switch between qa and dev
- Make it easy to add more of these handy wrapper functions
But perhaps it would be enlightening to see what I would've had to code to achieve the same result without using a macro:
(defun oracle-dev ()
(interactive)
(sqlplus "user/password@dev.host:port/sid" "oracle-dev"))
(defun oracle-qa ()
(interactive)
(sqlplus "user/password@qa.host:port/sid" "oracle-qa"))
(defun oracle-prod ()
(interactive)
(sqlplus "user/password@prod.host:port/sid" "oracle-prod"))
See how verbose and redundant that is? It's positively VERBOSIDUNDANT!!! And my naming/calling conventions are sprinkled all over the place. To change any one would require multiple redundant changes. And that's not DRY, and that's not good.
Friday, August 15, 2008
PoopTag = Social Defecation
(It's not as gross as it sounds!)
Everybody poops. Every U.S. president. Every Jonas Brother. Every Olympic athlete. Even Michael Phelps. Actually, at 12,000 calories a day, he poops a LOT!
And so while you're pooping, have you ever wondered who else is pooping at exactly the same time? Really? Me, too!
So with the help of some friends, I created PoopTag, a sort of game that strives to be both encouraging and enlightening. It's easy to play: you simply notify PoopTag whenever you poop. You do this by sending a direct message to the pooptag account on Twitter using your mobile device. PoopTag will then determine whether anyone else is pooping at that time. If they are, you will have successfully "tagged" them, and PoopTag will notify its followers of this happy event.
And in response to your direct message, whether you successfully tag anyone or not, you will receive a random pithy reply. That's the "encouraging and enlightening" part.
There are two modes of play: public and covert. You decide which way to play each time you poop.
Obviously, the public notification makes it easy for someone to tag you, though they must be pooping in order to do so.
Eventually, we'll publish statistics at pooptag.com, enabling services like Poop Cycle Compatibility Matching, prospective home-buyer bathroom sizing tools, and life-saving medical history to present to your doctor, among other things.
Come join the fun!
UPDATE 9/20/08: Changed covert pooping method from an empty message to one containing only punctuation due to Twitter no longer delivering blank direct messages.
UPDATE 10/13/08: Covert pooping now triggered by a message containing only a single character (letter, number, punctuation, whatever)
Everybody poops. Every U.S. president. Every Jonas Brother. Every Olympic athlete. Even Michael Phelps. Actually, at 12,000 calories a day, he poops a LOT!
And so while you're pooping, have you ever wondered who else is pooping at exactly the same time? Really? Me, too!
So with the help of some friends, I created PoopTag, a sort of game that strives to be both encouraging and enlightening. It's easy to play: you simply notify PoopTag whenever you poop. You do this by sending a direct message to the pooptag account on Twitter using your mobile device. PoopTag will then determine whether anyone else is pooping at that time. If they are, you will have successfully "tagged" them, and PoopTag will notify its followers of this happy event.
And in response to your direct message, whether you successfully tag anyone or not, you will receive a random pithy reply. That's the "encouraging and enlightening" part.
There are two modes of play: public and covert. You decide which way to play each time you poop.
- Public (send more than one character):
d pooptag This text will be broadcast to all PoopTaggers
- Covert (send any single character):
d pooptag !
Obviously, the public notification makes it easy for someone to tag you, though they must be pooping in order to do so.
Eventually, we'll publish statistics at pooptag.com, enabling services like Poop Cycle Compatibility Matching, prospective home-buyer bathroom sizing tools, and life-saving medical history to present to your doctor, among other things.
Come join the fun!
UPDATE 9/20/08: Changed covert pooping method from an empty message to one containing only punctuation due to Twitter no longer delivering blank direct messages.
UPDATE 10/13/08: Covert pooping now triggered by a message containing only a single character (letter, number, punctuation, whatever)
Saturday, August 09, 2008
Create
I'm big on creativity. You know, building stuff. Putting things where nothing was before. I think it's both the secret to happiness and the meaning of life. I think it's what humans are SUPPOSED TO DO.
Create. Re-create. Recreation. Play, especially at work.
I follow why the lucky stiff on Twitter. His tweets are usually funny and clever. And once in a while, he'll write something like this:
Very cool.
Create. Re-create. Recreation. Play, especially at work.
I follow why the lucky stiff on Twitter. His tweets are usually funny and clever. And once in a while, he'll write something like this:
when you don't create things, you become defined by your tastes rather than ability. your tastes only narrow & exclude people. so create.
Very cool.
Thursday, June 12, 2008
Sunday, May 18, 2008
Trying to Save the Crum & Forster
My grandfather, Lewis "Buck" Crook, was a prominent architect. He and his partner, Ernest "Ed" Ivey, comprised the firm of Ivey & Crook, responsible for almost 600 commissions over the course of 43 years, most of which were built in and around the city of Atlanta, Georgia.
One of them, the Crum & Forster building, a midtown landmark at 771 Spring Street since 1927, is in imminent danger of being torn down. Ironically, the demolition is being proposed by Georgia Tech, who purchased the property in December of 2007, apparently with the intent to demolish it and expand the development of their Technology Square.
Georgia Tech is the alma mater of both my grandfather and his partner, Ed Ivey, who helped found the school of architecture there one hundred years ago this year. Through their works, they made a substantial contribution to the esteemed academic reputation Georgia Tech enjoys today.
Rather than destroy it, Georgia Tech has an opportunity to celebrate the work of Ivey & Crook, just as world-renowned architect John Portman, another Tech graduate, did when he expanded Emory University's student center in the 1980's. Ivey & Crook designed and built the original dining hall at Emory. Portman could've torn it down, but instead he chose to incorporate it into his own vision for the Dobbs University Center.
Georgia Tech has every reason to do the same for Technology Square!
An online petition to save the Crum & Forster has been created. Please add your name.
The building has an official blog. More pictures and info can be found here, here, and here.
One of them, the Crum & Forster building, a midtown landmark at 771 Spring Street since 1927, is in imminent danger of being torn down. Ironically, the demolition is being proposed by Georgia Tech, who purchased the property in December of 2007, apparently with the intent to demolish it and expand the development of their Technology Square.
Georgia Tech is the alma mater of both my grandfather and his partner, Ed Ivey, who helped found the school of architecture there one hundred years ago this year. Through their works, they made a substantial contribution to the esteemed academic reputation Georgia Tech enjoys today.
Rather than destroy it, Georgia Tech has an opportunity to celebrate the work of Ivey & Crook, just as world-renowned architect John Portman, another Tech graduate, did when he expanded Emory University's student center in the 1980's. Ivey & Crook designed and built the original dining hall at Emory. Portman could've torn it down, but instead he chose to incorporate it into his own vision for the Dobbs University Center.
Georgia Tech has every reason to do the same for Technology Square!
An online petition to save the Crum & Forster has been created. Please add your name.
The building has an official blog. More pictures and info can be found here, here, and here.
Tuesday, May 06, 2008
Saturday, November 03, 2007
A Few Poems
While doing some digital housekeeping recently, I ran across some poems I wrote back in the mid 90's when I worked with a very creative group of people at a now-defunct company. We were programmers, each responsible for one or more inter-related components of a large distributed system. When someone released a new module, the others would have to recompile to incorporate the changes. Often, a new-release email would be accompanied by a poem -- most often original -- or maybe song lyrics or even particularly elegant snippets of code.
My component was called the Data Access Daemon or DAD, for short. My first release email included this haiku:
Many of the poems I wrote were related to some feature of the DAD, i.e. stupid when taken out of context in a blog post, but a few of them I think stand on their own, or are at least "somewhat less stupid":
My component was called the Data Access Daemon or DAD, for short. My first release email included this haiku:
Restless, like a child,
searching for its potential,
DAD, ever-changing.
Many of the poems I wrote were related to some feature of the DAD, i.e. stupid when taken out of context in a blog post, but a few of them I think stand on their own, or are at least "somewhat less stupid":
Quicksort
=========
I happened across a big pile of stuff
And decided that it should be sorted.
The time I could spend was hardly enough,
Any more time could not be afforded.
I picked up a thing, at random, I'm sure,
And to each of the rest I compared it.
I made two more piles of stuff, as it were:
The things found above and below it.
To each of those piles, the same thing I done,
And to those thereafter created.
And not 'til the size of each pile was one --
Were my efforts quite quickly abated.
Hardly a minute of time had passed by,
When I realized something important.
The stuff that had been on that very first pile,
After just a short while, was now sorted!
A Little Number I Call "9"
==========================
Begin with twelve and four,
then take away seven.
If you add two more,
You're left with eleven.
Eleven, you could say,
Is two one's combined.
If you take those away,
You end up with nine.
Stressed (Chill) Out
====================
Never settle (Be still) for less.
Always strive (until) to be best.
Never stop (your will) to rest.
Everything (is nil.) is a test.
Work It
=======
"Work it, baby, work it!"
I heard some dancers say.
"Toil it, baby, toil it!"
I joined in their refrain.
At this, they hesitated,
And turned, and frowned at me.
"What did you say?!" they shouted.
For dancers, they seemed mean.
A misinterpretation
Was all that had occurred.
Instead of inspiration,
It was "toilet" that they heard.
Men and Women
=============
"Men are from Mars," of course,
"And women are from Venus."
As if those stupid metaphors,
Some wisdom they could gain us!
Men and women are different,
But not because of Venus.
Only women can get pregnant,
And only men can have a... prostate!
Kicking Squirrels
=================
A squirrel came hopping through my yard,
And got a little careless.
The distance wasn't all that far --
From him to my Adidas.
Ducking behind the nearest tree,
I lied in vengeful wait.
And by the time he noticed me,
It was much too late.
In mid-jump I caught him square --
Just like that fella, Pele.
Soaring higher into the air,
I watched him sail away.
He landed atop my neighbor's house
A little shaken, but still alive.
About that time I spied a mouse,
But my mom called me back inside.
Cubicles
========
The chains descended from the ceiling,
And fastened to the tops --
Of the cubicles.
The motors began slowly turning,
Then rising were the walls --
Like theatre curtains.
The workers stopped what they were doing,
And looked up from their desks --
To peer at themselves.
Then came some men who owned the building,
Too late to warn the folks --
Of their new plans.
Nashville and Seattle
=====================
What if Nashville was Seattle
And therefore, vice versa?
In the place of Mother Maybelle
Could be Ann and Nancy Wilson.
Then Little Jimmy Dickens
And Minnie Pearl, yes ma'am,
Would be Little Jimi Hendrix
And Minnie Pearl Jam.
We could hear Kurt Cobain
Or even Courtney Love
Sing "Blue Eyes Cryin' in the Rain"
Or "Wings of a Dove"
Wouldn't it make you feel alive
To see Shania Twain
Hockin' loogies between stage-dives
Apparently in pain?
Well, maybe not "alive" per se,
But it would be pretty rad,
Seeing Alice and Reba in chains
Switching cities, switching fads.
Fonts
=====
Arial is fairly boring,
And Courier's way too plain.
Modern is the same ol' story,
And frankly, Script is a pain.
Garamond comes off as haughty.
Sans Serif a tad too smooth.
Lucida strikes me as naughty,
And Roman a bit long in the tooth.
Symbol is symbolic of nothing.
Terminal should be put to rest.
Hardly better is Times New Roman,
And Wingdings is simply a mess!
In the end, the font doesn't matter.
You may choose whichever you like.
Whether one or another is better --
Counts far less than how well you write.
Long-Eared, Pickle-Toed Pirate
==============================
Wouldn't it be funny --
If my ears were long, like a bunny?
I'd tuck 'em up under my arm pits.
That wouldn't be gross, would it?
Wouldn't you be tickled --
If instead of toes I had pickles?
I'd clip off the nails and make relish.
Everyone would be so jealous.
Wouldn't it be a riot --
If my leg was a peg, like a pirate?
I'd buckle swashes, say "Aye" and grin,
And eat relish, and... um... listen?
Tony Gwynn
==========
Tony Gwynn's amazing,
But a little overweight.
It won't be getting past him
If it makes it 'cross his plate.
Rooms
=====
There are rooms to do your sleeping,
And rooms to take your bath,
Rooms to do your eating,
And rooms in which you laugh.
Most realtors show you common rooms
In the houses that they sell.
But an agent took me on a tour
I'll long remember well.
"Here's a room for your improvement,
And one for you to grow.
These rooms are not the kind you rent
They're rooms you come to know."
That's what the agent sang to me.
That's right, my realtor sings --
'Bout rooms that ain't so roomy,
But let you spread your wings!
Childbirth
==========
It won't be long for her.
It will all be over soon.
What now seems like November -
Will tomorrow seem like June.
What Is Is
==========
By the definition
In my deposition,
I have no recollection
Of a sexual relation.
I mean to be truthful
Without being helpful.
To admit that I'm a mouthful
Would be very, very harmful.
A Crappy Poem
=============
"Roses are blue,
Violets are red."
Er...no... that's not true.
Nevermind what I said.
"Stars in the morning,
Sunshine at night."
Now wait just a minute!
That can't be right.
"Ships on the prairie
Planes in the sea."
What am I saying?
What's happening to me?
"My eyes won't steady --
I feel like a nap."
Shut up, already.
Enough of this crap!
Seven Fools
===========
Seven separate fools considered their lot,
And in turn, each of them ruled,
"I know well enough to know I know not,"
"I'm content", as fools are, "to be fooled".
"Be wary of too-happy people", one claimed,
"And also avoid the too-sad",
"Try to eat well!" "Make beautiful things!"
"And be a good mother or dad."
Mr. Stroustrup
==============
An email, from Bjarne,
today I received.
"Thanks" was all that --
it said.
Ironic that he
should be thanking me.
I should thank him --
instead.
Saturday, October 13, 2007
New Job, New Spin on Frank
Well, it's been a while since I've blogged about anything. Since then, I've signed on with WebSingularity. We're attempting to build a social networking site that doesn't suck. More on that later, but for now, in honor of Frank Zappa, here's this:
Friday, July 06, 2007
Java shell tips
In which jar file does the class, CLASSNAME, reside?
To recursively search through all jars in the current directory...
And here's a handy little Python script that recursively "explodes" J2EE artifacts into their constituent components. If they're already exploded, it'll "implode" them. This is a great tool for quickly updating deployment descriptors at the command line.
for i in lib/*.jar; do jar tf $i | grep CLASSNAME >/dev/null && echo $i; done
To recursively search through all jars in the current directory...
find . -name "*.jar" | while read i; do jar tf $i | grep CLASSNAME >/dev/null && echo $i; done
And here's a handy little Python script that recursively "explodes" J2EE artifacts into their constituent components. If they're already exploded, it'll "implode" them. This is a great tool for quickly updating deployment descriptors at the command line.
#!/usr/bin/env python
#
# Each file passed on the command line that is a valid j2ee archive
# (war, jar, ear, sar) will be either exploded if archived or archived
# if exploded.
import sys,os
from zipfile import is_zipfile,ZipFile
from os.path import isdir,exists,realpath,dirname,basename,join
from shutil import rmtree
# The result of this try block is a function named 'mktempdir' that
# makes a temporary directory and returns its path. Python 2.3
# provides tempfile.mkdtemp, which does exactly what we need, but we
# must implement it ourself in older versions.
try:
from tempfile import mkdtemp as mktempdir
except ImportError:
from tempfile import mktemp
def mktempdir():
path = mktemp()
os.mkdir(path)
return path
# The result of this try block is a function called 'move', which
# moves a file or directory, possibly across filesystems. Python 2.3
# provides shutil.move, but older versions do not, so we revert to a
# system call. The os.rename function will not work across
# filesystems.
try:
from shutil import move
except ImportError:
def move(src,dest):
os.system("mv "+src+" "+dest)
# The presence of one of these indicates a J2EE artifact.
DEPLOYMENT_DESCRIPTORS = ("META-INF/application.xml", # ear
"WEB-INF/web.xml", # war
"META-INF/ejb-jar.xml", # jar
"META-INF/jboss-service.xml") # sar
def explode(file):
'''Recursively unpack the contents of a J2EE file into a directory of same name.'''
file = realpath(file)
tmp = mktempdir()
os.system("cd "+tmp+"; jar xMf "+file)
[explode(join(tmp,f)) for f in os.listdir(tmp) if is_j2ee_zip(join(tmp,f))]
os.remove(file)
move(tmp,file)
print "Exploded "+file
def implode(dir):
'''Recursively package the contents of a directory into a J2EE file with same name.'''
dir = realpath(dir)
[implode(join(dir,f)) for f in os.listdir(dir) if is_j2ee_dir(join(dir,f))]
tmp = mktempdir()+"/"+basename(dir)
os.system("cd "+dir+"; jar cMf "+tmp+" *")
rmtree(dir)
move(tmp,dir)
os.rmdir(dirname(tmp))
print "Imploded "+dir
def is_j2ee_dir(path):
'''Return True if directory contents are structured as a valid J2EE artifact.'''
if isdir(path):
for dd in DEPLOYMENT_DESCRIPTORS:
if exists(join(path,dd)): return True
def is_j2ee_zip(path):
'''Return True if path refers to a valid J2EE artifact.'''
if is_zipfile(path):
files = ZipFile(path).namelist()
for dd in DEPLOYMENT_DESCRIPTORS:
if dd in files: return True
for path in sys.argv[1:]:
if is_j2ee_zip(path):
explode(path)
elif is_j2ee_dir(path):
implode(path)
Monday, June 18, 2007
Stupidest Guitar Chord: D# minor
I love the Beatles' song, "I'm Only Sleeping", especially the very first chord, played with a beautiful, exaggerated "up-stroke" for emphasis. Strictly speaking, listening to the recording, the chord is a D# minor, but a guitar just ain't made to play that! For a guitarist, a D#m is a pretty stupid chord.
Why? Because E minor sounds so much better! The open strings allow it to ring long and full, using the lowest note possible in standard tuning. And it's easier to play! And it's only a half-step away! A mere semi-tone!
So either George Martin altered the speed of the recording, or maybe the guitars were tuned down a half step. But either way, nobody was playing a D#m on their guitar that day. The Beatles weren't stupid.
So you shouldn't be, either. But it's a pain to detune your guitar just to play along, right? Thankfully, there's an easier way: Audacity, a free audio editor for Linux, can do magic for the budding guitarist. It can change the pitch of a song without changing its tempo. Conversely, it can slow the tempo without changing the pitch, which is really handy for figuring out solos. What was impossible with analog recordings is now amazingly simple with digital audio.
Nobody should be forced to play D# minor.
Why? Because E minor sounds so much better! The open strings allow it to ring long and full, using the lowest note possible in standard tuning. And it's easier to play! And it's only a half-step away! A mere semi-tone!
So either George Martin altered the speed of the recording, or maybe the guitars were tuned down a half step. But either way, nobody was playing a D#m on their guitar that day. The Beatles weren't stupid.
So you shouldn't be, either. But it's a pain to detune your guitar just to play along, right? Thankfully, there's an easier way: Audacity, a free audio editor for Linux, can do magic for the budding guitarist. It can change the pitch of a song without changing its tempo. Conversely, it can slow the tempo without changing the pitch, which is really handy for figuring out solos. What was impossible with analog recordings is now amazingly simple with digital audio.
Nobody should be forced to play D# minor.
Tuesday, May 15, 2007
Tease
Tanya Thompson, the third tallest tenured teaching typist, told Timmy Timmington to try to take the touch-typing test tomorrow.
Ta-da!
Ta-da!
Friday, May 11, 2007
Remote Monitoring with Ruby and sshfs
At work I've been doing some performance analysis as of late. Specifically, I've been combing JBoss server.log files recording the time it takes various transactions to complete. Admittedly, I'm fighting a little with Werner Heisenberg because configuring the app to produce the log messages on which I depend will affect performance, but my interest is in relative rather than overall performance, i.e. I'm looking for potential bottlenecks.
Still, I don't want to take up any more resources from the app server's CPU than absolutely necessary. The Ruby script that processes the log files obviously makes much use of regular expressions. Here's an example of a Request object from my script.
Here's an example of a line from the server.log:
The Ruby script organizes the log messages by thread id. If it finds matches for each expected regular expression in the correct order without finding any it should reject, it records the total time for the Request. When finished, it reports various statistics (average, mean, stddev, etc) for each type of Request it found in the log.
Ideally, I would run that script on a log after a particular load test was performed, but hey that's no fun! I'd rather see how the server is doing in realtime, monitoring its performance while it's running. Here comes sshfs to the rescue!
With sshfs, I can mount the file system of the app server on a remote machine running my Ruby script. The app server doesn't need Samba or NFS installed, only ssh, and what respectable app server wouldn't have sshd running?
With the server.log mounted via sshfs, far fewer CPU resources are taken up by the analysis script, no more than if you were "tailing the log" yourself while the app server was running.
Installing sshfs is very simple on modern Debian-based distros:
Depending on the distro, you may need to add your user to the 'fuse' group or modprobe fuse, or chmod /dev/fuse, or some other such minor detail, but once installed, remote, realtime log file analysis is a breeze:
Still, I don't want to take up any more resources from the app server's CPU than absolutely necessary. The Ruby script that processes the log files obviously makes much use of regular expressions. Here's an example of a Request object from my script.
Request.new("Typical request") { |request|
request.expect /Processing Request/
request.expect /No existing Session found/
request.reject /ERROR/
request.reject /reaping required/
request.expect /Processing message took \d+ millis/
}
Here's an example of a line from the server.log:
2007-05-09 16:28:51,511 INFO [UDPListener] (Thread-16) UDPListener thread running
The Ruby script organizes the log messages by thread id. If it finds matches for each expected regular expression in the correct order without finding any it should reject, it records the total time for the Request. When finished, it reports various statistics (average, mean, stddev, etc) for each type of Request it found in the log.
Ideally, I would run that script on a log after a particular load test was performed, but hey that's no fun! I'd rather see how the server is doing in realtime, monitoring its performance while it's running. Here comes sshfs to the rescue!
With sshfs, I can mount the file system of the app server on a remote machine running my Ruby script. The app server doesn't need Samba or NFS installed, only ssh, and what respectable app server wouldn't have sshd running?
With the server.log mounted via sshfs, far fewer CPU resources are taken up by the analysis script, no more than if you were "tailing the log" yourself while the app server was running.
Installing sshfs is very simple on modern Debian-based distros:
$ sudo apt-get install sshfs
Depending on the distro, you may need to add your user to the 'fuse' group or modprobe fuse, or chmod /dev/fuse, or some other such minor detail, but once installed, remote, realtime log file analysis is a breeze:
So you use local CPU resources to parse a remote log file. Cool, huh?
$ sshfs appsrvr:/ /mnt/appsrvr
$ cd /mnt/appsrvr/usr/local/jboss/server/default/log
$ tail -f server.log | statistics.rb
Sunday, April 08, 2007
My Mother's Blog
I've been meaning to create a blog for my mother for some time now. I think she's a pretty good writer. I have to. She's my mother.
So here is My Mother's Blog.
So here is My Mother's Blog.
Sunday, March 11, 2007
1952 Vincent Black Lightning
The theme for this weekend: 1952 Vincent Black Lightning. That's because I heard Richard Thompson's song for the first time. It immediately appealed to me, what with its protagonist sharing my name and his girl being red-headed, not to mention some pretty tasty finger picking throughout... but who's this cat, Vincent?
And so began my journey of not only learning what the heck a Vincent is -- it's a famous motorcycle -- but also about Richard Thompson, his technique, and a new open tuning I'd never tried before.
I also learned about Rollie Free and decided that being famous for this photo is darn cool.
And even though I've previously cautioned against it, here is why YouTube is so freaking great:
So I ignored all the stuff I had wanted to get done this weekend and attempted to learn to play the tune. My youngest daughter thinks I'm a bit obsessed with it. Maybe so, but it's fun to sing, and especially funny to hear her sing it.
I ended up with a decent approximation... and a newfound respect for both Richard Thompson and Phil Vincent.
And so began my journey of not only learning what the heck a Vincent is -- it's a famous motorcycle -- but also about Richard Thompson, his technique, and a new open tuning I'd never tried before.
I also learned about Rollie Free and decided that being famous for this photo is darn cool.
And even though I've previously cautioned against it, here is why YouTube is so freaking great:
So I ignored all the stuff I had wanted to get done this weekend and attempted to learn to play the tune. My youngest daughter thinks I'm a bit obsessed with it. Maybe so, but it's fun to sing, and especially funny to hear her sing it.
I ended up with a decent approximation... and a newfound respect for both Richard Thompson and Phil Vincent.
Sunday, March 04, 2007
Testing EJB3 (JPA) with Maven2
This tutorial describes how to test EJB3 beans as a part of a Maven2 build, i.e. without an EJB container but with an in-memory, transactional database. I make the following assumptions:
As with many enterprise applications, much of our business logic is embedded within database queries, most in JPQL, and some can be quite complex. It's vital that we're able to test them outside of a container as a part of our normal, continuously-integrated build. For that, we use the Hypersonic (hsqldb) in-memory database. We'll need a JPA implementation, of course, and because we deploy to JBoss 4.0 in production, which uses Hibernate's JPA, we choose the same. Of course, any certified JPA implementation should work just as well.
pom.xml
So our first step is to declare our testing dependencies in our POM:
Aside: I toyed with using the new H2 database instead of Hypersonic, but the version I tested, 1.0.20061217, included only table locks, causing my app to deadlock. Hypersonic supports row locking.
Although not strictly necessary, I recommend adding this to your POM, too:
You'll see why in the next section.
Note that we didn't need to change the POM much. In particular, no special configuration of the Surefire plugin (the testcase invoker) is required.
src/test/resources/META-INF/persistence.xml
All JPA apps need a persistence.xml file. Your app will likely already have one beneath src/main/resources. Both will be in your CLASSPATH when the tests are run. The JPA will aggregate them, so you need to make sure all your persistence-units, both real and test, are uniquely named. How you organize your persistence-units is up to you. It's probably easiest to have just one for testing, or you might, for example, prefer to create one per test class.
Here's what I recommend:
Note the jar-file element. The ${...} expansion only works when filtering is turned on in the testResources element of the POM (see above). By setting the jar-file this way, we're telling the persistence provider to find our entities by searching for persistence annotations in our classes. From these, the required DDL is generated and the schema is created. Alternatively, you could use one or more class elements instead. This would allow you finer-grained control over which classes and/or packages are involved in a test. This strategy would probably also lead to multiple persistence-unit elements defined.
One thing to keep in mind about Hibernate. There's a known bug in which it won't create a schema before creating tables for your entities, so you're in trouble if you've set the schema attribute in your @Table annotations. I recommend removing this attribute from your entities anyway: it's a violation of the DRY principle. A better solution is to set the hibernate.default_schema property in your persistence-unit.
One debugging tip: you may want to include the following property in your persistence-unit:
That way, you could add something along these lines in your POM:
So when you want to see the SQL produced by your unit tests...
src/test/resources/import.sql
Some apps make use of Hibernate's ability to run some SQL after a schema is created. If it finds a file named import.sql on the classpath, it'll run the SQL within it. This can be a problem if, for example, your import.sql is exploiting Oracle PL/SQL commands that wouldn't make any sense to Hypersonic. Fortunately, you can easily solve the problem by simply putting a different -- possibly empty -- import.sql file beneath src/test/resources, because Hibernate will find that one first in the CLASSPATH.
Aside: I experienced some problems with a test-specific import.sql file, strange stack traces generated by Hibernate's SchemaExport class dumped to stdout without actually failing the tests. I ended up truncating src/test/resources/import.sql completely.
src/test/java/org/yours/SomeTest.java
Speaking of unit tests, here's an obviously-contrived example:
There is a lot of room for artistic freedom within this structure, but the essential point is the create/close of the EntityManagerFactory in the setUp/tearDown methods. This provides a clean database to each testXXX method.
If your transactional requirements are minimal, you may also want to create/close an EntityManager member variable in setUp/tearDown, too.
If your seed data requirements are complex, you may want to look into something like DBUnit, but in my experience, it's often easier to construct "builder" objects that can model various situations for your tests.
Transactions
For a lot of test cases, you can probably safely ignore transactions. Within one test, letting each of your beans use the same EntityManager without ever beginning or committing its transaction may go a long way toward testing your app sufficiently, especially if all of your beans are simply propagating the transaction anyway.
But transactions can get icky quickly, especially when you have multiple cooperating session beans with their transaction attribute set to REQUIRES_NEW. I'm going to show you one way of solving the problem using Java's dynamic proxies, but there's probably a more elegant solution.
Here's something I call a TransactionProxy. Others might call it "a good argument for AOP". It assumes your target bean class has an EntityManager member named em.
The idea is that you create it with an instance of say, a stateless session bean (SLSB) and an EntityManagerFactory, and subsequent invocations on the proxy will wrap calls to the real bean inside a transaction obtained from the EntityManagerFactory. For example:
This allows you to combine your beans in any number of transactional contexts. But it's not optimal since it's not actually using the @TransactionAttribute annotation of the classes under test. Hopefully, that's what the more elegant solution mentioned above is doing.
I'll update this tutorial after I've confirmed that. Until then, happy hacking!
- JDK 1.5
- EJB 3.0, i.e. Java Persistence API (JPA) 1.0
- Maven 2 (2.0.5)
As with many enterprise applications, much of our business logic is embedded within database queries, most in JPQL, and some can be quite complex. It's vital that we're able to test them outside of a container as a part of our normal, continuously-integrated build. For that, we use the Hypersonic (hsqldb) in-memory database. We'll need a JPA implementation, of course, and because we deploy to JBoss 4.0 in production, which uses Hibernate's JPA, we choose the same. Of course, any certified JPA implementation should work just as well.
pom.xml
So our first step is to declare our testing dependencies in our POM:
<!-- For in-memory sql testing -->
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>3.2.0.ga</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>hsqldb</groupId>
<artifactId>hsqldb</artifactId>
<version>1.8.0.7</version>
<scope>test</scope>
</dependency>
Aside: I toyed with using the new H2 database instead of Hypersonic, but the version I tested, 1.0.20061217, included only table locks, causing my app to deadlock. Hypersonic supports row locking.
Although not strictly necessary, I recommend adding this to your POM, too:
<testResources>
<testResource>
<directory>src/test/resources</directory>
<filtering>true</filtering>
</testResource>
</testResources>
You'll see why in the next section.
Note that we didn't need to change the POM much. In particular, no special configuration of the Surefire plugin (the testcase invoker) is required.
src/test/resources/META-INF/persistence.xml
All JPA apps need a persistence.xml file. Your app will likely already have one beneath src/main/resources. Both will be in your CLASSPATH when the tests are run. The JPA will aggregate them, so you need to make sure all your persistence-units, both real and test, are uniquely named. How you organize your persistence-units is up to you. It's probably easiest to have just one for testing, or you might, for example, prefer to create one per test class.
Here's what I recommend:
<persistence>
<persistence-unit name="hibernate-hsqldb">
<provider>org.hibernate.ejb.HibernatePersistence</provider>
<jar-file>${project.build.directory}/classes</jar-file>
<properties>
<property name="hibernate.dialect" value="org.hibernate.dialect.HSQLDialect"/>
<property name="hibernate.connection.driver_class" value="org.hsqldb.jdbcDriver"/>
<property name="hibernate.connection.url" value="jdbc:hsqldb:."/>
<property name="hibernate.connection.username" value="sa"/>
<property name="hibernate.connection.password" value=""/>
<property name="hibernate.hbm2ddl.auto" value="create-drop"/>
</properties>
</persistence-unit>
</persistence>
Note the jar-file element. The ${...} expansion only works when filtering is turned on in the testResources element of the POM (see above). By setting the jar-file this way, we're telling the persistence provider to find our entities by searching for persistence annotations in our classes. From these, the required DDL is generated and the schema is created. Alternatively, you could use one or more class elements instead. This would allow you finer-grained control over which classes and/or packages are involved in a test. This strategy would probably also lead to multiple persistence-unit elements defined.
One thing to keep in mind about Hibernate. There's a known bug in which it won't create a schema before creating tables for your entities, so you're in trouble if you've set the schema attribute in your @Table annotations. I recommend removing this attribute from your entities anyway: it's a violation of the DRY principle. A better solution is to set the hibernate.default_schema property in your persistence-unit.
One debugging tip: you may want to include the following property in your persistence-unit:
<property name="hibernate.show_sql" value="${hibernate.show_sql}"/>
That way, you could add something along these lines in your POM:
<properties>
<hibernate.show_sql>false</hibernate.show_sql>
</properties>
<profiles>
<profile>
<id>debug</id>
<properties>
<hibernate.show_sql>true</hibernate.show_sql>
</properties>
</profile>
</profiles>
So when you want to see the SQL produced by your unit tests...
$ mvn -Pdebug test
src/test/resources/import.sql
Some apps make use of Hibernate's ability to run some SQL after a schema is created. If it finds a file named import.sql on the classpath, it'll run the SQL within it. This can be a problem if, for example, your import.sql is exploiting Oracle PL/SQL commands that wouldn't make any sense to Hypersonic. Fortunately, you can easily solve the problem by simply putting a different -- possibly empty -- import.sql file beneath src/test/resources, because Hibernate will find that one first in the CLASSPATH.
Aside: I experienced some problems with a test-specific import.sql file, strange stack traces generated by Hibernate's SchemaExport class dumped to stdout without actually failing the tests. I ended up truncating src/test/resources/import.sql completely.
src/test/java/org/yours/SomeTest.java
Speaking of unit tests, here's an obviously-contrived example:
package org.yours;
import javax.persistence.*;
import junit.framework.*;
import org.apache.log4j.Logger;
public class SomeTest extends TestCase
{
public void testService() throws Exception
{
log.info ("testService");
EntityManager em = emf.createEntityManager();
TestData.build (em);
ServiceBean slsb = new ServiceBean();
slsb.em = em;
assertTrue (slsb.something());
}
protected void setUp() throws Exception
{
log.debug ("setUp");
emf = Persistence.createEntityManagerFactory ("hibernate-hsqldb");
}
protected void tearDown() throws Exception
{
log.debug ("tearDown");
emf.close();
}
private Logger log = Logger.getLogger(getClass());
private EntityManagerFactory emf;
}
There is a lot of room for artistic freedom within this structure, but the essential point is the create/close of the EntityManagerFactory in the setUp/tearDown methods. This provides a clean database to each testXXX method.
If your transactional requirements are minimal, you may also want to create/close an EntityManager member variable in setUp/tearDown, too.
If your seed data requirements are complex, you may want to look into something like DBUnit, but in my experience, it's often easier to construct "builder" objects that can model various situations for your tests.
Transactions
For a lot of test cases, you can probably safely ignore transactions. Within one test, letting each of your beans use the same EntityManager without ever beginning or committing its transaction may go a long way toward testing your app sufficiently, especially if all of your beans are simply propagating the transaction anyway.
But transactions can get icky quickly, especially when you have multiple cooperating session beans with their transaction attribute set to REQUIRES_NEW. I'm going to show you one way of solving the problem using Java's dynamic proxies, but there's probably a more elegant solution.
Here's something I call a TransactionProxy. Others might call it "a good argument for AOP". It assumes your target bean class has an EntityManager member named em.
public class TransactionProxy implements InvocationHandler
{
private EntityManagerFactory emf;
private Object target;
private Field field;
private TransactionProxy (Object target, EntityManagerFactory emf)
{
this.emf = emf;
this.target = target;
try {
this.field = target.getClass().getDeclaredField ("em");
this.field.setAccessible (true);
} catch (Exception e) {
throw new RuntimeException (e);
}
}
public static Object newInstance (Object target, EntityManagerFactory emf)
{
return Proxy.newProxyInstance (target.getClass().getClassLoader(),
target.getClass().getInterfaces(),
new TransactionProxy (target, emf));
}
public Object invoke (Object proxy, Method m, Object[] args)
throws Throwable
{
EntityManager em = emf.createEntityManager();
try {
field.set (target, em);
em.getTransaction().begin();
Object result = m.invoke (target, args);
em.getTransaction().commit();
return result;
} catch (InvocationTargetException e) {
em.getTransaction().rollback();
throw e.getTargetException();
} catch (Exception e) {
throw new RuntimeException (e);
} finally {
em.close();
}
}
}
The idea is that you create it with an instance of say, a stateless session bean (SLSB) and an EntityManagerFactory, and subsequent invocations on the proxy will wrap calls to the real bean inside a transaction obtained from the EntityManagerFactory. For example:
Service service = (Service) TransactionProxy.newInstance (new ServiceBean(), emf);
This allows you to combine your beans in any number of transactional contexts. But it's not optimal since it's not actually using the @TransactionAttribute annotation of the classes under test. Hopefully, that's what the more elegant solution mentioned above is doing.
I'll update this tutorial after I've confirmed that. Until then, happy hacking!
From Gentoo+Postfix+Courier to Debian+Exim+Dovecot
Ok, let's say -- hypothetically, of course -- you chose Gentoo for your home email server a few years ago. You found one of Gentoo's great docs on setting up Postfix, Courier and SpamAssassin. You accepted all the reasonable default options during the install, and things seemed to work great.
Over time, you began to take the little email server for granted. Oh sure, you'd occasionally login and "emerge" some security updates, but the dang thing just worked. Why bother it?
Of course, entropy is a cold bitch, and "for granted taking" turned to neglect. After a while, you began to forget all the arcane portage commands, having to use rpms at work and happily experimenting with Ubuntu's adopted dpkg on your desktop. In the beginning, spam was under control, but it's a constant battle, and forgetting your distro's packaging commands won't exactly help you in that fight.
And then terror struck: the Gentoo maintainers decided to release a portage update that was incompatible with the version running on your little email server. And at that precise version did your server remain for the rest of its life. The prospect of recompiling your entire system was just too much to bear. You decided it was a good time to look for another distro, maybe in a couple weeks or so, when you're not so busy...
Well, a couple of weeks turned into a couple of years. And over that time, you came to realize the benefits of the maildir storage format over the traditional mbox one, the latter being one of those reasonable defaults mentioned above. More importantly, you began to appreciate apt/dpkg as "portage without all the compiling", and you became aware of Debian's reputation for server stability.
One day when the spam/ham ratio was particularly high, you decided the time had finally come to switch the ol' home server from Gentoo to Debian. You decided you'd build a new box, transfer the family email over and convert it to the Maildir format. Just because you're a total dumbass-glutton-for-punishment, you also decided to switch MTA's (Postfix to Exim) and IMAP daemons (Courier to Dovecot). "Why the hell not?" you wondered.
Um... I'm not sure I can keep up the hypothetical perspective any longer. I'm thinking you were seeing right through it, anyway. Sorry.
So you -- I mean, I -- found a cheap Dell P3 on Ebay for $33+s/h, installed Debian Etch, and then fired up aptitude...
Exim
By default, Debian installs exim4-daemon-light, which is fine for most purposes, but exim4-daemon-heavy is required for integration with most spam/virus detection packages and, despite its name, it's really not that heavy.
Configuring Exim is easy enough, though it took me a while to grok the split vs non-split formats. I wasted a lot of time googling for docs/howtos: I wish I had just started with /usr/share/doc/exim4/README.Debian.gz like the maintainers suggest. I don't know why Google results seem so much more glamorous than the packaged docs.
I went through the normal routine:
I chose non-split, mail sent by smarthost, received via SMTP, and accepted mail for *.internal.domain;internal.domain;external.domain. Your -- my? -- domain names will differ, of course. I chose to relay mail for my local network, set the name of the outgoing smarthost, hid the local mail names making only my external domain visible, and made sure to set Maildir as my delivery format.
I then created user accounts for all my family members, and sent them some test mail from a box configured to relay outgoing mail to my new server. In addition to testing the above config, it also caused the Maildir structures to be created in the users' home directories. Easy-peazy-lemon-squeazy.
Something else cool: I can add entries to /etc/aliases and exim will pick them up immediately without restarting or rerunning newaliases or some such. I do that for extended relatives so I don't have to remember their real email addresses, only their alias, e.g. unclebob@crossleys.org. Let me know if you want one.
Dovecot
So delivery is working. Now I need an IMAP server for retrieval.
Configuring Dovecot was simple: I just needed to enable plaintext logins (I'm not exposed to the bad ol' Internet) and the imap protocol. Here's a diff of my changes to /etc/dovecot/dovecot.conf:
After saving those changes, a restart is required:
imapsync
Now comes the hard part: how to move my email from the old server to the new one? After googling a bit, I came across a utility called mb2md, but its interface is a little weird, seemed to expect files to be in certain places, and seemed more geared toward a single user rather than a group. Somewhat discouraged, I restarted my search, and came across imapsync. This discovery simplified my approach: instead of copying mbox files (both spooled and saved) from the old server to the new and *then* converting them, I could simply encapsulate the format differences behind the IMAP interface, i.e. imapsync allows me to copy email from one IMAP server to another, irrespective of the underlying formats of each! Neat-Oh!
Installing imapsync is easy:
Using it takes some practice. Fortunately it provides a --dry option for just that purpose. Here's an example of my Courier-to-Dovecot imapsync incantations for a particular user. I needed to invoke it twice: once for the main inbox, and again for the categories of saved mail I kept beneath the ~/mail directory on the old server:
The nice thing about imapsync is that it's a true sync. You can run it multiple times and it only copies over what's changed. This allowed me to be pretty sloppy about when I actually updated my firewall to send SMTP requests to the new server.
ClamAV
Ok, so we've (we, as in me, not you) accomplished a lot. The only thing remaining is to reject viruses and spam. ClamAV will handle the viruses:
To integrate with Exim, two changes to /etc/exim4/exim4.conf.template were required. I set av_scanner to this:
And uncommented these lines (presumably put there by exim4-daemon-heavy):
After running update-exim4.conf, restarting exim and sending a fake virus (as described here), I noticed some "lstat() failed" errors in the mainlog. Smells like a perms problem. To fix that, I did this:
SpamAssassin, er... greylistd
At this point in our story, I expected to tell you how easy it was to integrate SpamAssassin with our (my) new setup, but frankly I haven't done it yet.
Instead, I discovered greylisting. This has eliminated 95% of my spam. For this reason, I'm holding off on SpamAssassin integration for the time being.
The greylistd package provides a convenient script that completely idiot-proofs exim4 integration:
Simple, huh?
Conclusion
That's it. We're done. Whew!
Now I have a stable home email server that I can actually update and try to secure. Sweet.
Over time, you began to take the little email server for granted. Oh sure, you'd occasionally login and "emerge" some security updates, but the dang thing just worked. Why bother it?
Of course, entropy is a cold bitch, and "for granted taking" turned to neglect. After a while, you began to forget all the arcane portage commands, having to use rpms at work and happily experimenting with Ubuntu's adopted dpkg on your desktop. In the beginning, spam was under control, but it's a constant battle, and forgetting your distro's packaging commands won't exactly help you in that fight.
And then terror struck: the Gentoo maintainers decided to release a portage update that was incompatible with the version running on your little email server. And at that precise version did your server remain for the rest of its life. The prospect of recompiling your entire system was just too much to bear. You decided it was a good time to look for another distro, maybe in a couple weeks or so, when you're not so busy...
Well, a couple of weeks turned into a couple of years. And over that time, you came to realize the benefits of the maildir storage format over the traditional mbox one, the latter being one of those reasonable defaults mentioned above. More importantly, you began to appreciate apt/dpkg as "portage without all the compiling", and you became aware of Debian's reputation for server stability.
One day when the spam/ham ratio was particularly high, you decided the time had finally come to switch the ol' home server from Gentoo to Debian. You decided you'd build a new box, transfer the family email over and convert it to the Maildir format. Just because you're a total dumbass-glutton-for-punishment, you also decided to switch MTA's (Postfix to Exim) and IMAP daemons (Courier to Dovecot). "Why the hell not?" you wondered.
Um... I'm not sure I can keep up the hypothetical perspective any longer. I'm thinking you were seeing right through it, anyway. Sorry.
So you -- I mean, I -- found a cheap Dell P3 on Ebay for $33+s/h, installed Debian Etch, and then fired up aptitude...
Exim
By default, Debian installs exim4-daemon-light, which is fine for most purposes, but exim4-daemon-heavy is required for integration with most spam/virus detection packages and, despite its name, it's really not that heavy.
# aptitude install exim4-daemon-heavy
Configuring Exim is easy enough, though it took me a while to grok the split vs non-split formats. I wasted a lot of time googling for docs/howtos: I wish I had just started with /usr/share/doc/exim4/README.Debian.gz like the maintainers suggest. I don't know why Google results seem so much more glamorous than the packaged docs.
I went through the normal routine:
# dpkg-reconfigure exim4-config
I chose non-split, mail sent by smarthost, received via SMTP, and accepted mail for *.internal.domain;internal.domain;external.domain. Your -- my? -- domain names will differ, of course. I chose to relay mail for my local network, set the name of the outgoing smarthost, hid the local mail names making only my external domain visible, and made sure to set Maildir as my delivery format.
I then created user accounts for all my family members, and sent them some test mail from a box configured to relay outgoing mail to my new server. In addition to testing the above config, it also caused the Maildir structures to be created in the users' home directories. Easy-peazy-lemon-squeazy.
Something else cool: I can add entries to /etc/aliases and exim will pick them up immediately without restarting or rerunning newaliases or some such. I do that for extended relatives so I don't have to remember their real email addresses, only their alias, e.g. unclebob@crossleys.org. Let me know if you want one.
Dovecot
So delivery is working. Now I need an IMAP server for retrieval.
# aptitude install dovecot-imapd
Configuring Dovecot was simple: I just needed to enable plaintext logins (I'm not exposed to the bad ol' Internet) and the imap protocol. Here's a diff of my changes to /etc/dovecot/dovecot.conf:
21c21
< protocols =
---
> protocols = imap
46c46
< #disable_plaintext_auth = yes
---
> disable_plaintext_auth = no
After saving those changes, a restart is required:
# /etc/init.d/dovecot restart
imapsync
Now comes the hard part: how to move my email from the old server to the new one? After googling a bit, I came across a utility called mb2md, but its interface is a little weird, seemed to expect files to be in certain places, and seemed more geared toward a single user rather than a group. Somewhat discouraged, I restarted my search, and came across imapsync. This discovery simplified my approach: instead of copying mbox files (both spooled and saved) from the old server to the new and *then* converting them, I could simply encapsulate the format differences behind the IMAP interface, i.e. imapsync allows me to copy email from one IMAP server to another, irrespective of the underlying formats of each! Neat-Oh!
Installing imapsync is easy:
# aptitude install imapsync
Using it takes some practice. Fortunately it provides a --dry option for just that purpose. Here's an example of my Courier-to-Dovecot imapsync incantations for a particular user. I needed to invoke it twice: once for the main inbox, and again for the categories of saved mail I kept beneath the ~/mail directory on the old server:
imapsync --host1 oldbox --user1 jim --password1 oldpass \
--host2 newbox --user2 jim --password2 newpass \
--folder INBOX
imapsync --host1 oldbox --user1 jim --password1 oldpass \
--host2 newbox --user2 jim --password2 newpass \
--include "^mail/.+"
The nice thing about imapsync is that it's a true sync. You can run it multiple times and it only copies over what's changed. This allowed me to be pretty sloppy about when I actually updated my firewall to send SMTP requests to the new server.
ClamAV
Ok, so we've (we, as in me, not you) accomplished a lot. The only thing remaining is to reject viruses and spam. ClamAV will handle the viruses:
# aptitude install clamav-daemon
To integrate with Exim, two changes to /etc/exim4/exim4.conf.template were required. I set av_scanner to this:
av_scanner = clamd:/var/run/clamav/clamd.ctl
And uncommented these lines (presumably put there by exim4-daemon-heavy):
deny
malware = *
message = This message was detected as possible malware ($malware_name).
After running update-exim4.conf, restarting exim and sending a fake virus (as described here), I noticed some "lstat() failed" errors in the mainlog. Smells like a perms problem. To fix that, I did this:
# adduser clamav Debian-exim
# /etc/init.d/clamav-daemon restart
SpamAssassin, er... greylistd
At this point in our story, I expected to tell you how easy it was to integrate SpamAssassin with our (my) new setup, but frankly I haven't done it yet.
Instead, I discovered greylisting. This has eliminated 95% of my spam. For this reason, I'm holding off on SpamAssassin integration for the time being.
The greylistd package provides a convenient script that completely idiot-proofs exim4 integration:
# aptitude install greylistd
# greylistd-setup-exim4 add
Simple, huh?
Conclusion
That's it. We're done. Whew!
Now I have a stable home email server that I can actually update and try to secure. Sweet.
Sunday, February 18, 2007
Hating Maven less than Ant
I've been writing Java code since 1996 and struggling with build tools at least as long. Ant is the most comprehensive and Maven is the most ambitious. I hate both of them, mostly because they each require me to describe my project's build requirements using XML, which should never be used as a programming language, though often it's used as a configuration file format, but because all configuration files ultimately will evolve into programming languages... I digress.
For different reasons, Ant and Maven are both terrific and terrible at once. Ant allows you to describe your build however you please, but you must effectively do it from scratch for each project. Maven encourages many best practices by providing a lot of build features out-of-the-box, adopting a sort of "convention over configuration" philosophy popularized by the Ruby on Rails project, among others.
So for my money, Maven is the logical choice for a typical Java project. Unfortunately, few Java projects are typical, especially few enterprise Java projects. And for those projects, Maven can be frustrating because, like a child or a turtle, it's difficult to make it do something it doesn't want to do.
For example, let's say I'm debugging a build and I want to echo the value of a property to the console during a build. With Ant, I could add a target to build.xml:
With Maven 1, I can invoke Ant tasks within a goal defined in maven.xml:
Maven 1 provides easy access to Ant tasks, as you can see. Unfortunately, doing anything non-trivial with Maven 1 involves Jelly, which is pure "executable XML hell". Maven 2 did away with Jelly, but...
With Maven 2, it's bewilderingly complicated to echo a simple property to the console. You must include the "antrun" plugin in your pom.xml, configure an execution to run Ant's echo task, and bind that execution to a particular phase of your build (but see comments for other options). Ready? Here we go:
Note that our choice of the "validate" phase is completely arbitrary, but in order for our task to execute, it must be bound to some phase. Unlike our Ant and Maven 1 examples above, Maven 2 provides no way to invoke a single task without invoking all other tasks bound to the same phase along with all those before it. Maven 2 introduced the concept of profiles to address this perceived limitation, but I'll save that discussion for a future blog post.
Clearly, the Maven designers have violated a fundamental tenet: "the solution cannot be more abstract than the problem!" In trying to define and encourage best practices, Maven unapologetically sacrifices simplicity and flexibility, arguably two of the most important qualities of any software build tool, even for those projects exhibiting all accepted best practices.
Somewhat surprisingly, I generally still think it's worthwhile to convert your Ant projects to Maven, no matter how difficult. Maven's pain is concentrated, sort of like filling a cavity early in the project, resulting in a relatively pain-free existence once your project leaves the dentist's office.
Ant is more like a minor toothache annoying you for the entire duration of your project, and then spreading to the other projects in your mouth as each begins its life with a copy of some other's build.xml file. Yes, of course it's simpler to do simple things with Ant, but adding enough of those simple things to projects over time can (and will, in my experience) lead to a pretty darn complex, brittle, and incomprehensible build.
In the long run, I think you'll end up hating Maven less than Ant, too. But if you insist on using Ant, use Ivy to manage your dependencies.
UPDATED Maven 2 example, removing unnecessary <property> element to reflect Eric's comment.
For different reasons, Ant and Maven are both terrific and terrible at once. Ant allows you to describe your build however you please, but you must effectively do it from scratch for each project. Maven encourages many best practices by providing a lot of build features out-of-the-box, adopting a sort of "convention over configuration" philosophy popularized by the Ruby on Rails project, among others.
So for my money, Maven is the logical choice for a typical Java project. Unfortunately, few Java projects are typical, especially few enterprise Java projects. And for those projects, Maven can be frustrating because, like a child or a turtle, it's difficult to make it do something it doesn't want to do.
For example, let's say I'm debugging a build and I want to echo the value of a property to the console during a build. With Ant, I could add a target to build.xml:
<project>
<target name="debug">
<echo>The value is ${some.property}</echo>
</target>
</project>
With Maven 1, I can invoke Ant tasks within a goal defined in maven.xml:
<project>
<goal name="debug">
<echo>The value is ${some.property}</echo>
</goal>
</project>
Maven 1 provides easy access to Ant tasks, as you can see. Unfortunately, doing anything non-trivial with Maven 1 involves Jelly, which is pure "executable XML hell". Maven 2 did away with Jelly, but...
With Maven 2, it's bewilderingly complicated to echo a simple property to the console. You must include the "antrun" plugin in your pom.xml, configure an execution to run Ant's echo task, and bind that execution to a particular phase of your build (but see comments for other options). Ready? Here we go:
<project>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>debug</id>
<phase>validate</phase>
<configuration>
<tasks>
<echo>The value is ${some.property}</echo>
</tasks>
</configuration>
<goals>
<goal>run</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
Note that our choice of the "validate" phase is completely arbitrary, but in order for our task to execute, it must be bound to some phase. Unlike our Ant and Maven 1 examples above, Maven 2 provides no way to invoke a single task without invoking all other tasks bound to the same phase along with all those before it. Maven 2 introduced the concept of profiles to address this perceived limitation, but I'll save that discussion for a future blog post.
Clearly, the Maven designers have violated a fundamental tenet: "the solution cannot be more abstract than the problem!" In trying to define and encourage best practices, Maven unapologetically sacrifices simplicity and flexibility, arguably two of the most important qualities of any software build tool, even for those projects exhibiting all accepted best practices.
Somewhat surprisingly, I generally still think it's worthwhile to convert your Ant projects to Maven, no matter how difficult. Maven's pain is concentrated, sort of like filling a cavity early in the project, resulting in a relatively pain-free existence once your project leaves the dentist's office.
Ant is more like a minor toothache annoying you for the entire duration of your project, and then spreading to the other projects in your mouth as each begins its life with a copy of some other's build.xml file. Yes, of course it's simpler to do simple things with Ant, but adding enough of those simple things to projects over time can (and will, in my experience) lead to a pretty darn complex, brittle, and incomprehensible build.In the long run, I think you'll end up hating Maven less than Ant, too. But if you insist on using Ant, use Ivy to manage your dependencies.
UPDATED Maven 2 example, removing unnecessary <property> element to reflect Eric's comment.
Saturday, February 17, 2007
Linux Podcasts
Since I got my new Meizu, I've been listening to a lot of podcasts, especially those dedicated to Linux. There are a surprising number of them out there, and I've listened to quite a few, but I wanted to share my favorites.
- Linux Reality - Chess Griffin's generous, informative and practical podcast, presumably for newbies but really for anyone interested in Linux
- LugRadio - can't understand them half the time, but usually very funny
- JaK Attack - kind of hit and miss, but nice to hear a female voice
- ITConversations Programming series - not active any longer, and focused on software development rather than linux, but great interviews with industry heavyweights like Paul Graham, Kent Beck, etc
- Linux Action Show - often silly and self-conscious, but not boring and usually informative
- Run Your Own Server - smart cats, more geared toward BSD than Linux, though
Saturday, February 03, 2007
Charter DNS still sucks
I set up dnsmasq on the Debian box I had been running bind9 on. It was infinitely easier to configure than a "real" DNS server -- just edit /etc/hosts -- and it perfectly suited the host-naming requirements for my admittedly tiny home network.
As with all things computer, of course, I still had hurdles to overcome, and they all involved difficulties with the commercial operating systems in my network, specifically Win XP and Mac OS X. All the Linux boxes were blissfully easy to setup and predictably functional and even adaptable to the weirdness required by the commercial ones.
As far as the Mac goes, it wouldn't resolve my local domain name, apparently because it ended in .local. I used that suffix because I read somewhere that it was an illegal TLD (top level domain, e.g. .com, .org, .edu, etc). Apparently, Mac's lookupd process eats those names because of the file /etc/resolver/local. I could fix the problem by adding a file for my domain to /etc/resolver, but I decided it was easier to use the suffix .home instead. I'm pretty sure it's an illegal TLD, too.
The WinXP box was sad for another reason. I had initially tried configuring dnsmasq as a DHCP daemon, which was cool, because then I could control all my static IP's in /etc/dnsmasq.conf instead of having to configure each host on the network. But alas, that broke my work VPN client on the WinXP box. I didn't spend any time trying to figure out why, though.
So now my router still acts as the DHCP server, even though the work laptop is the only thing that uses it. The other hosts are configured with static IP's that are replicated in the /etc/hosts file on the host running dnsmasq. Suboptimal, but functional.
As with all things computer, of course, I still had hurdles to overcome, and they all involved difficulties with the commercial operating systems in my network, specifically Win XP and Mac OS X. All the Linux boxes were blissfully easy to setup and predictably functional and even adaptable to the weirdness required by the commercial ones.
As far as the Mac goes, it wouldn't resolve my local domain name, apparently because it ended in .local. I used that suffix because I read somewhere that it was an illegal TLD (top level domain, e.g. .com, .org, .edu, etc). Apparently, Mac's lookupd process eats those names because of the file /etc/resolver/local. I could fix the problem by adding a file for my domain to /etc/resolver, but I decided it was easier to use the suffix .home instead. I'm pretty sure it's an illegal TLD, too.
The WinXP box was sad for another reason. I had initially tried configuring dnsmasq as a DHCP daemon, which was cool, because then I could control all my static IP's in /etc/dnsmasq.conf instead of having to configure each host on the network. But alas, that broke my work VPN client on the WinXP box. I didn't spend any time trying to figure out why, though.
So now my router still acts as the DHCP server, even though the work laptop is the only thing that uses it. The other hosts are configured with static IP's that are replicated in the /etc/hosts file on the host running dnsmasq. Suboptimal, but functional.
Subscribe to:
Posts (Atom)