Mark S. Kolich

Integrating Ant and Google's Closure Compiler

2010-08-21T18:45:00Z

With a little spare time on my hands, I dug into Google's Closure Compiler, an open-source tool that minifies and optimizes JavaScript for better performance. Unlike other JavaScript compression tools like JSMin, Packer or the YUI compressor, Google's Closure Compiler converts your decent JavaScript into "better" JavaScript. It does so by searching for and pointing out obvious JavaScript pitfalls, deleting dead code, and optimizing what's left to make your web-applications more efficient. This all sounds great, so I gave it a whirl and eventually integrated it into my Ant build for a new project. Heck, I even found a bug and suggested an enhancement.

1 - $jQuery Users be Warned

Before you run off and have a field day with the Closure Compiler, you should know that optimizing jQuery with the compiler's ADVANCED_OPTIMIZATIONS mode failed miserably. Even with a proper JS externs file, I could not get jQuery, or my JavaScript that uses jQuery, to compile under ADVANCED_OPTIMIZATIONS mode. Ok, actually it compiled fine, but when I tried to run this JavaScript in a browser: welcome to JavaScript Error City, population one.

Eventually, I gave up and reverted back to SIMPLE_OPTIMIZATIONS mode instead. This works much better. If you know something I don't about compiling jQuery in advanced mode, please let me know.

2 - Get 'er done

First things first, you'll need to download the Closure Compiler JAR and integrate it into your Ant build file by defining a new Ant task:

  location="${lib.dir}/google-closure-compiler.jar" />

  classname="com.google.javascript.jscomp.ant.CompileTask"
  classpath="${jscompjar}"/>

Yay, now you have a Ant task.

3 - Concatenate

Next, you'll need Ant to concatenate each of your JavaScript files into a single file. Technically, you don't have to do this, but I found it much easier and I like having all of my JavaScript compressed into a single resource:

Note that I'm using a FileList instead of a FileSet. This is because the order in which the resources are concatenated is important; they should be concatenated in the order in which they are required, and using a FileList ensures proper order. More on that here.

Also, note that if you eventually plan on compiling your JavaScript with ADVANCED_OPTIMIZATIONS enabled, you'll need to concatenate your JavaScript files together anyways. Further, using a single JavaScript resource across your entire site has obvious performance advantages. HTTP operations are usually expensive (especially on mobile devices), so having a web-browser download one large JavaScript file with everything in it is usually better than having it download multiple smaller files. Of course, you could even take it a step further and use a tool like LABjs to load your JavaScript dynamically, as needed.

4 - Compile

Ok, now that your JavaScript files have been concatenated into a single resource, let's use the Ant task we defined earlier to compile it:

  debug="false" output="${release.js.dir}/project.min.js">

Assuming your code compiled cleanly, you'll have a Closure Compiler optimized JavaScript file sitting around ready for deployment!

Enjoy.

Spring 3 and Spring Security: Setting your Own Custom /j_spring_security_check Filter Processes URL

2010-07-25T02:20:00Z

While working on a new personal project, I decided to pick up and dig into Spring 3 MVC and Spring Security. I've touched both of these technologies here and there in a number of other projects, but this new opportunity has really opened the door for a deep dive into Spring. I know the Ruby fanatics out there would say I'm crazy, but once I got all of the awful XML configuration mess out of the way, I'm really enjoying Spring's stability and reliability. I know Ruby and Rails is hip and what not, but Spring 3 has really made an impression on me. It's hard to beat such a mature enterprise framework.

I setup a few Spring 3 controllers, and integrated Spring Security into my web-app. All went great and so I added a simple form-based login to my Spring Security XML configuration.

Problem: Overriding UsernamePasswordAuthenticationFilter

When setting up a form-based login via a default Spring Security configuration, Spring auto generates and configures a UsernamePasswordAuthenticationFilter bean. This filter, by default, responds to the URL /j_spring_security_check when processing a login POST from your web-form. First, I want to override Spring Security's default login process URL to /login instead of /j_spring_security_check. Second, I've configured a Spring 3 controller to display my login web-form when a user visits /login.

That said, here's the underlying problem with Spring Security's default UsernamePasswordAuthenticationFilter: I want it to accept and process POST's to /login, but a GET or any HTTP method to /login should be forwarded to the next filter in the chain. Not surprisingly, you cannot do this with Spring Security's default UsernamePasswordAuthenticationFilter because it does not @Override the doFilter() method of AbstractAuthenticationProcessingFilter. In short, there's no way to get and check the incoming HTTP request method and re-route it using the default UsernamePasswordAuthenticationFilter.

Solution: Write your own Spring Security Authentication Filter

If you want a Spring controller to process GET requests to /login, but Spring Security to intercept and process a POST to /login, then you'll need to write your own Spring Security authentication filter. Here's the idea:

public class MyFilter extends AbstractAuthenticationProcessingFilter {

  private static final String DEFAULT_FILTER_PROCESSES_URL = "/login";
  private static final String POST = "POST";

  public MyFilter () {
    super(DEFAULT_FILTER_PROCESSES_URL);
  }

  @Override
  public Authentication attemptAuthentication(HttpServletRequest request,
    HttpServletResponse response) throws AuthenticationException,
    IOException, ServletException {
    // You'll need to fill in the gaps here.  See the source of
    // UsernamePasswordAuthenticationFilter for a working implementation
    // you can leverage.
  }

  @Override
  public void doFilter(ServletRequest req, ServletResponse res,
    FilterChain chain) throws IOException, ServletException {
    final HttpServletRequest request = (HttpServletRequest) req;
    final HttpServletResponse response = (HttpServletResponse) res;
    if(request.getMethod().equals(POST)) {
      // If the incoming request is a POST, then we send it up
      // to the AbstractAuthenticationProcessingFilter.
      super.doFilter(request, response, chain);
    } else {
      // If it's a GET, we ignore this request and send it
      // to the next filter in the chain.  In this case, that
      // pretty much means the request will hit the /login
      // controller which will process the request to show the
      // login page.
      chain.doFilter(request, response);
    }
  }

}

Note the good stuff inside of doFilter(). If the incoming request method is a POST, then we send it up to our AbstractAuthenticationProcessingFilter to actually process the login. If it's a GET, or any other HTTP request method for that matter, we simply send it to the next filter in the chain.

Finally, remember that you'll need to define your own FORM_LOGIN_FILTER inside of your Spring Security XML configuration to override the default /j_spring_security_check URL:

  entry-point-ref="LoginUrlAuthenticationEntryPoint">
  


  class="org.springframework.security.web.authentication.LoginUrlAuthenticationEntryPoint">

Enjoy!

Extended Thoughts on Custom Tiny URL Engines

2010-05-31T23:35:00Z

A blog reader recently contacted me via email and asked, "hey, how does your koli.ch tiny URL thing work?" Well, I would be happy to explain. As previously discussed here, and here, I'm not using Apache's mod_rewrite RewriteMap engine. No, koli.ch tiny URL's are using an interesting combination of Apache's mod_rewrite RewriteRule directive and Onyx, which is basically PHP talking to a MySQL database.

So, let's follow http://koli.ch/x3dh through the redirection process. The "3dh" you see on the end of each URL is a base-36 encoded number, that maps to a unique bookmark resource stored inside of Onyx.

Step 1 - koli.ch to mark.koli.ch

First, you may have noticed that http://koli.ch always redirects to http://mark.koli.ch by default. This is accomplished using Apache's mod_rewrite in my virtual host configuration ...

RewriteCond %{HTTP_HOST} !^mark\.koli\.ch [NC]
RewriteRule ^/(.*)$ http://mark.koli.ch/$1 [R=301,L]

You'll notice that the Perl compatible regular expression in the RewriteRule directive captures the URL-path content after the first slash, allowing me to gracefully forward URL's like http://koli.ch/about-mark.html to http://mark.koli.ch/about-mark.html. Consequently, tiny URL's like http://koli.ch/x3dh are forwarded to http://mark.koli.ch/x3dh. And, this redirect is a 301 Moved Permanently instead of a default 302 Found redirect.

Step 2 - mark.koli.ch to onyx.koli.ch

Next, in the same virtual host configuration, I use mod_rewrite's RewriteRule directive to match URL-paths that represent a tiny URL. As far as Onyx is concerned, all of my tiny URL's all start with "x" and contain at least one, but no more than ten, word characters following the "x" ...

RewriteCond %{REQUEST_URI} !\.(htm?l)$ [NC]
RewriteRule ^/x(\w{1,10})$ https://onyx.koli.ch/x$1 [L]

Notice the regular expression group, I'm only capturing the word-characters following the "x" in the request URI, then appending them onto https://onyx.koli.ch/x before handing it off to Onyx. Note that I have not specified the redirect response code in the RewriteRule options. In this case, Apache will 302 Found redirect to Onyx, which tells search engines and other bots that this redirect is much less permanent than a 301 Moved Permanently.

Step 3 - onyx.koli.ch to ?

Once the browser hits https://onyx.koli.ch/x, I use RewriteRule one last time to call a redirection controller I wrote in PHP which actually takes the tiny URL, sanitizes the request URI then looks it up in a MySQL database to retrieve the final URL to redirect to. This is part of Onyx. Here's the pseudo code ...


$uri = (isset($_SERVER['REQUEST_URI'])) ? $_SERVER['REQUEST_URI'] : null;
if(empty($uri)) {
  throw new InternalErrorException();
}

// Strip the leading / from the front of the URI
$uri = preg_replace("/^\//","",$uri);
@list(, $resource) = preg_split("/[\/\?]/", $uri, -1, PREG_SPLIT_NO_EMPTY);
if(!empty($resource)) {
  $resource = preg_replace("/\D/","",$resource);
} else {
  throw new BadRequestException();
}

// ... do some MySQL things here, look up the $resource in the DB.
$url = ... some URL from the database

// Redirect the user to their destination, via an HTTP 302 Found
header(HTTP_302_FOUND);

// Unset the Pragma HTTP response header; to avoid
// redirection issues on IE.
header(PRAGMA.": ");
header(CACHE_CONTROL.": no-store, no-cache");

header(LOCATION_HEADER.': '.$url);

?>

That's it. Cheers.

MySQL Triggers and SUPER Privileges: "Access denied; you need the SUPER privilege for this operation."

2010-05-27T19:20:00Z

I just discovered that dealing with MySQL triggers, in many instances, is quite painful. For example, here's a trigger that deletes a bunch of rows in a table on every INSERT:

delimiter |
CREATE TRIGGER delete_expired_tweets AFTER INSERT ON tweets
  FOR EACH ROW BEGIN
    DELETE FROM tweets WHERE DATEDIFF(NOW(), created_at) > 365;
  END;
|

delimiter ;

Ok, let's load this trigger into MySQL:

#/> mysql -h myhost -u normaluser -p mydatabase
Enter password: **********
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.0.41-community-nt MySQL Community Edition (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> source trigger.sql
ERROR 1227 (42000): Access denied; you need the SUPER privilege
    for this operation

Access denied? I dug into it, and confirmed that you can only add triggers if your user account has the SUPER privilege enabled. You're probably thinking, "No kidding Sherlock, that's what the error message says." Yes, I know that's what the error message says. But here's the problem. Normal database users created using "GRANT ALL PRIVILEGES ON database.* TO..." will not have the SUPER privilege assigned to them by default. As described here, the SUPER privilege in MySQL let's the account do some things that normal database users, in most environments, should not be able to do (like kill database threads, modify global system variables, etc.). As a result, it's a very bad idea to grant the SUPER privilege to normal database users, even if they just need the SUPER privilege to load a trigger. You know better than that!

Even worse, suppose you GRANT SUPER PRIVILEGES to a single user, on a single database. Well, that still won't be enough to load a trigger. Unfortunately, loading triggers requires SUPER PRIVILEGES at the global level (e.g., GRANT SUPER PRIVILEGES ON *.*). Again, it's a very bad ideal to grant normal database users the SUPER privilege.

So how exactly am I supposed to load this trigger? Well as far as I can tell, assuming I refuse to give myself SUPER PRIVILEGES for the reasons I just explained, I have two options:

Don't use triggers, and find another way to cleanup rows in my table.
Log into the database as root/admin and load the trigger on behalf of the normal user. If I wasn't the owner of this database server, this would probably involve asking my database administrator to load the trigger for me.

This is just one of many common annoyances with MySQL. Sucks, I know.

OAuth and the Twitter API: Generate a one-time access token and token secret

2010-05-23T19:05:00Z

You may have heard that Twitter plans to stop supporting HTTP Basic Authentication on June 30, 2010. This means that starting on June 30th, to use Twitter's API, your application must support OAuth. OAuth is a nice step up from basic authentication but it makes developing web or desktop applications that communicate with Twitter, slightly more painful. Well, painful isn't the right word, but you definitely have to jump through more hoops to get things to work. Gone are the days of simply sending a username and password to the API.

In response to this change, Twitter API proxy services like SuperTweet have popped up. Turns out, if you know what you're doing with OAuth, SuperTweet and other API proxy services are entirely unnecessary, not to mention unsafe. You're better off upgrading your applications to use OAuth the right way, instead of making them rely on potentially insecure third-party proxy services. And again, it's not difficult, just a bit annoying.

Scenario

You're a developer, and you need to write some code that pulls in Tweets from one or more users. Maybe you also need to pull down a list of followers for each of these users. Not surprisingly, it's entirely unreasonable to ask each of them to authenticate your application using OAuth. You just want to write code that pulls down their public timeline, followers, etc. avoiding the whole OAuth dance with each user, every time.

Solution

Register a new application on Twitter. Then, dig into your application control panel and find your new "single access token" and "single access token secret" for the application you just registered.

As described here, "Twitter offers the ability for you to retrieve a single access token (complete with oauth_token_secret) from application detail pages found in your application control panel. This is ideal for applications migrating to OAuth with single-user use cases ... By using a single access token, you don't need to implement the entire OAuth token acquisition dance. Instead, you can pick up from the point where you are working with an access token to make signed requests for Twitter resources."

This token and token secret is as close as you'll get to a username/password equivalent in OAuth. In other words, once you have this one-time token and token secret for your application, you can issue signed OAuth requests against the Twitter API just like you would with a basic username and password. If you want to think about it this way, the token is like your username and the token secret is like your password. Don't share them. Once you have these credentials, you can pull in Tweets for any public user, get their followers, read the public timeline, etc.

Here are several examples in a number of popular languages showing how you can use this one-time token and token secret in your project.

Yay for OAuth.

Formatting a Java Date into a Specific TimeZone and Conversion Between TimeZone's

2010-05-15T18:51:00Z

Date objects in Java, and probably most other robust languages, simply represent a snapshot of a point in time. In other words, java.util.Date knows nothing about the time zone you're referring to when instantiating or manipulating a Date object. Fact is, java.util.Date does not have to care about your time zone, because internally a Date is really nothing more than a count of the number of milliseconds since the standard base time known as "the epoch", namely January 1, 1970, 00:00:00 GMT.

If it helps, think about it this way: X milliseconds since the epoch is X milliseconds since the epoch in the US-Pacific time zone, X milliseconds since the epoch in GMT-0 (London), X milliseconds since the epoch in India, etc. In short, when it's 1273947282085 milliseconds since the epoch, it's 1273947282085 milliseconds since the epoch everywhere in the world at the same time regardless of what time zone you're sitting in. And since Java's util.Date is simply a snapshot of the number of milliseconds at a specific point in time, you can see why Date doesn't care about your time zone. It's irrelevant.

But Mark, how do I convert a java.util.Date into a different time zone? You can't, and that question makes no sense. That's like asking me to "take a picture of the sound." Here's some B.S. code that you should not use, but I've put it here for illustrative purposes:

// Do NOT use this, it does nothing and makes no sense.
public static final Date convertIntoTimeZone(Date date, TimeZone tz) {
  final Calendar cal = Calendar.getInstance();
  cal.setTime(date);
  cal.setTimeZone(tz);
  return cal.getTime();
}

You can't convert a Date into a different time zone, but you can use Java's handy DateFormat class to format a Date into the time zone of your choice. To put it differently, let Date do its thing. Then, when you're ready to display or print out a String representation of Date, that's when you tell DateFormat what time zone you want it in. So, here's some code that makes sense, and actually works:

final Date currentTime = new Date();

final SimpleDateFormat sdf =
        new SimpleDateFormat("EEE, MMM d, yyyy hh:mm:ss a z");

// Give it to me in US-Pacific time.
sdf.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
System.out.println("LA time: " + sdf.format(currentTime));

// Give it to me in GMT-0 time.
sdf.setTimeZone(TimeZone.getTimeZone("GMT"));
System.out.println("GMT time: " + sdf.format(currentTime));

// Or maybe Zagreb local time.
sdf.setTimeZone(TimeZone.getTimeZone("Europe/Zagreb"));
System.out.println("Zagreb time: " + sdf.format(currentTime));

// Even 10 hours and 10 minutes ahead of GMT
sdf.setTimeZone(TimeZone.getTimeZone("GMT+0010"));
System.out.println("10/10 ahead time: " + sdf.format(currentTime));

Yay for Dates and time zones! Enjoy.

HTTP Digest Access Authentication using MD5 and HttpClient 4

2010-05-04T20:30:00Z

Dealing with HTTP's Digest authentication mechanism isn't too bad once you have the basic building blocks in place. Luckily HttpClient 4 can automatically solve many types of authentication challenges for you, if used correctly. Using HttpClient 4, I built an app that authenticates against a SOAP based web-service requiring WWW-Authenticate Digest authentication. In a nutshell, the fundamental principal behind HTTP Digest authentication is simple:

The client asks for a page that requires authentication.
The server responds with an HTTP 401 response code, providing the authentication realm and a randomly-generated, single-use value called a "nonce". The authentication "challenge" itself is encapsulated inside of the WWW-Authenticate HTTP response header.
The client "solves" the authentication challenge and a solution is sent back to the web-server via the HTTP Authorization header on a subsequent request. The solution usually contains some type of MD5 hashed mess of your username, password, and "nonce".
Assuming the solution is acceptable the server responds with a successful type response, usually an HTTP 200 OK.

Here's a sample with a bit of pseudo code mixed in (so, you get the idea):

// A org.apache.http.impl.auth.DigestScheme instance is
// what will process the challenge from the web-server
final DigestScheme md5Auth = new DigestScheme();

// This should return an HTTP 401 Unauthorized with
// a challenge to solve.
final HttpResponse authResponse =
        doPost(url, postBody, contentType);

// Validate that we got an HTTP 401 back
if(authResponse.getStatusLine().getStatusCode() ==
      HttpStatus.SC_UNAUTHORIZED) {
  if(authResponse.containsHeader("WWW-Authenticate")) {
    // Get the challenge.
    final Header challenge =
            authResponse.getHeaders("WWW-Authenticate")[0];
    // Solve it.
    md5Auth.processChallenge(challenge);
    // Generate a solution Authentication header using your
    // username and password.
    final Header solution = md5Auth.authenticate(
            new UsernamePasswordCredentials(username, password),
            new BasicHttpRequest(HttpPost.METHOD_NAME,
                  new URL(url).getPath()));
    // Do another POST, but this time include the solution
    // Authentication header as generated by HttpClient.
    final HttpResponse goodResponse =
            doPost(url, postBody, contentType, solution);
    // ... do something useful with goodResponse, which assuming
    // your credentials were valid, should contain the data you
    // requested.
  } else {
    throw new Error("Web-service responded with Http 401, " +
      "but didn't send us a usable WWW-Authenticate header.");
  }
} else {
  throw new Error("Didn't get an Http 401 " +
      "like we were expecting.");
}

Enjoy.

Pingdom Incorrectly Reports Page Load Time

2010-04-24T20:00:00Z

I recently discovered that Pingdom, in many instances, does not correctly report page-load time. As it turns out, when you ask Pingdom to tell you how fast your site loads it will incorrectly download all images and other resources referenced in your CSS, regardless of whether they are actually used on your site or not. Of course this isn't ideal, because Pingdom will claim the page you're testing takes longer to load than it really does! By comparison, smart web-browsers do not blindly download every image referenced in the CSS, on every page. Instead, they download the resources as needed.

For folks who really care about site optimization and performance, this is a big deal. Especially if you're hired to help a client optimize the page-load time of their web-site, but services like Pingdom lie and report junk.

Here's an example; imagine this is your homepage:



 My Homepage
 

 
  header

  content

  footer

Ok, so you've got a header and a footer that will appear on every page. In addition, you've got a few other CSS selectors for the homepage, for your "about us" page, and for your "contact us" page. In a real web-browser, when you visit this page the browser will load the following images as referenced in your CSS:

header.png
homepage.png
footer.png

However, when you test this same page using Pingdom, Pingdom will claim your site is loading the following resources:

header.png
homepage.png
about-us.png
contact.png
footer.png

Wait a second, that's not accurate at all! On this hypothetical homepage, I'm definitely not using #aboutus or #contact anywhere, so why should the resources associated with these selectors be factored into my page load time? In reality, they shouldn't be because that's not how real web-browsers work.

To further validate my findings, I used the Rackspace Cloud and allocated a new machine with a running web-server. I loaded this hypothetical homepage HTML onto it, and asked Pingdom to tell me how fast it loads:

Wrong! I'm not using aboutus.png or contactus.png anywhere on the homepage, so why should Pingdom factor the load time of those images into my test results? In contrast, using Firefox and HttpFox, I verified that real web-browsers only load the resources they need to properly render the page:

Here you can see the correct resources are loaded in Firefox: header.png, homepage.png, and footer.png.

Bottom line, you really shouldn't use Pingdom and other external performance analysis tools as "bibles" for improving your page-load time. Clearly they're only tools to help you identify obvious problems, and are definitely not a complete solution. For more accurate results, I would suggest using tools like YSlow and Google's Page Speed.

Cheers.

Understanding Java's CountDownLatch and CyclicBarrier

2010-04-09T18:20:00Z

While working on some nifty multi-threaded Java recently, a colleague pointed me to a few really useful Java classes: CountDownLatch and CyclicBarrier. My code was quite typical, a parent worker thread spawns a bunch of children to do real work, and needs to wait for the children to finish before continuing. The catch though, is that the child worker threads may or may not finish successfully, and in all likelihood will finish at different times. Even so, the parent thread must wait until all of its children have finished because the parent can only make forward progress once the children are complete. I whipped up a little demo (screen shot left) that spawns five worker threads which update a JProgressBar at a random interval. The demo finishes once each progress bar hits 100%.]]> 1 - CountDownLatch

Meet CountDownLatch. As described in the Java 6 API docs, a CountDownLatch is "a synchronization aid that allows one or more threads to wait until a set of operations being performed in other threads completes." In other words, the developer says new CountDownLatch(N) which waits for N threads to finish before the latch is "released" allowing the calling thread to make forward progress. Couldn't be more perfect here. To make my life a little easier, I wrote a few wrapper classes that encapsulate a CountDownLatch which allow me to easily synchronize on a List, a list of worker threads:

ThreadRunner.java -- A class that accepts a List (a List of BaseWorker's), creates a new CountDownLatch(list.size()), starts each BasedWorker then allows the developer to await() on the runner for all BaseWorker's to finish.
BaseWorker.java -- An abstract class that represents each worker thread, and defines a set of methods each BaseWorker must implement to be used with a ThreadRunner.

So, using these wrappers, let's create a new worker:

public class MyWorker extends BaseWorker {

  private int someField_;

  public MyWorker(int worker) {
    super();
    someField_ = worker;
    // ...
  }

  @Override
  public void myRun() throws Exception {
    // ...
  }

  @Override
  public String getWorkerName() {
    return String.format("%s #%s",
        getClass().getSimpleName(), someField_);
  }

}

Now let's setup a new ThreadRunner that will take a bunch of BaseWorker's, start them, then wait for all to finish:

public class MyRunner {

  private static final List workers__;
  static {
    workers__ = new ArrayList();
    workers__.add(new MyWorker(1));
    workers__.add(new MyWorker(2));
    workers__.add(new MyWorker(3));
  }

  public static void main(String[] args) {
    final ThreadRunner runner = new ThreadRunner(workers__);
    // Start all of the threads in this runner.
    runner.start();
    // Wait for all of the threads to finish.
    runner.await();
    // Did all of our workers complete without error?
    if(runner.wasSuccessful()) {
      System.out.println("All workers finished cleanly.");
    } else {
      System.out.println("Not all workers finished cleanly.");
    }
  }

}

In this example, I built a List of BaseWorker's, gave the list to the ThreadRunner and asked the runner to start them. Upon calling runner.await(), the ThreadRunner blocks waiting for all of the workers to finish. Note that my concept of "finish" here means either successfully, or unsuccessfully (an Exception or Error case). Subsequently, I call runner.wasSuccessful() to check if all of the workers finished cleanly, basically asking the runner did all of your workers finish without throwing any Exception's or Error's?

If you're interested, you can download my complete ThreadRunner demo/example that further demonstrates the usage of these wrapper classes using Swing and several JProgressBar's.

2 - CyclicBarrier

A CyclicBarrier is similar to that of a CountDownLatch, except that a CyclicBarrier is "a synchronization aid that allows a set of threads to all wait for each other to reach a common barrier point." Like a CountDownLatch, a CyclicBarrier can be used to synchronize a number of threads. But instead of exiting upon completion, theads using a CyclicBarrier await() for all other threads in the pool to finish. Here's a usage example of a CyclicBarrier built around my BaseWorker class:

public class MyCyclicWorker extends BaseWorker {

  private CyclicBarrier barrier_;

  public MyWorker(CyclicBarrier barrier) {
    super();
    barrier_ = barrier;
    // ...
  }

  @Override
  public void myRun() throws Exception {
    // ...
    // Wait here for all other threads
    // in the CyclicBarrier to finish.
    barrier_.await();
  }

  @Override
  public String getWorkerName() {
    return getClass().getSimpleName();
  }

}

Here's the class that starts up a bunch of these MyCyclicWorkers, then runs a single "cleanup" thread once all of the workers are done:

public class CyclicExample {

  private static final int CYCLIC_THREADS = 5;

  public static void main(String[] args) {
    final CyclicBarrier barrier =
            new CyclicBarrier(CYCLIC_THREADS,
              new Runnable() {
                @Override
                public void run() {
                  // Cleanup thread, or completion thread.
                  // Called when all of the worker threads
                  // are finished.
                  // ...
                }
              });
    for(int i=0; i < CYCLIC_THREADS; ++i) {
      new MyCyclicWorker(barrier).start();
    }
    // ...
  }

}

Enjoy!

HOWTO: Ignore Files and Directories in Subversion (SVN)

2010-04-08T16:21:10Z

Subversion is fun. Especially when it tries to commit intermediate build files and other junk you want to keep out of your SVN repository. In my environment, Ant is configured to place intermediate build files into my build/ directory. The final product is then packaged up and placed into my dist/ directory. So, I need SVN to ignore everything inside of my build/ directory since the contents of this folder are going to change on every build.

Meet svn propedit svn:ignore ...

#/> svn propedit svn:ignore build
#/> svn -m "ignoring build" commit build

When you run svn propedit, your local text editor will open (usually vi) which will allow you to specify a series of file names to ignore under build/. If you want to ignore all files and folders, just enter a single * wildcard character, save, and commit:

Or, maybe you want to ignore just .class files:

*.class

Of course, if you want to use another editor like emacs to set your ignore properties, you can do so by exporting SVN_EDITOR before running svn propedit:

#/> export SVN_EDITOR=emacs

Enjoy!

HOWTO: Scheduling Cron Style Timers and Jobs with Spring and Quartz

2010-03-27T17:25:00Z

Spring Timer's are great if you need a job or task to run on a fixed interval. For example, if you need a bean to wake up and run itself every 30-minutes. But what if you need finer control over the scheduling of a timer? Say you need a job to run every morning at 3AM, regardless of when the JVM is started. Or, say you need a bean to run "every half hour between the hours of 8 AM and 10 AM on the 5th and 20th of every month." You won't get that type of granular control with a straight-up Spring Timer.

Meet Quartz, a full-featured, open source job scheduling service that can be integrated with, or used along side virtually any Java EE or Java SE application. Similar to a native UNIX Cron job, Quartz lets you define powerful CronExpression's that give you very precise control over when a job or task is started and on what interval.

Setting up Quartz with your Spring powered web-application is easy.]]> 1 - Download Quartz

The latest Quartz libraries are available here. Download them, and add to your build-path and build-files accordingly.

2 - Add Beans to Your applicationContext.xml

In your applicationContext.xml, add the necessary beans:


  class="com.kolich.app.beans.UpdaterBean"
  init-method="init"
  depends-on="AnotherBean">
  





  class="org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean">
  
  
  



  class="org.springframework.scheduling.quartz.CronTriggerBean">

Note that you can add as many jobs/triggers as desired to the Quartz SchedulerFactoryBean.

3 - Sample Cron Expressions

Here are some other sample cronExpression's you might enjoy (I borrowed these from the Quartz trigger tutorial) ...

Fires every five minutes:

"0 0/5 * * * ?"

Fires every 5 minutes, at 10 seconds after the minute:

"10 0/5 * * * ?"

Fires at 10:30, 11:30, 12:30, and 13:30, on every Wednesday and Friday:

"0 30 10-13 ? * WED,FRI"

Fires every half hour between the hours of 8 am and 10 am on the 5th and 20th of every month:

"0 0/30 8-9 5,20 * ?"

Enjoy!

Fixing Blown Thermal Fuse in Krups KM7000 Coffee Maker; Triangle Head/Bit Tamper Proof Screws Too!

2010-03-27T00:50:00Z

In 2008, my girlfriend bought me a Krups KM7000 Grind-and-Brew Coffee Maker for Christmas. I love it, it grinds and brews an incredible pot of high quality coffee for me each morning. This past weekend (about 15-months since I got it), my KM7000 unexpectedly stopped brewing mid-cycle. And to make matters worse, the one-year manufacturers warranty period already expired. Well, I wasn't about to let this one get away, so I decided to open it up and give it a look-see. After all, shouldn't a high-end coffee maker last a little longer than just over a year?

With a little effort and some engineering know-how, I found the troublemaker part and repaired my coffee maker. My KM7000 is as good as new. Here's how I did it ...
]]> 1 - Symptoms

The symptoms of my failing KM7000 were pretty straightforward. Basically everything on the machine worked great, except the hot plate would not get hot nor would the water pump from the reservoir into the brew-pot. The grinder worked perfectly, the LCD display was completely functional, but the machine just wouldn't get hot and wouldn't brew any coffee. Further, when I started a brew-cycle, I could hear a little "click" inside the machine (this is normal) but the coffee maker just sat there doing nothing.

After doing a little research online, I discovered that this is a very common problem with the KM7000. Apparently, this unit is well known for dying just past the one-year warranty period. Interesting. Time to open this puppy up for a little surgery.

NOTE: My KM7000 is out of warranty, and could not be replaced. If you have a KM7000 that's still under warranty, and it's exhibiting the symptoms I described above, you're better off calling Krups for a new replacement unit. If you open your KM7000, and it's still under warranty, I can say with much certainty that Krups won't give you a free replacement. Better not risk it.

DISCLAIMER: If you attempt to self-fix your KM7000 coffee maker, you should know what you're doing. If you follow this blog post and destroy your KM7000 or set your house on fire, you should know that I'm in no way responsible. If you don't feel comfortable doing this by yourself, find a friend or family member who is more mechanically or electrically inclined to help you.

2 - Triangle Bit Tamper Proof Screws

I took the coffee maker out to my workbench, and flipped it over (making sure that I poured out all of the water, and emptied the coffee bean bin). Using a standard Phillips head screwdriver, I removed all but one screw from the underside of the unit. As it turned out, I discovered that I could not remove one of the screws because it had a non-standard tamper-proof, triangle shaped inset. Note that the head was round, but the inset/bit was literally an equilateral triangle. In other words, this was no standard screw. Krups obviously put this screw in place to prevent people like me from trying to open the unit. Screw that! I tried to loosen this screw using a tiny flat-head screwdriver, being careful not to strip the head. Unfortunately, it was too tight and I gave up fearing that I might strip the head making it impossible for me to remove the screw altogether.

First shot, I called a few local hardware stores looking for a triangle shaped replaceable bit or screwdriver, but they never heard of this type of tamper-proof screw. I checked online, and found a few places that sell triangle shaped machine bits, but I wasn't about to pay a few bucks plus $10 USD shipping for a screwdriver bit I'll really only use once. Plus, the online stores I found had 4 sizes of triangular bits so I really had no idea what size I should order, and didn't want to risk wasting my money. For the record, I found these bits at McMaster-Carr online:

If you're curious, here's a close up shot of a standard Phillips head (left) and triangular bit tamper-proof screw (right):

In the end, my solution to this problem involved an old flat-head screwdriver, a vice and metal file. I snagged an old beat-up flat-head screwdriver, locked it into my vice, and filed the tip into a triangular point. About 10-minutes later, with a little trial and error, I had myself a pretty sweet triangular screwdriver bit that could easily handle these ridiculous tamper-proof screws!

3 - Inside the Machine

Once I removed all of the screws from the underside of the unit, the hard plastic bottom slid right out. Note that there are more "hidden" screws under the little rubber feet; this wasn't immediately obvious to me. You'll have to pop the rubber plugs out of their sockets to reveal about 5-6 more screws.

The internals of the KM7000 seem pretty basic. There's a length of flexible orange tubing that connects the water reservoir to the heating coil (and the hot plate!), then upwards to the brew-pot. A tiny circuit board connects the LCD display to the rest of the unit. So, as I see it, here's how the machine brews coffee. Water from the reservoir flows through the orange tubing into the heating ring, connected to the hot-plate with a little thermal adhesive. From there, the heating element heats the water which forces it upwards through the rest of the orange tubing and into the brew-pot. Remember, the symptoms of my problem were that the hot-plate would not get hot, nor would the machine brew any coffee. Well, the heating coil boils the water as it passes through the tube which propels it upwards into the brew-pot. And, because the heating coil is connected to the underside of the hot plate, the coil warms the hot-plate while it boils the water! In other words, if the machine isn't boiling any water then the hot plate ain't gonna be hot, and vice versa. So clearly, the problem was with the heating coil itself or the circuit that powers the heating coil.

4 - Problematic Thermal Fuses

With a little research, I found this page and this great blog post describing the same problem, and a do-it-yourself fix. Turns out there are dual inline one-shot thermal fuses on the circuit that connects the heating coil to the rest of the unit. Both are Therm-o-disc G4 series Microtemp axial leaded fuses; one is rated up to 216C (~420F), the other is rated to 240C (~464F). The specs and other details are clearly printed on the fuses themselves. Based on what others reported elsewhere, and the symptoms of my failing KM7000, these fuses were highly suspect.

At first, I had difficulty discovering the location of the thermal fuses. As it turns out, they are assembled together in series, hidden under a white plastic sheath on the circuit leading to/from the heating element. Gently side the white sheath away to expose the fuses.

Using a multimeter, I checked the resistance across both fuses to see which one had bit the dust. Theoretically speaking, the resistance across an open circuit should be infinite, so my multimeter helped me pinpoint the problematic fuse. Turns out, the 240C fuse had blown wide open; but the 216C fuse was perfectly in tact. One would expect that if the unit really was overheating because of an electrical malfunction, the 216C fuse would have blown first given that it's built to trigger at a lower temperature. In any event, I strongly agree with Jimmy Su, that these fuses probably aren't made all that well and eventually fail under normal operating conditions. One would think that a high-end appliance manufacturer wouldn't cheap-out on basic components. Especially on a one-shot fuse; if it blows, a $150 coffee maker is rendered useless by a $1 part.

To further prove to myself that the 240C thermal fuse was the source of the problem, I used a pair of alligator clips to bypass the fuse all together. With alligator clips in place on my workbench, I turned the machine on and sure enough the heating coil powered right up, and got hot very quickly. So, the 240C thermal fuse was definitely the problem!

5 - Replacement Thermal Fuses

Looking for a few replacement fuses, Jimmy kindly pointed me to goodmans.net where I ordered three 216C and three 240C thermal fuses. I ordered three of each just in case I ruined one or two during the repair or needed more to fix a another blown fuse down the road. In total, the six replacement fuses plus expedited shipping came out to just over 14-bucks. Not bad at all. You might ask why didn't I just drive down to the local Radio Shack? Well, smart guy, turns out Radio Shack does not carry these type of specialty fuses. I'm sure I could have hunted something down at a local electronics parts supplier, but I had no interest in wasting my time. In any event, the exact manufacturer part numbers of the replacement fuses I ordered online are as follows:

Therm-O-Disc G4A01216C (Microtemp G4 Series, Axial Leaded)
Therm-O-Disc G4A01240C (Microtemp G4 Series, Axial Leaded)

Once my replacement fuses arrived, I grabbed the closest soldering gun and got down to business. Even though I didn't need to, I opted to replace both fuses. I felt like that's a safer bet than replacing one, only to find out I need to open up the unit again a month later to replace the other fuse. Note that if you opt to solder in the replacement fuses, you should remember to heat sink the fuses so the heat of the soldering gun doesn't accidentally trip one. Placing a few alligator clips on each end of the fuse works nicely by helping to direct some heat away from the fuse itself.

I opted to use a few simple butt-connectors between the fuses, and the main line:

With both fuses replaced, I verified each with a multimeter and carefully reassembled my KM7000. I dusted it off, brought it back into my kitchen, plugged it in, and brewed up one heck of a great tasting pot of coffee.

Can you smell that? That's the smell of delicious do-it-yourself satisfaction.

Cheers!

6 - Additional Resources

Here are some additional resources you might find helpful.

https://onyx.koli.ch/x3bn -- Complete set of photos from my repair procedure, including more pictures of those lame triangle bit tamper-proof screws.
https://onyx.koli.ch/x3c2 -- Fantastic blog post from Jimmy Su which helped me discover the problem with my KM7000.
https://onyx.koli.ch/x14m -- My archived digital copy of the Krups KM7000 User-Manual.
https://onyx.koli.ch/x3c3 -- A PDF spec of the G4 fuse from Therm-O-Disc, the manufacturer of the thermal fuses used in the KM7000.

Very special thanks to Jimmy Su for all of his help and advice along the way. Thanks, Jimmy!

Randy M. has confirmed that these instructions also work to fix a broken Krups KM7005. Thanks, Randy!

HOWTO: Setting Up Your Own SVN Server (Using Apache and mod_dav_svn)

2010-03-16T21:30:00Z

Setting up your own SVN source control server is surprisingly easy. At home, I recently setup an SVN server in a CentOS 5.4 virtual machine with Apache 2.2 and mod_dav_svn. With a little work, I had a secure and fully functional SVN server up and running in about 20 minutes.

Note that this HOWTO is specific to CentOS/RHEL/Fedora. The location of configuration files, and other tools, might be different depending on your Linux distro. For the most part though, everything should be pretty similar and you should be able to figure it out.]]> 1 - Install Apache, Subversion, mod_dav_svn and mod_ssl

On CentOS, installing the Apache web-server, Subversion, and the Apache mod_dav_svn and mod_ssl modules are a snap with yum:

#(root)/> yum -y install httpd subversion mod_dav_svn mod_ssl openssl

If you're on Ubuntu you can probably install the required packages using apt-get install. Note that you need to install mod_ssl if you plan on securing your SVN server with HTTPS. If you don't care about HTTPS, then you can ignore mod_ssl and skip Step #6 below.

2 - Create your SVN Root Directory Structure

On my SVN server, I created a new SVN root at /svn. From here on out, all of my SVN repositories will live under /svn/repos.

#(root)/> mkdir -p /svn/repos
#(root)/> chown -R apache:apache /svn

Once done, you're ready to create your first SVN repository.

3 - Create your First Repository

Using the svnadmin command, create a repository under /svn/repos. For the sake of this example, the repository I'm creating is named "myproject". Of course, you can name your own repository whatever you'd like. Oh, and you can create as many repositories as you'd like under /svn.

#(root)/> cd /svn/repos
#(root)/svn/repos> svnadmin create --fs-type fsfs myproject
#(root)/svn/repos> chown -R apache:apache myproject
#(root)/svn/repos> chmod -R g+w myproject
#(root)/svn/repos> chmod g+s myproject/db

You'll notice that the svnadmin command created a new directory named myproject/. If you look inside myproject/ you'll see a bunch of SVN repository data and configuration files.

#(root)/svn/repos> ll myproject
total 28
drwxrwxr-x 2 apache apache 4096 Mar 13 11:49 conf
drwxrwxr-x 2 apache apache 4096 Mar 13 12:01 dav
drwxrwsr-x 5 apache apache 4096 Mar 13 12:23 db
-r--rw-r-- 1 apache apache 2    Mar 13 11:49 format
drwxrwxr-x 2 apache apache 4096 Mar 13 11:49 hooks
drwxrwxr-x 2 apache apache 4096 Mar 13 11:49 locks
-rw-rw-r-- 1 apache apache 229  Mar 13 11:49 README.txt

Great, looks like our new SVN repository was setup correctly!

4 - Configure mod_dav_svn

Now that you have an SVN repository setup and ready to go, let's configure Apache and the mod_dav_svn module. Open /etc/httpd/conf.d/subversion.conf in your favorite text editor, and tweak the configuration to match your installation. My subversion.conf file looks like this:

LoadModule dav_svn_module     modules/mod_dav_svn.so
LoadModule authz_svn_module   modules/mod_authz_svn.so



   DAV svn
   SVNParentPath /svn/repos

   # Require SSL connection for password protection.
   SSLRequireSSL

   AuthType Basic
   AuthName "Marks SVN Server"
   AuthUserFile /svn/repos/users
   Require valid-user

First, note that when you install mod_dav_svn using yum, the installation process will create a standard cookie cutter template /etc/httpd/conf.d/subversion.conf for you. This template has a LimitExcept directive in it, and a few other things. For security, I think it's best to require a user to authenticate before they are able to issue any request. Hence, why I removed the LimitExcept directive and did my own thing. If you want your SVN server to be read-only for anonymous users, and read-write for authenticated users, then my subversion.conf file is not for you. My subversion.conf file shown above allows no anonymous access; all users must authenticate (enter a valid username and password) before they can do anything with the SVN server.

Second, note that I have enabled the SSLRequireSSL directive. This triggers mod_dav_svn to reject all non-HTTPS requests. This ensures that any communication between the server and my SVN client will be sent via HTTPS; usernames, passwords, and source code will be reasonably secured. I'll show you how to setup HTTPS here in a moment. Note that if you don't want to enable HTTPS on your SVN server, then you can comment out or remove the SSLRequireSSL line in your subversion.conf configuration file.

Finally, note that my AuthUserFile is /svn/repos/users. This is a standard Apache htpasswd file that we'll create in the next step.

5 - Create your SVN Users File

Create your SVN users file using the htpasswd command. This is the file that stores a list of usernames and passwords declaring who is allowed to access your SVN server.

#(root)/> htpasswd -c /svn/repos/users mark

Replace "mark" above with your desired username. Repeat this command for however many users you need to add access.

6 - Configure mod_ssl and Setup HTTPS

If you have decided to make your SVN sever an HTTPS only server, we'll need to setup Apache's HTTPS configuration. This involves tweaking /etc/httpd/conf.d/ssl.conf and creating a new self-signed SSL certificate. If you're interested, I wrote a blog post a while back explaining how to create your own self-signed SSL certificate here. However, for your convenience, I've included the same set of instructions below. Note that my SVN server is named svn.kolich.local. Yours will probably be different. Whatever it is, make sure that you enter the correct server name when openssl prompts you for a "Common Name" in your certificate. The "Common Name" in your SSL certificate should match the fully qualified name of your SVN server.

Note that if you have an SSL certificate signed by a legitimate Certificate Authority (Network Solutions, Verisign, Thawte) you shouldn't need to generate a new SSL key and self-signed certificate. You can simply use the one issued to you by your CA.

First, create a new SSL key with the openssl command:

#(root)/> mkdir /etc/httpd/ssl
#(root)/> cd /etc/httpd/ssl
#(root)/etc/httpd/ssl> openssl genrsa 4096 > svn.kolich.local.key

Now that you have a private key, create a self-signed certificate:

#/etc/httpd/ssl> openssl req -new -key svn.kolich.local.key -x509 \
     -days 1095 -out svn.kolich.local.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:California
Locality Name (eg, city) [Newbury]:My Town
Organization Name (eg, company) [My Company Ltd]:Mark Kolich
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:svn.kolich.local
Email Address []:

Finally, edit /etc/httpd/conf.d/ssl.conf to point to your newly generated SSL key and SSL certificate. This involves updating the SSLCertificateFile and SSLCertificateKeyFile directives accordingly:

##
## SSL Virtual Host Context
##


 ...
 SSLCertificateFile /etc/httpd/ssl/svn.kolich.local.crt
 SSLCertificateKeyFile /etc/httpd/ssl/svn.kolich.local.key
 ...

Note that you should not place your SSL private key and certificate in a location accessible by the web-server. Usually placing them under /etc/httpd is sufficient. It would be less desirable and quite insecure to place them under /var/www/html for example.

7 - Configure HTTP to HTTPS Redirection

If you've bothered to setup HTTPS in the previous step, you probably want Apache to gracefully redirect clients from HTTP to HTTPS. If you don't automatically redirect, and you have SSLRequireSSL enabled in your subversion.conf file, when clients try to communicate with your SVN server via HTTP they'll see a 403 Forbidden error. Instead, let's 301 Moved Permanently redirect them to HTTPS. Open /etc/httpd/conf/httpd.conf in your favorite text editor, jump to the bottom of the file, and edit your VirtualHost configuration. Mine is as follows:

NameVirtualHost *:80



  ServerAdmin example@example.com
  DocumentRoot /var/www/html
  ServerName svn.kolich.local
  ServerAlias svn

  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^/(.*)$ https://svn.kolich.local/$1 [R=301,L]

  ErrorLog logs/svn.kolich.local-error_log
  CustomLog logs/svn.kolich.local-access_log common

Save it, and you're done. Now when a client tries to communicate with my SVN server via HTTP, it'll see a 301 Moved Permanently redirect to HTTPS. If my SVN client is smart enough, it will gracefully follow this redirect to HTTPS, and all is well. Of course, you'll need to change the HTTPS URL shown above in the RewriteRule directive to match your server hostname (your SVN server is not svn.kolich.local).

8 - Start Apache, and Enjoy

That's it! Start Apache and checkout your new repository.

#(root)/> /etc/init.d/httpd start

On another machine, try to checkout the repository:

#(mark@fiji)~> svn co http://svn.kolich.local/svn/myproject
svn: PROPFIND request failed on '/svn/myproject'
svn: PROPFIND of '/svn/myproject': 301 Moved Permanently (http://svn.kolich.local)

Yep, HTTP to HTTPS redirection is working as expected. Unfortunately my SVN client isn't smart enough to follow the redirect on its own. Oh well, change that repository URL to HTTPS, and try again:

#(mark@fiji)~> svn co https://svn.kolich.local/svn/myproject
Error validating server certificate for 'https://svn.kolich.local:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: svn.kolich.local
 - Valid: from Mar 15 20:17:38 2010 GMT until Mar 14 20:17:38 2013 GMT
 - Issuer: Mark Kolich, California, US
 - Fingerprint: ff:ee:b6:9c:d8:d7:78:3b:ce:9e:09:dd:4a:99:93:11:3e:12:07:85
(R)eject, accept (t)emporarily or accept (p)ermanently? p
Authentication realm:  Marks SVN Server
Password for 'mark': ...
A    myproject
Checked out revision 0.

It worked! Note that the "Error validating server certificate" warning is because I'm using a self-signed SSL certificate. When SVN asks if you want to accept the certificate, if you permanently accept it you will not be prompted about this again. If you use an SSL certificate issued by a real Certificate Authority like Network Solutions, Verisign, or Thawte, you shouldn't see this warning.

We're all set! Time to start hackin'!

Cheers.

HOWTO: Setting Up Your Own Local DNS Server

2010-03-16T04:10:00Z

This weekend I setup my own SVN source control server running inside of a CentOS 5.4 virtual machine (fun project, ~~another blog post on this to come soon~~ here's my how to setup your own SVN server post). Once my new SVN server was setup and ready to roll, I tried checking out one of the repositories. From the nearest command line, I started typing "svn co http://192.168.1." ... damn, what was the IP-address of my local SVN server again? I just picked an IP-address about 15 minutes ago during my CentOS install, but already forgot it! Ok, time to setup a decent local DNS server for my home network. Since 2006, I've been limping along manually editing my /etc/hosts files on multiple machines and memorizing the IP-addresses of critical devices on my network. Looking back, this was just plain silly.]]> 1 - My Network Topology

My network topology, shown in the diagram below, isn't too complicated. In fact, it's probably quite standard for an average geek. I'm not very good at making pretty charts and graphics, so don't laugh at my artistic abilities. In any event, I've got your typical wireless router and firewall appliance connected to an HP Procurve 1800-8G Gigabit switch which fans out the bandwidth from there. Generally speaking, most traffic on my home network is local so everything sits behind the HP Procurve Gigabit switch. I'm more concerned about the internal network speed between devices, instead of device to the outside world. And of course, all of my devices sit behind NAT, which the router handles for me.

2 - The Problem

The DNS for my publicly visible domain names are hosted by my registrar, Network Solutions. I'm not planning on hosting my own DNS for these services anytime soon, so I just needed something to help me out around the house. In other words, I just need a local DNS server so I can resolve addresses internally on my network. I was getting really tired of typing "ping 192.168.1.102" when I could have been typing "ping somebox" this whole time. In addition to resolving locally, if the DNS server receives a request to resolve a name it doesn't know about, I want it to forward the request to my favorite DNS provider, OpenDNS. So if my DNS server receives a request for "somebox", and "somebox" is a local device on my network, it should resolve to 192.168.1.whatever. However, if it receives a request to resolve google.com, or some other domain it knows nothing about, it should forward the request to OpenDNS and cache the response if necessary.

I should also add that every device with a name on my local network, has a hostname that ends with "kolich.local" (this is my new DNS zone). This is so I can distinguish internal devices from servers, and services, running externally under kolich.com, koli.ch, or another domain.

3 - Installing the DNS Server

I cloned another CentOS Linux virtual machine, booted it up, and installed Bind (an open source DNS server usually called "named"):

#(dns)/> yum -y install bind

As of 3/15/10 CentOS 5.4 ships with Bind 9.3.6, which should be perfectly stable for what I need. This new virtual machine is now my own private internal DNS server, not visible to the outside world. Once installed, I used chkconfig to start named on system startup:

#(dns)/> /sbin/chkconfig --level 35 named on

You should note that named on CentOS, and probably other newer Linux distros, runs inside of a chroot'ed environment which I have no interest in explaining here. If you want to read more about chroot, check out this decent overview of it on Wikipedia. For the sake of this blog post, you probably don't have to care about how named runs under chroot.

4 - Configuring the DNS Server

Once installed, I tweaked my /etc/named.conf file so it looks something like this:

options {
  directory           "/var/named"; // the default
  dump-file           "data/cache_dump.db";
  statistics-file     "data/named_stats.txt";
  memstatistics-file  "data/named_mem_stats.txt";
  version             "currently unavailable";
  ;; Forward anything this DNS server can't resolve to OpenDNS
  forwarders { 208.67.222.222; 208.67.220.220; };
};

;; All devices under my local network sit behind
;; the kolich.local zone
zone "kolich.local" in {
  type master;
  file "kolich.local.ns";
  allow-update { none; };
};

;; For reverse lookups, going IP addy back to a hostname
zone "1.168.192.in-addr.arpa" in {
  type master;
  file "1.168.192.in-addr.arpa.ns";
  allow-update { none; };
};

I have two zone files: kolich.local.ns and 1.168.192.in-addr.arpa.ns. The first zone file maps a list of hostname's to IP-addresses, and the second defines a reverse lookup zone that maps a list IP-addresses back to hostname's. Note that because named is running under a chroot'ed environment, it's usually safest to place these zone files under /var/named/chroot/var/named (yes, that is the correct path) on a typical CentOS/RHEL/Fedora installation.

In my /etc/named.conf file, notice I've given two forwarders in the options section. This declaration lets me tell named to forward any requests it cannot resolve to OpenDNS at 208.67.222.222 or 208.67.220.220.

Now let's take a quick look at my zone files. First, here is my /var/named/chroot/var/named/kolich.local.ns zone file for my new "kolich.local" zone:

$TTL    1d
kolich.local.  IN    SOA   ns.kolich.local. support.kolich.com. (
    2010031500 ; se = serial number
    3h         ; ref = refresh
    15m        ; ret = update retry
    3w         ; ex = expiry
    3h         ; min = minimum
    )

    IN    NS    ns.kolich.local.

; private hosts
ns         IN    A    192.168.1.2

cat        IN    A    192.168.1.3
fish       IN    A    192.168.1.4
whale      IN    A    192.168.1.5
monkey     IN    A    192.168.1.6
horse      IN    A    192.168.1.7
cow        IN    A    192.168.1.8

And here's my /var/named/chroot/var/named/1.168.192.in-addr.arpa.ns reverse lookup zone file that is used to map IP-addresses back to hostname's:

$TTL    1d
@   IN    SOA   ns.kolich.local. support.kolich.com. (
    2010031500 ; se = serial number
    3h         ; ref = refresh
    15m        ; ret = update retry
    3w         ; ex = expiry
    3h         ; min = minimum
    )

    IN    NS    ns.kolich.local.

; private hosts, reverse lookup
2     IN    PTR    ns.kolich.local.

3     IN    PTR    cat.kolich.local.
4     IN    PTR    fish.kolich.local.
5     IN    PTR    whale.kolich.local.
6     IN    PTR    monkey.kolich.local.
7     IN    PTR    horse.kolich.local.
8     IN    PTR    cow.kolich.local.

Note that my 1.168.192.in-addr.arpa.ns zone file is basically a reverse map of my kolich.local.ns file. Of course, depending on your local network settings, your IP addresses might be different. And no, the systems and devices on my network are not named after animals; the names shown here are just for the sake of this example.

5 - Starting the DNS Server

Ok, once all of your configuration files are in place, it's time to start your new local DNS server:

#(dns)/> /etc/init.d/named start

That's it! Assuming named started correctly, I can configure the clients on my local network to point to my new in-house DNS server. On Linux, this involves editing the domain and nameserver configuration inside of /etc/resolv.conf ...

domain kolich.local
nameserver 192.168.1.2

Once /etc/resolv.conf is configured properly, I can use the nslookup or dig commands to verify that all is working as expected:

#(dns)/> nslookup monkey
Server:         192.168.1.2
Address:        192.168.1.2#53

Name:   monkey.kolich.local
Address: 192.168.1.6

Great! Hostname resolution works well. Now, let's verify that I can go the other way (reverse lookups):

#(dns)/> nslookup 192.168.1.3
Server:         192.168.1.2
Address:        192.168.1.2#53

2.1.168.192.in-addr.arpa    name = cat.kolich.local.

Yup, works fine. So what about that forwarding stuff? Good call, we should also check that the server is forwarding requests for hostname's it can't resolve to OpenDNS:

#(dns)/> nslookup twitter.com
Server:         192.168.1.2
Address:        192.168.1.2#53

Non-authoritative answer:
Name:   twitter.com
Address: 128.242.240.20

Perfect. Internal hosts resolve correctly, and my DNS server is forwarding all requests it can't resolve to OpenDNS as desired!

Finally, with a proper DNS server up and running, I can get back to my coding project.

Enjoy.

Clok: A new way to view time

2010-01-21T01:45:00Z

Clok is a "very fuzzy" world-clock I built, modeled after Caskey Dickson's idea and Java Clok implementation. Clok is not built to give you the exact time in every time zone; for that, get another, more accurate, world clock. Instead, my version of Clok is used to help me answer basic time related questions, like:

Several of my co-workers are based in London; can I email them at the office, or are they at home?
I need to call my girlfriend, but she's in Manila visiting family. Is it daytime there?
Some interesting world news just broke in Dubai; roughly what time is there?

These questions are all a slight variation of the timeless (no pun intended), "what time is it?" The latest version of Clok can be found at http://clok.koli.ch.

1 - How Do I Read this Clok?

Each column represents a single hour, in a 24-hour day. Each hour is color coded, according to the various pieces, or periods, of a typical day. As explained here, "sleeping is obviously the least productive and so that is represented in black. The morning is the time between sleeping and lunch, lunch is a time of recovery and planning for the rest of the day. Then there is the afternoon, which for many is the time when the majority of their work gets done. Finally comes evening, which is personal time, time spent with family, or time taking care of those responsibilities that have to do with running our lives and not earning a paycheck." The colors of each hour in the day directly correspond to these periods.

Each row is a different time zone, modeled after this list on Wikipedia. And finally, the red line is the approximate current time in that time zone.

Clok automatically updates itself every minute while open; no browser refresh is required.

2 - The User Interface

You may have noticed that when you mouseover each row on the Clok, four tiny buttons appear on the far right of each time zone. These represent the various modes: H for human, K for hacker/developer/coder, S for server, and R for raccoon (nocturnal). When clicked, these buttons change the mode of Clok. If you're a relatively normal individual with decent sleeping habits, you will find Human mode most useful. If you, or a friend across the world participates in a more nocturnal lifestyle, then you might enjoy Raccoon mode.

If you find the time zone annotations annoying, and would like to hide them, click anywhere in the orange header bar and the name/UTC ID of each time zone should disappear.

3 - Technical Details

My version of Clok is pure CSS, JavaScript, and PHP (no Flash!). When the DOM is ready, AJAX is used behind the scenes to call a PHP controller on the server, which actually does the work of computing the time in every requested time zone. This controller returns a block of JSON that maps a time zone to the position of its red "current time" bar. The JavaScript loops over these JSON objects in the response, and uses jQuery to move (animate) the red bars into position. To stay current, an interval fires once every 60,000 ms (1 minute) which triggers Clok to refresh itself.

4 - Where Can I Find this Clok?

Try http://clok.koli.ch.

Enjoy.

Java: JumpToLine, Jump or Seek to a Line in a File

2010-01-18T20:13:00Z

Seeking to a line number in a text file isn't too hard to implement in Java if you use a few common and trusted API's like Apache's Commons I/O library. Recently I needed some Java that could automatically seek to a given line in a file and then remember the line number of the last line read. The next time I open the log reader, it should automatically seek itself to the last line read, and let me read any subsequent lines added by another user or process. This is perfect for log file monitoring: the first invocation of the reader would read lines 1 through X and the next invocation would read lines X+1 through Y, and so on. Using Apache's Commons I/O API, this isn't difficult at all. You may or may not know that the Commons I/O API contains a very convenient LineIterator class, which lets the developer iterate over lines in a file using a Reader.

With that in mind, meet JumpToLine, a somewhat hackish class I wrote that wraps Apache's Commons I/O LineIterator in a way that lets you seek ahead to a specific line in a file, and is smart enough to remember the last line read (so that you don't read the same line twice).]]> Example #1

Here's how you might use JumpToLine to seek to line 10 in mylog.log, then read that line and every line after it:

final JumpToLine jtl = new JumpToLine(new File("mylog.log"));

try {
  // Open the underlying reader and LineIterator.
  jtl.open();
  // Seek to line 10; will throw a NoSuchElementException if
  // out of range.
  jtl.seek(10);
  // While there are any lines after and including line 10,
  // read them.
  while(jtl.hasNext()) {
    final String line = jtl.readLine();
    System.out.println(line);
  }
} catch (Exception e) {
  e.printStackTrace(System.err);
} finally {
  // Close the underlying reader and LineIterator.
  jtl.close();
}

Example #2

Here's how you might use JumpToLine to seek to the last line read in mylog.log, and then read any subsequent lines added to the file since it was last read:

final JumpToLine jtl = new JumpToLine(new File("mylog.log"));

try {
  // Open the underlying reader and LineIterator.
  jtl.open();
  // Seek to the last line read since we last tried to
  // read any lines from this file.
  jtl.seek();
  // While there are any more lines to read from the last
  // line read position, then read them.
  while(jtl.hasNext()) {
    final String line = jtl.readLine();
    System.out.println(line);
  }
  // For grins, what is the last line number we read?
  System.out.println("Last line number read: " +
      jtl.getLastLineRead());
} catch (Exception e) {
  e.printStackTrace(System.err);
} finally {
  // Close the underlying reader and LineIterator.
  jtl.close();
}

Download JumpToLine here. Note that JumpToLine is dependent on Apache's Commons I/O API which you can download here.

Happy New Year's

2009-12-31T15:55:00Z

Last year I wrote up a quick blog post to ring in the New Year, highlighting some of my accomplishments and failures of 2008. In that spirit, keeping the tradition alive, here's my 2009 in a nutshell:

I kicked off 2009 figuring out how to block Trackback Spam, and wrote up an opinionated piece re: why I dislike and therefore don't use, Facebook. MyCougarLand.com failed to update their DNS records, which generated a lot of traffic (not the kind you would expect) to my blog. In late January, I discovered that I own a bottle of wine from a convicted felon. Ala February, I joined Twitter, and heard that Network Solutions featured my WHOIS Firefox Plugin on their homepage. I bought a new battery backup UPS for my home data center, after a little physics exercise to discover the exact type of UPS I needed. I launched kolich.cc, but then later built and released Onyx. With summer approaching, I made my own solar shield out of cardboard and tinfoil while thinking up ten awesome .htaccess hacks for Movable Type. Continuing the awesomeness, I launched a mobile version of my blog at kolich.mobi for all of my readers on mobile devices. Like any good year, a few of my systems crashed, then recovered but I had fun with HP LaserJet printers at the office. I registered kolich.tel, went green for Iran, and figured out how to include base-64 encoded binary data in CSS. And, of course, I released Gagawa PHP 1.2 before hiking Mount San Gorgonio in the San Bernardino National Forest, but not before I wasted several days at the office on a stupid bug. I registered koli.ch (thank you Network Solutions!), and gave hot linkers a nice big 5000x5000px animated GIF to chew on. And to cap it all off, I blew the whistle on Twitter (they're spying on us) and then dove into some C++ to discover how Windows UAC works, which by the way, reminded me how much I hate Windows.

2009, the end.

Experimenting with More Domains, and Fun with Http 302 Found

2009-12-29T00:10:00Z

My somewhat narcissistic obsession with domain names involving my last name has recently driven me to purchase markkolich.com. The saga leading up to this acquisition started when I snagged koli.ch, and saw my ranking in a few search engine results absolutely plummet. I kicked off my own investigation and realized that Google doesn't seem to understand a custom domain hack of my last name. For the record, Yahoo! does. I suppose that's one thing Yahoo! actually does better than Google: properly interpreting domain hacks and other URL trickery.

In any event, I setup the markkolich.com VirtualHost to redirect to mark.koli.ch via a 302 Found. It seems that when Google and Yahoo encounter an HTTP 302 Found, they gracefully redirect their spiders to the destination but maintain the redirector's address. For example, this URL https://onyx.koli.ch/x2fr redirects to http://plugins.jquery.com/project/Timer using a 302 Found. When spiders encounter https://onyx.koli.ch/x2fr they are forwarded to the destination, but add http://plugins.jquery.com/project/Timer under https://onyx.koli.ch/x2fr to their index. In other words, the jQuery Timer plugin page will be listed under the URL https://onyx.koli.ch/x2fr in the search results! Interestingly enough, HTTP 301 Moved Permanently does not exhibit this behavior.

For the time being, I'm going to see how the new domain stacks up against koli.ch. My expectation is that spiders will index markkolich.com, see a 302 Found, and add it to their index as is without worrying about the destination address.

Stay tuned.

A Better Way to Include JavaScript and CSS in your Web-Apps (Break Proxy and Browser Caching too!)

2009-12-23T02:30:00Z

Most web-apps include a number of JavaScript and CSS files, that in most cases, need to be included/sourced on every page of the application. Doing so can bring up a few, while trivial, often annoying problems:

Dependencies between scripts. For example, if you attempt to use a jQuery plugin before the base jQuery library has loaded into the browser you'll obviously see an error. Working out these dependencies before hand can save you a few headaches.
Web-browsers and web-proxies cache your JavaScript and CSS files too often. I can't count the number of times on two hands when I made a small change to a JavaScript file, saved the change, refreshed my browser, and .... nothing. The web-browser didn't load the changed file; instead, it loaded the JavaScript from cache. Using a mechanism to always force the web-browser or web-proxy to reload your JavaScript and CSS can save you time.

Meet Gagawa, an open source object-oriented HTML generation engine written in Java and PHP. You probably wouldn't build an entire site with Gagawa, but it's absolutely perfect for small HTML tasks, like solving this irritating dependency and caching problem.]]> Using Gagawa PHP, problem solved:


require_once("gagawa.php");

$js = array(
  // The base jQuery library, needed by all other
  // jQuery plugins and your jQuery code.
  "/js/jquery-1.3.2.min.js",

  // jQuery plugins; jQuery base needs to be loaded
  // before you source these.
  "/js/jquery.ui.min.js",
  "/js/jquery.hotkeys.min.js",

  // Some other, unrelated JavaScript library like
  // underscore JS.
  "/js/underscore.min.js",

  // Your JavaScript, with dependencies on jQuery, the
  // jQuery plugins and Underscore JS, should be loaded
  // last.
  "/js/yourjs.min.js"
  );

$css = array(
  "/css/othercss.min.css",
  "/css/yourcss.css"
  );

// Build  for each JavaScript
// we need, in order. Use time() to break browser
// caching.
foreach ($js as $url) {
  $script = new Script("text/javascript");
  echo $script->setSrc($url."?".time())->write();
  }

// Build  for each CSS file we need,
// in order.  Use time() to break browser caching.
foreach ($css as $url) {
  $link = new Link();
  $link->setHref($url."?".time())->setRel("stylesheet")->setType("text/css");
  echo $link->write();
  }

?>

This code snippet generates some solid output that fits nicely into the of your HTML:

Note that each JavaScript and CSS file is loaded and added to your HTML, in order as desired, eliminating any stray dependency issues. Also, note that I'm appending "?" and the current Epoch timestamp to the end of the JavaScript source URL, and CSS Href. PHP's time() function works nicely here because I'll get a different timestamp on each page load. And as far as the browser (or a web-proxy) is concerned, "/js/yourjs.min.js?1261534594" is different than "/js/yourjs.min.js?1234567890" and will be treated as such. With this mechanism, every time you refresh the page, the browser will be forced to reload and reevaluate your scripts. Say goodbye to browsers caching and not recognizing your changes!

You can download the latest version of Gagawa at http://code.google.com/p/gagawa/downloads/list. Also, if you're interested, other Gagawa examples can be found on the Gagawa homepage.

If you're looking for a real, live, example of this technique in action check out Onyx.

Merry Christmas.

Writing Your Own Animation Loop with javax.swing.Timer

2009-12-20T20:55:00Z

In the last week or so, while working on some Java Swing code for a project at work, I hit Swing bug #4480705. When the user clicked a button to start an "activation", I changed the ImageIcon on a JButton to an animated GIF to indicate activity (e.g., "we're doing something, please wait"). So, while the application was busy, I disabled the JButton and set its ImageIcon to an animated spinner. Loading this animated GIF and setting it on the JButton using setIcon() caused the Java Swing EventQueue threads to deadlock, as described here, and my entire application hung. Well technically speaking, not the entire application, but just the Swing portion of the JVM. Clearly, Swing doesn't like animated GIF's as much as the Sun documentation claims.

The solution to this deadlock is to avoid using animated GIF's all together, and write your own animate loop using javax.swing.Timer. That's exactly what I did, and it works great.

For your viewing pleasure, I wrote up a quick demo/example (see Beavis and Butthead screencap) which demonstrates the use of javax.swing.Timer and how you can integrate it into your Swing application.

Download just the source, or the full Eclipse project.

Rock on.