<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
    <title>Mark S. Kolich</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/" />
    <link rel="self" type="application/atom+xml" href="http://mark.koli.ch/atom.xml" />
    <id>tag:,2008-10-25:/1</id>
    <updated>2012-01-03T17:22:39Z</updated>
    <subtitle>Software Engineer, Consultant</subtitle>
    <generator uri="http://www.sixapart.com/movabletype/">Movable Type 4.21-en</generator>

<entry>
    <title>Configuring Apache to Tunnel SSH Through an HTTP Web-Proxy with Proxytunnel</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2011/12/configuring-apache-to-support-ssh-through-an-http-web-proxy-with-proxytunnel.html?rss" />
    <id>tag:mark.koli.ch,2011://1.252</id>

    <published>2011-12-31T21:00:00Z</published>
    <updated>2012-01-03T17:22:39Z</updated>

    <summary><![CDATA[Here's the situation, I'm often on a network that does not allow outbound traffic on port 22.&nbsp; Meaning, I cannot directly "SSH out" from that network to my Linux box at home. Fair enough. However, this network does allow outbound...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="apache" label="apache" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="http" label="http" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="proxies" label="proxies" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="ssh" label="ssh" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="whatcouldpossiblygowrong" label="whatcouldpossiblygowrong" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[Here's the situation, I'm often on a network that <b>does not allow</b> outbound traffic on port 22.&nbsp; Meaning, I cannot directly "SSH out" from that network to my Linux box at home. Fair enough. However, this network <b>does allow</b> outbound traffic on ports 80, 443, and 8443 via a web-proxy. &nbsp;That said, if I want to "SSH out" from this network to my Linux box at home, I can do so with a little tweaking of my remote Apache server and my local SSH client.<br /><br />Here's how ...]]>
        <![CDATA[<br /><span style="font-size: 1.1em; font-weight: bold;">1 - Overview</span><br /><br />First, you'll need to configure your Apache web-server to accept traffic on a port that's acceptable to the web-proxy. &nbsp;In my case, I don't have anything running on port 8443, and the web-proxy allows traffic through port 8443, so that's perfect. &nbsp;Apache will be configured to listen on 8443, and act as a "proxy" between an SSH client and an SSH server (usually the SSH server running on the box you're trying to connect to).<br /><br />Second, on the client side, you'll be using something like <a href="http://proxytunnel.sourceforge.net/">Proxytunnel</a> to punch a hole through the web-proxy allowing your SSH client to connect to an SSH server of your choice.<br /><br />Putting it all together, the basic flow is ...<br /><br /><ol><li>Your local SSH client uses Proxytunnel to connect to web-proxy.corp.example.com:3128</li><li>Web-proxy.corp.example.com connects to Apache running at yourwebserver:8443</li><li>Your Apache server, acting as yet another proxy, connects to yoursshserver:22</li><li>It works!</li></ol><br /><br /><span style="font-size: 1.1em; font-weight: bold;">2 - Install and Configure Proxytunnel</span><br /><br />If you're on Ubuntu, you can install Proxytunnel with the following command:<br /><br /><pre class="prettyprint">#/&gt; sudo apt-get install proxytunnel</pre><br />Once installed, edit your <b>~/.ssh/config</b> file to instruct your SSH client to use Proxytunnel when connecting to the destination host:<br /><br /><pre class="prettyprint">## ~/.ssh/config<br /><br />Host kolich.com
  Hostname kolich.com
  ProtocolKeepAlives 30
  ProxyCommand /usr/bin/proxytunnel \<br />    -p web-proxy.corp.example.com:3128 \<br />    -r kolich.com:8443 -d %h:%p \<br />    -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)"</pre><br />In this example, when my SSH client makes an attempt to connect to kolich.com:22, it will spawn Proxytunnel which will then route my connection through web-proxy.corp.example.com:3128 and on to kolich.com:8443.&nbsp; This seems really convoluted, but it actually works quite well.<br /><br />Note that I'm spoofing a somewhat real User-Agent to prevent suspicion from the system administrators running web-proxy.corp.example.com.&nbsp; If you're a system administrator that runs such a web-proxy, please accept my apologies for making your life even more difficult.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">3 - Configure mod_proxy on Apache</span><br /><br />Now that you've got the client part figured out, you'll need to configure Apache's mod_proxy module to proxy traffic between yourwebserver:8443 and yoursshserver:22.&nbsp; In all likelihood, your web-server and SSH server are the same box.&nbsp; At least, in my home, they are.<br /><br />Oh, and I assume you already have Apache and mod_proxy installed, and working.&nbsp; There are ton of other tutorials and nice blog posts online about how to install and setup Apache if you don't already have it installed and functional.<br /><br />In my Apache virtual host configuration, I've added another V-host listening on port 8443 that will only accept CONNECT requests bound for kolich.com on port 22:<br /><br /><pre class="prettyprint">## Load the required modules.<br />LoadModule proxy_http_module modules/mod_proxy_http.so<br />LoadModule proxy_connect_module modules/mod_proxy_connect.so<br /><br />## Listen on port 8443 (in addition to other ports like 80 or 443)<br />Listen 8443<br /><br />&lt;VirtualHost *:8443&gt;<br /><br />  ServerName youwebserver:8443<br />  DocumentRoot /some/path/maybe/not/required<br />  ServerAdmin admin@example.com<br /><br />  ## Only ever allow incoming HTTP CONNECT requests.<br />  ## Explicitly deny other request types like GET, POST, etc.<br />  ## This tells Apache to return a 403 Forbidden if this virtual<br />  ## host receives anything other than an HTTP CONNECT.<br />  RewriteEngine On<br />  RewriteCond %{REQUEST_METHOD} !^CONNECT [NC]<br />  RewriteRule ^/(.*)$ - [F,L]<br /><br />  ## Setup proxying between youwebserver:8443 and yoursshserver:22<br /><br />  ProxyRequests On<br />  ProxyBadHeader Ignore<br />  ProxyVia Full<br />  <br />  ## IMPORTANT: The AllowCONNECT directive specifies a list<br />  ## of port numbers to which the proxy CONNECT method may<br />  ## connect.  For security, only allow CONNECT requests<br />  ## bound for port 22.<br />  AllowCONNECT 22<br /><br />  ## IMPORTANT: By default, deny everyone.  If you don't do this<br />  ## others will be able to connect to port 22 on any host.<br />  &lt;Proxy *&gt;<br />    Order deny,allow<br />    Deny from all<br />  &lt;/Proxy&gt;<br /><br />  ## Now, only allow CONNECT requests bound for kolich.com<br />  ## Should be replaced with yoursshserver.com or the hostname<br />  ## of whatever SSH server you're trying to connect to.  Note<br />  ## that ProxyMatch takes a regular expression, so you can do<br />  ## things like (kolich\.com|anothersshserver\.com) if you want<br />  ## to allow connections to multiple destinations.<br />  &lt;ProxyMatch (kolich\.com)&gt;<br />    Order allow,deny<br />    Allow from all<br />  &lt;/ProxyMatch&gt;<br /><br />  ## Logging, always a good idea.<br />  LogLevel warn<br />  ErrorLog logs/yourwebserver-proxy_error_log<br />  CustomLog logs/yourwebserver-proxy_request_log combined  <br /><br />&lt;/VirtualHost&gt;<br /></pre><br />Once you get everything integrated, restart Apache and you should be golden.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">4 - Under the Hood</span><br /><br />To prove that everything works, let's try a few things.<br /><br />First I'm going to telnet to web-proxy.corp.example.com:3128.&nbsp; Then, I'm going to tell it to connect to kolich.com:8443.&nbsp; Finally, I'm going to tell Apache on kolich.com:8443 to connect to kolich.com:22.&nbsp; This is <b>exactly the same flow used by Proxytunnel</b> under the hood.<br /><br /><pre class="prettyprint">(mark@ubuntu)~&gt; telnet web-proxy.corp.example.com 3128<br />Trying 10.10.10.10...<br />Connected to web-proxy.corp.example.com (10.10.10.10).<br />Escape character is '^]'.<br />CONNECT kolich.com:8443 HTTP/1.1<br />Host: kolich.com<br /><br />HTTP/1.0 200 Connection Established<br /><br />CONNECT kolich.com:22 HTTP/1.1<br />Host: kolich.com<br /><br />HTTP/1.0 200 Connection Established<br />Proxy-agent: Apache<br /><br />SSH-2.0-OpenSSH_4.3<br /></pre><br />Sweet!&nbsp; Notice the raw "SSH-2.0-OpenSSH_4.3" response from the SSH server, indicating a successful connection.&nbsp; Now, If I was a real SSH client, I'd continue the handshake and away we go.<br /><br />So, from a real SSH client with Proxytunnel enabled ...<br /><br /><pre class="prettyprint">(mark@ubuntu)~&gt; ssh mark@kolich.com<br />Via web-proxy.corp.example.com:3128 -&gt; kolich.com:8443 -&gt; kolich.com:22<br />mark@kolich.com's password:<br /><br />Last login: Sat Dec 31 12:53:22 2011 from gateway.kolich.local<br />(mark@server)~&gt;</pre><br />It works!&nbsp; Notice the intermediate "Via web-proxy.corp.example.com:3128 -&gt; kolich.com:8443 -&gt; kolich.com:22" output from Proxytunnel telling me what it's doing to connect.&nbsp; And of course, look at that beautiful shell prompt.<br /><br />SSH through a web-proxy, I love it.<br /><br />Enjoy.<br />]]>
    </content>
</entry>

<entry>
    <title>Resolve Custom Object Arguments in a Spring 3 Controller @RequestMapping Annotated Method</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2011/07/auto-populate-non-standard-arguments-in-a-spring-3-controller-requestmapping-method.html?rss" />
    <id>tag:mark.koli.ch,2011://1.251</id>

    <published>2011-07-31T01:25:00Z</published>
    <updated>2011-07-31T16:16:29Z</updated>

    <summary><![CDATA[Spring 3 is great at automatically resolving standard arguments into a controller request method. &nbsp;For example, a primitive Spring controller might look like this ...@Controller @RequestMapping(value="/somepath") public class MyController { @RequestMapping(method={RequestMethod.GET, RequestMethod.HEAD}) public ModelAndView someMethod(final HttpServletRequest request, final Principal principal)...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="spring" label="spring" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="springsecurity" label="spring-security" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[Spring 3 is great at automatically resolving standard arguments into a controller request method. &nbsp;For example, a primitive Spring controller might look like this ...<br /><br /><pre class="prettyprint">@Controller
@RequestMapping(value="/somepath")
public class MyController {<br /><br />  @RequestMapping(method={RequestMethod.GET, RequestMethod.HEAD})
  public ModelAndView someMethod(final HttpServletRequest request,
    final Principal principal) {<br />    // Extract some special object needed to process the request from<br />    // the session -- this object is bound to the session elsewhere on<br />    // a successful authentication.<br />    final MyObject obj = (MyObject)request.getSession().getAttribute("myobjkey");<br />    // Do actual work.<br />    /* ... */<br />    return new ModelAndView("someview");<br />  }<br /><br />}<br /></pre><br />In this case, Spring knows the <a href="http://download.oracle.com/javaee/6/api/javax/servlet/http/HttpServletRequest.html">HttpServletRequest</a> argument represents the incoming Servlet request, and the <a href="http://download.oracle.com/javase/6/docs/api/java/security/Principal.html">Principal</a> argument is the object representing the authenticated user (in the event that you're using Spring Security to manage authentication in your web-application).&nbsp; On method invocation, Spring automatically resolves these arguments for you.&nbsp; Neat!<br /><br />However, the repetition becomes obvious where in every controller, you need to fetch the same MyObject from the session, over and over again.&nbsp; Instead of repeating that line of code in every method of every controller that needs access to MyObject, what if you could tell Spring how to resolve MyObject automatically on invocation?]]>
        <![CDATA[Let's say you've defined a custom object and <b>bound it to the session on a successful authentication</b> (either on your own or via Spring Security) ...<br /><br /><pre class="prettyprint">import java.util.UUID;<br /><br />public final class MyObject {<br /><br />  // A unique and static identifier.<br />  private UUID id_;<br /><br />  public MyObject() {<br />    id_ = UUID.randomUUID();<br />  }<br /><br />  public UUID getId() {<br />    return id_;<br />  }<br /><br />}<br /></pre><br />Good news!&nbsp; Spring can automatically resolve an argument of type MyObject if used as an argument into a controller request method.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">1 - Meet WebArgumentResolver</span><br /><br />The <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/bind/support/WebArgumentResolver.html">WebArgumentResolver</a> interface let's you define a bean that tells Spring where to find a custom argument of any type when used in a controller request method.&nbsp; Here's an example that tells Spring where to find an argument of type MyObject bound to the session ...<br /><br /><pre class="prettyprint">import static javax.servlet.jsp.PageContext.SESSION_SCOPE;<br /><br />import org.springframework.core.MethodParameter;<br />import org.springframework.web.bind.support.WebArgumentResolver;<br />import org.springframework.web.context.request.NativeWebRequest;<br /><br />public class SessionExtractingWebArgumentResolver implements WebArgumentResolver {<br /><br />  @Override<br />  public Object resolveArgument(final MethodParameter mp,<br />    final NativeWebRequest nwr) throws Exception {<br />    Object argument = UNRESOLVED;<br />    if(mp.getParameterType().equals(MyObject.class)) {<br />      // Assumes that a MyObject is bound to the session elsewhere using<br />      // attribute key "myobjkey" on a successful authentication.<br />      if((argument = nwr.getAttribute("myobjkey", SESSION_SCOPE)) == null) {<br />        throw new Exception("Fail, no MyObject bound to session!");<br />      }<br />    }<br />    return argument;<br />  }<br /><br />}<br /></pre><br />Now that we've defined our <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/bind/support/WebArgumentResolver.html">WebArgumentResolver</a> bean, we can wire it into in our Spring MVC configuration.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">2 - Wire it Into Spring MVC</span><br /><br />Wire your custom <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/bind/support/WebArgumentResolver.html">WebArgumentResolver</a> into Spring's <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/servlet/mvc/annotation/AnnotationMethodHandlerAdapter.html">AnnotationHandlerMethodAdapter</a> using a quick declaration in your Spring MVC XML configuration.&nbsp; Here's an example ...<br /><br /><pre class="prettyprint">&lt;bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter"&gt;<br />  &lt;property name="customArgumentResolver"&gt;<br />    &lt;bean class="your.package.SessionExtractingWebArgumentResolver" /&gt;<br />  &lt;/property&gt;<br />&lt;/bean&gt;<br /></pre><br />Yay!&nbsp; Now, if Spring encounters a MyObject argument in a controller request method, it will look it up using our custom <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/bind/support/WebArgumentResolver.html">WebArgumentResolver</a> and extract it from the session accordingly.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">3 - An Improved Controller</span><br /><br />Now that Spring can resolve MyObject automatically, we can use it as an argument into any controller request method ...<br /><br /><pre class="prettyprint">@RequestMapping(method={RequestMethod.GET, RequestMethod.HEAD})<br />public ModelAndView betterMethod(<strong>final MyObject my</strong>) {<br />  /* ... */<br />}<br /></pre>Great!&nbsp; No more ugly repetition, and less code.&nbsp; That's a win-win, baby.<br /><br />Enjoy.<br />]]>
    </content>
</entry>

<entry>
    <title>More Fun with RFC 2397 -- The &quot;data&quot; URL scheme and Mobile Networks</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2011/01/more-fun-with-rfc-2397----the-data-url-scheme.html?rss" />
    <id>tag:mark.koli.ch,2011://1.250</id>

    <published>2011-01-21T22:20:00Z</published>
    <updated>2011-01-21T23:46:57Z</updated>

    <summary><![CDATA[In July of '09, when I first learned of the "data" URL scheme, I was pumped.&nbsp; With a little work, my web-applications could use the "data" URL scheme to embed actual base-64 encoded binary image data directly inside of my...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="compression" label="compression" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="css" label="css" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="http" label="http" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="url" label="url" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[In <a href="http://mark.koli.ch/2009/07/howto-include-binary-image-data-in-cascading-style-sheets-css.html">July of '09, when I first learned of the "data" URL scheme</a>, I was pumped.&nbsp; With a little work, my web-applications could use the "data" URL scheme to embed actual base-64 encoded binary image data directly inside of my HTML and CSS.&nbsp; In the <a href="http://mark.koli.ch/2009/07/howto-include-binary-image-data-in-cascading-style-sheets-css.html">same post</a>, I subsequently commented on why this scheme can be incredibly useful, especially for mobile web-applications or API's that service mobile apps.&nbsp; Even with significant advances in wireless networks over the past several years, traditional HTTP continues to lag (for the most part) over poor 3G and 4G networks.&nbsp; For this reason, the "data" URL scheme can be a life saver -- you can embed binary image data directly inside of your HTML and CSS, freeing the device from initiating wasteful HTTP transactions to load these images later.<br /><br />Today marked yet another personal milestone for my usage of the "data" URL scheme.&nbsp; Building an API that services a mobile app for the HP/Palm webOS platform, I quickly rediscovered the importance of this scheme. &nbsp;It turns out I can embed base-64 encoded binary image data in a JSON response payload that is sent directly to a wireless webOS device! &nbsp;What this means, is that I can build my API resource to send everything the app requested, including any additional external resources like images, in a single HTTP response!]]>
        <![CDATA[<br /><span style="font-size: 1.1em; font-weight: bold;">1 - An Example</span><br /><br />Imagine the app is fetching details about a user from my API.&nbsp; The HTTP request leaving the app might look something like this ...<br /><br /><pre class="prettyprint">GET /user/markkolich.json HTTP/1.1<br />Host: api.example.com<br />User-Agent: webOS</pre><br />... and a normal HTTP response might look something like this ...<br /><br /><pre class="prettyprint">HTTP/1.0 200 OK<br />Date: Fri, 21 Jan 2011 23:09:32 GMT<br />Content-Type: application/json;charset=UTF-8<br />Content-Length: 78<br />Vary: Accept-Encoding,User-Agent<br />Connection: close<br /><br />{<br /> "user":"markkolich",<br /> "name":"Mark Kolich",<br /> "avatar":"http://api.example.com/images/markkolich.png"<br />}</pre><br />Nothing special.&nbsp; But note that the API response, a JSON object, contains a URL to an avatar image which in all likelihood the app will need to fetch and display later.&nbsp; The problem here, is that now that the app has the response in its hands, it has to turn around and kick off yet another HTTP transaction to load the image from the provided URL.&nbsp; And that additional fetch could be very painful over a slow wireless network with quite a bit of latency.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">2 - The "data" URL Scheme to the Rescue</span><br /><br />Instead of the API sending just a URL that points to an image, it could also include the actual image data itself, a base-64 encoded "data" URL in the JSON response.&nbsp; For example:<br /><br /><pre class="prettyprint">{<br /> "user":"markkolich",<br /> "name":"Mark Kolich",<br /> "avatar":{<br />  "url":"http://api.example.com/images/markkolich.png",<br />  "data_uri":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAA<br />    ABCAAAAAA6fptVAAAAAnRSTlMA/1uRIrUAAAACYktHRAD+8Ij8KQAAAAlwSFlz<br />    AAAASAAAAEgARslrPgAAAAl2cEFnAAAAAQAAAAEAx5Vf7QAAAApJREFUCNdj+A<br />    8AAQEBABu27lYAAAAASUVORK5CYII="<br /> }<br />}<br /></pre><br />I know that this isn't technically valid JSON because I'm line wrapping in the middle of the base-64 encoded image payload; I'm wrapping so that I can fit the entire JSON block into a single column on my blog for display purposes.&nbsp; For the record, your data payload shouldn't have any line breaks in it.<br /><br />Looking at the new response, this is far superior to sending just a URL that points to an image.&nbsp; Yes, we're sending a little more data, but now the app does not have to initiate another HTTP transaction to load the avatar.&nbsp; Due to the usually poor latency of wireless networks (EDGE, 3G, 4G, etc.), and the natural overhead of HTTP, it's far better to send more data at once in a single transaction than over multiple smaller transactions.<br /><br />Finally, in JavaScript, sourcing the encoded image data into an actual Image object is trivial:<br /><br /><pre class="prettyprint">// Assume userObj is the JSON object returned<br />// in my example response above.<br />var userObj = { /*...*/ };<br /><br />// Straight up.<br />(new Image).src = userObj.avatar.data_uri;<br /><br />// Maybe you prefer jQuery?<br />$("&lt;img&gt;").attr("src", userObj.avatar.data_uri);</pre><br />Assuming your app platform supports the "data" URL scheme (yes, the HP/Palm webOS platform does), this strategy wins every time, hands down.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">3 - The "data" URL Format</span><br /><br />Putting it all together, the <a href="http://tools.ietf.org/html/rfc2397">RFC 2397</a> says the accepted syntax/format of data URI's are as follows:<br /><br /><pre class="prettyprint">data:[&lt;mediatype&gt;][;base64],&lt;data&gt;<br /></pre><br />So you'll need to define a media type (the Content-Type), declare that the data is base-64 encoded and provide an encoded payload.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">4 - Determining the Content-Type</span><br /><br />If you don't already know the Content-Type of the image you plan to base-64 encode, then you'll have to discover it. &nbsp;This isn't too hard, and involves writing a bit of code that checks the header of your image to determine its type.&nbsp; If you examine the specs of each image format you plan to support, you'll probably find that ...<br /><br />a. Every JPEG-image starts with a quick 2-byte "Start of Image" (SOI) marker:<br /><br /><pre class="prettyprint">0xFF D8</pre><br />b. Every PNG-image starts with a fixed 8-byte signature:<br /><br /><pre class="prettyprint">0x89 50 4E 47 0D 0A 1A 0A<br /></pre><br />c. Every GIF-image starts with a fixed 3-byte signature:<br /><br /><pre class="prettyprint">0x47 49 46<br /></pre><br />I'm not going to post any sample solutions here, because iterating over byte[] arrays and comparing values to determine an image format is trivial.&nbsp; However, in Java, it may help you to think of each image format as a value in an enumeration:<br /><br /><pre class="prettyprint">public enum ImageContentType {<br />  <br />  PNG("image/png", new byte[]{<br />    (byte)0x89, (byte)0x50, (byte)0x4E, (byte)0x47, <br />    (byte)0x0D, (byte)0x0A, (byte)0x1A, (byte)0x0A<br />    }),<br /><br />  JPEG("image/jpeg", new byte[]{<br />    (byte)0xFF, (byte)0xD8<br />    }),<br /><br />  GIF("image/gif", new byte[]{<br />    (byte)0x47, (byte)0x49, (byte)0x46<br />    });<br />  <br />  private String contentType_;<br />  private byte[] header_;<br />  <br />  private ImageContentType(String contentType, byte[] header) {<br />    contentType_ = contentType;<br />    header_ = header;<br />  }<br />  <br />  public byte[] getHeader() {<br />    return header_;<br />  }<br />  <br />  public String getContentType() {<br />    return contentType_;<br />  }<br />  <br />  @Override<br />  public String toString() {<br />    return getContentType();<br />  }<br />  <br />  public static final ImageContentType getContentType(final byte[] image) {<br />    ImageContentType ict = null;<br />    for(final ImageContentType type : ImageContentType.values()) {<br />      /* compare the header of image[] to type.getHeader() */<br />    }<br />    return ict;<br />  }<br />  <br />}</pre><br />Of course, you could always just check the file extension of the image you plan to encode, and determine a Content-Type based on that.&nbsp; In other words, if the resource ends in <b>.jpg</b> then you could assume, with reasonable certainty, that the Content-Type is <b>image/jpeg</b>.&nbsp; However, only relying on the file extension to tell you the format can be a dangerous strategy if you're not careful.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">5 - Encode and Assemble</span><br /><br />Now that you know the Content-Type, you can base-64 encode the image payload and assemble a valid "data" URI string.&nbsp; Don't bother writing a base-64 encoder from scratch, since there are many wonderful open-source implementations available for free.&nbsp; The most popular seems to be in the <a href="http://commons.apache.org/codec/">Apache Commons Codec library</a>.&nbsp; Specifically, take a peek at <a href="http://commons.apache.org/codec/api-release/org/apache/commons/codec/binary/Base64.html">org.apache.commons.codec.binary.Base64</a>.<br /><br />So, here's some pseudo code illustrating how to put it all together ...<br /><br /><pre class="prettyprint">import org.apache.commons.codec.binary.Base64;<br /><br />private static final String DATA_URI_SCHEME = "data:%s;base64,%s";<br /><br />/* ... */<br /><br />// Create a new Base64 object, setting the line length to zero<br />// so that the output is not chunked (e.g., no line breaks).<br />final Base64 b64 = new Base64(0);<br /><br />// Some image data, that you've fetched elsewhere.<br />final byte[] image = ...;<br /><br />// Discover the image format.<br />final ImageContentType ict = ImageContentType.getContentType(image);<br /><br />// Encode the image.<br />byte[] encoded = b64.encodeBase64(image);<br /><br />// Convert the encoded byte array into its String representation.<br />// Base-64 is just ASCII, to this is totally fine.<br />String imageEncoded = new String(encoded, "UTF-8");<br /><br />/* ... */<br /><br />// Build the data URI String for inclusion in our API response.<br />final String dataUri = String.format(DATA_URI_SCHEME,<br />  ict.toString(), imageEncoded);</pre><br />Now that you've assembled a valid data URI String, it's simple to attach it to a JSON object and return it to the caller.<br /><br />Have fun!]]>
    </content>
</entry>

<entry>
    <title>Remember to Close Your Streams When Using Java&apos;s Runtime.getRuntime().exec()</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2011/01/leaky-pipes-remember-to-close-your-streams-when-using-javas-runtimegetruntimeexec.html?rss" />
    <id>tag:mark.koli.ch,2011://1.249</id>

    <published>2011-01-05T22:25:00Z</published>
    <updated>2011-01-06T00:37:19Z</updated>

    <summary><![CDATA[For more than a year, I got away with forgetting to close my standard I/O streams when spawning a process in Java with Runtime.getRuntime().exec().&nbsp; On Linux, I was using exec() to spawn the df command to check my file system...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="bug" label="bug" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[For more than a year, I got away with forgetting to close my standard I/O streams when spawning a process in Java with <a href="http://download.oracle.com/javase/6/docs/api/java/lang/Runtime.html#exec%28java.lang.String%29">Runtime.getRuntime().exec()</a>.&nbsp; On Linux, I was using exec() to spawn the <a href="http://en.wikipedia.org/wiki/Df_%28Unix%29">df command</a> to check my file system disk space usage.&nbsp; Standard out from <b>df</b> was piped into the parent (Java) where I parsed the output to see if any partitions were getting full.&nbsp; Simple enough, right?<br /><br />In December 2010, I began experimenting with Java's next generation garbage collection engine, aptly named <a href="http://blogs.sun.com/theplanetarium/entry/java_vm_trying_a_new">G1</a> (a.k.a., Garbage First).&nbsp; Assuming you have Java 6 Update 14 or later you can enable the next-generation G1 garbage collector (still experimental as of Jan 2011) using the following JVM options:<br /><br /><pre class="prettyprint">-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC</pre><br />This post isn't about G1, so I'm not going to dig into the nitty-gritty on garbage collection.&nbsp; However, I discovered that G1 isn't as aggressive as the current Java garbage collector with regards to cleaning up streams.&nbsp; Bug in G1?&nbsp; Maybe, maybe not.&nbsp; Regardless, my app ran for a week or two with G1 enabled then I started to see all sorts of silly java.net.SocketException's claiming I had "Too many open files".&nbsp; Using the trusty <a href="http://en.wikipedia.org/wiki/Lsof">lsof command</a>, I saw that my Java process had left open a ton of stranded pipes.&nbsp; Definitely an indication of a leak somewhere ...<br /><br /><pre class="prettyprint">#/&gt; lsof -p 23064 | grep pipe<br />...<br />java    23064 mark  996w  FIFO    0,7    152581 pipe<br />java    23064 mark  997r  FIFO    0,7    152309 pipe<br />java    23064 mark  998r  FIFO    0,7    152448 pipe<br />java    23064 mark  999w  FIFO    0,7    152720 pipe<br />java    23064 mark 1000w  FIFO    0,7    152859 pipe<br />java    23064 mark 1001r  FIFO    0,7    152583 pipe<br />java    23064 mark 1002w  FIFO    0,7    153134 pipe<br />java    23064 mark 1003r  FIFO    0,7    152722 pipe<br />java    23064 mark 1004w  FIFO    0,7    154801 pipe<br />java    23064 mark 1005r  FIFO    0,7    152861 pipe<br />java    23064 mark 1006w  FIFO    0,7    152997 pipe<br />java    23064 mark 1007w  FIFO    0,7    153564 pipe<br />java    23064 mark 1008r  FIFO    0,7    152999 pipe<br />java    23064 mark 1009r  FIFO    0,7    153136 pipe<br />java    23064 mark 1010r  FIFO    0,7    153278 pipe<br />java    23064 mark 1011w  FIFO    0,7    153406 pipe<br />java    23064 mark 1012w  FIFO    0,7    153713 pipe<br />...</pre><br />With a little persistence, I crawled through my code looking for any obvious problem spots -- places where I forgot to close a stream -- and discovered that my calls to exec() were problematic.&nbsp; Calling exec() returns a Process object for the child where all standard I/O ops are redirected to the parent through three streams: STDOUT, STDIN, STDERR.&nbsp; It turns out you have to explicitly close these streams when you're done with the child otherwise they are left open!&nbsp; And, as you can see in the <b>lsof</b> output above, I was not closing these streams causing a nasty leak which eventually brought down my application.<br /><br />Going back to differences in the garbage collectors, it seems that the current default garbage collector cleaned up after my 
mess (closed the streams for me), but G1 did not.&nbsp; Hence why I never saw the "Too many open files" exception until I enabled G1.<br /><br />That said, the <b>undocumented proper</b> way of handing a Process object and its corresponding I/O streams is to wrap the exec() call in a try-finally block, closing the STDOUT, STDIN, and STDERR streams when you're done with the Process object.&nbsp; The abstract class java.lang.Process exposes these three streams to you via <a href="http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getOutputStream%28%29">getOutputStream()</a>, <a href="http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getInputStream%28%29">getInputStream()</a> and <a href="http://download.oracle.com/javase/6/docs/api/java/lang/Process.html#getErrorStream%28%29">getErrorStream()</a> which you must explicitly close.<br /><br />Here's the pseudo code:<br /><br /><pre class="prettyprint">import static org.apache.commons.io.IOUtils.closeQuietly;<br /><br />Process p = null;<br />try {<br />  p = Runtime.getRuntime().exec(...);<br />  // Do something with p.<br />} finally {<br />  if(p != null) {<br />    closeQuietly(p.getOutputStream());<br />    closeQuietly(p.getInputStream());<br />    closeQuietly(p.getErrorStream());<br />  }<br />}<br /></pre><br />Note that <a href="http://commons.apache.org/io/api-1.2/org/apache/commons/io/IOUtils.html">closeQuietly()</a> is part of the Apache Commons IOUtils library -- it's a helper method to close a stream ignoring nulls and exceptions.&nbsp; With this change in place, I redeployed my app and sure enough the problem was resolved.<br /><br />Lesson learned: regardless of what garbage collector you're using, it's always a good idea to explicitly close the STDOUT, STDIN, and STDERR streams associated with a Process object when you are done with it.<br /><br />Enjoy.]]>
        
    </content>
</entry>

<entry>
    <title>Reduce Storage Costs in the Cloud: GZIP Compress Binary Streams with Java</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/12/reduce-storage-costs-in-the-cloud-gzip-compress-your-binary-streams-in-java.html?rss" />
    <id>tag:mark.koli.ch,2010://1.248</id>

    <published>2010-12-23T19:00:00Z</published>
    <updated>2010-12-23T19:04:17Z</updated>

    <summary><![CDATA[Let me start by saying that most cloud storage solutions are relatively cheap to begin with, so compressing entities or streams stored in a database table or an elastic cloud store may not save you all that much.&nbsp; Purely in...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="cloud" label="cloud" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="compression" label="compression" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="gzip" label="gzip" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[Let me start by saying that most cloud storage solutions are relatively cheap to begin with, so compressing entities or streams stored in a database table or an elastic cloud store <b>may not </b>save you all that much.&nbsp; Purely in terms of cloud storage costs, you might save pennies or at most dollars if you compress optimally.&nbsp; In this case, optimally meaning you know your payloads will GZIP compress nicely and it makes sense to do so.&nbsp; In fact, if you forcefully compress entities unnecessarily you may actually increase their size!<br /><br />Let's say you have a "bucket" in which you plan to store hundreds or even thousands of cached HTML documents.&nbsp; Common sense might tell you that HTML compresses well.&nbsp; Your tiger like Computer Science instincts were right: HTML, generally speaking, does compress well and is a great candidate for compression.&nbsp; Obviously, less bytes stored in the cloud usually means slightly reduced storage costs.<br /><br />In Java, most applications represent HTML as a <a href="http://download.oracle.com/javase/6/docs/api/java/lang/String.html">String</a> literal which is really just a sequence of characters.&nbsp; Or to look at it another way, HTML can be thought of an array of bytes with a known character encoding.&nbsp; Thinking in terms of bytes, it's quite easy to GZIP compress and uncompress byte[] arrays on the fly in Java.&nbsp; Meet <a href="http://download.oracle.com/javase/6/docs/api/java/util/zip/GZIPInputStream.html">GZIPInputStream</a> and <a href="http://download.oracle.com/javase/6/docs/api/java/util/zip/GZIPOutputStream.html">GZIPOutputStream</a>.<br /><br />GZIP <b>compress</b> an <a href="http://download.oracle.com/javase/6/docs/api/java/io/InputStream.html">InputStream</a> and return the result as a new byte[] array:<br /><br /><pre class="prettyprint">public static final byte[] compress(final InputStream is)<br />  throws IOException {<br />  GZIPOutputStream gzos = null;<br />  try {<br />    ByteArrayOutputStream baos = new ByteArrayOutputStream();<br />    gzos = new GZIPOutputStream(baos);<br />    copy(is, gzos);<br />    gzos.finish(); // Important!<br />    return baos.toByteArray();<br />  } finally {<br />    closeQuietly(gzos);<br />  }<br />}</pre><br />GZIP <b>uncompress</b> an <a href="http://download.oracle.com/javase/6/docs/api/java/io/InputStream.html">InputStream</a> and return the result as a new byte[] array:<br /><br /><pre class="prettyprint">public static final byte[] uncompress(final InputStream is)<br />  throws IOException {<br />  GZIPInputStream gzis = null;<br />  try {<br />    ByteArrayOutputStream baos = new ByteArrayOutputStream();    <br />    gzis = new GZIPInputStream(is);<br />    copy(gzis, baos);<br />    return baos.toByteArray();<br />  } finally {<br />    closeQuietly(gzis);<br />  }<br />}</pre><br />First, note that I'm using <a href="http://download.oracle.com/javase/6/docs/api/java/io/ByteArrayOutputStream.html">ByteArrayOutputStream</a>'s to store the resulting compressed or uncompressed byte[] array <b>in memory</b>.&nbsp; Naturally, this means that this solution may not be ideal for you depending on your application.&nbsp; If you're planning to compress gigs of data in memory, that could be a bad idea unless you really know what you're doing.&nbsp; Proper usage of this really depends on your application and your intentions.<br /><br />Second, the copy() and closeQuietly() pseudo methods above are implemented for you <a href="http://mark.koli.ch/2010/12/23/GzipCompressor.java">here in my GzipCompressor utility class</a>.<br /><br />Looping back to our cached HTML example, let's compress a tiny HTML document represented here as a String:<br /><br /><pre class="prettyprint">// Note that this HTML is tiny, and probably won't compress well at all.<br />// In fact, the "compressed" result may actually be larger in size than<br />// the uncompressed original String.  This is just an example however,<br />// to show you how to compress a String literal.<br />final String html = "&lt;html&gt;&lt;body&gt;&lt;h1&gt;Horrible HTML&lt;/h1&gt;&lt;/body&gt;&lt;/html&gt;";<br /><br />// Get the UTF-8 encoded bytes from the input String.  I'm assuming<br />// that my HTML document is UTF-8 encoded.<br />final byte[] uncompressed = html.getBytes(UTF_8);<br />System.out.println("Uncompressed: " + uncompressed.length + "-bytes.");<br /><br />// Compress and report the result.<br />final byte[] compressed = compress(uncompressed);<br />System.out.println("Compressed: " + compressed.length + "-bytes.");<br /></pre><br />Now that you have a compressed byte[] array, it should be trivial to store it in the cloud using your favorite cloud storage engine or database.&nbsp; Ideally, you'll want to tweak your entities so that PUT's and GET's automatically compress and uncompress these entities on the fly for you. <br /><br />My <a href="http://mark.koli.ch/2010/12/23/GzipCompressor.java">full GzipCompressor utility class can be found here</a>.<br /><br />Enjoy.<br />]]>
        
    </content>
</entry>

<entry>
    <title>Set the Cache-Control and Expires Headers on a Redirect with mod_rewrite</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/12/set-cache-control-and-expires-headers-on-a-redirect-with-mod-rewrite.html?rss" />
    <id>tag:mark.koli.ch,2010://1.247</id>

    <published>2010-12-11T20:25:00Z</published>
    <updated>2010-12-11T20:29:33Z</updated>

    <summary><![CDATA[In many web-service infrastructures, it's often desirable to disable the caching of redirects.&nbsp; Specifically, you might want to set the Expires or Cache-Control headers so that your 301 or 302 redirects from Apache's mod_rewrite are never cached upstream.&nbsp; Off the...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="apache" label="apache" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="http" label="http" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="mod_rewrite" label="mod_rewrite" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[In many web-service infrastructures, it's often desirable to disable the caching of redirects.&nbsp; Specifically, you might want to set the <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21">Expires</a> or <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9">Cache-Control</a> headers so that your 301 or 302 redirects from Apache's <a href="http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html">mod_rewrite</a> are never cached upstream.&nbsp; Off the top of my head, I can think of a number of reasons why you might want to prevent the caching of a redirect:<br /><br /><ul><li>Your redirect may change from one request, to the next.&nbsp; Disable caching so the client (the browser) isn't redirected to the same destination every time.<br /><br /></li><li>Your web-application is behind a reverse caching proxy, and you don't want the caching proxy to cache the redirect.<br /><br /></li><li>In development, you're sitting behind a corporate web-proxy that is notorious for caching content when it really shouldn't.&nbsp; Disable caching on the redirects so you can verify that your web-application is working as expected during testing (assuming the web-proxy obeys your Cache-Control and Expires headers).<br /><br /></li><li>Your web-application counts how many times someone is redirected.&nbsp; Disable caching so your click-through statistics are a bit more accurate.</li></ul><br />Surprisingly, this seemingly common need isn't well documented in the official Apache docs.&nbsp; So, here's how to do it.<br /><br />In this example, I'm redirecting based on the Host.&nbsp; If the incoming request does not match the Host I require, mod_rewrite triggers a 301 redirect to the correct Host.&nbsp; Of course, your <a href="http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewritecond">RewriteCond</a>'s might be different.<br /><br /><pre class="prettyprint">RewriteCond %{HTTP_HOST} !^mark\.koli\.ch [NC]<br />RewriteRule ^/(.*)$ http://mark.koli.ch/$1 [R=301,L,E=nocache:1]<br /><br />## Set the response header if the "nocache" environment variable is set<br />## in the RewriteRule above.<br />Header always set Cache-Control "no-store, no-cache, must-revalidate" env=nocache<br /><br />## Set Expires too ...<br />Header always set Expires "Thu, 01 Jan 1970 00:00:00 GMT" env=nocache<br /></pre><br />In this example, when the RewriteRule is fired the "nocache" environment variable is set.&nbsp; Note the <b>E=nocache:1</b> <a href="http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewriteflags">rewrite flag</a> in the RewriteRule.&nbsp; Subsequently, mod_headers will set the Cache-Control and Expires headers only if this "nocache" environment variable is set.&nbsp; In other words, "nocache" is only set on a 301 redirect from the RewriteRule.<br /><br />This works nicely.<br /><br /><pre class="prettyprint">GET /wombat HTTP/1.1<br />Host: koli.ch<br /><br />HTTP/1.1 301 Moved Permanently<br />Date: Sat, 11 Dec 2010 19:36:09 GMT<br />Location: http://mark.koli.ch/wombat<br />Server: Apache<br />Cache-Control: no-store, no-cache, must-revalidate<br />Expires: Thu, 01 Jan 1970 00:00:00 GMT<br />Content-Length: 230<br />Content-Type: text/html; charset=iso-8859-1<br />Connection: close<br /></pre><br />Yay for HTTP.]]>
        
    </content>
</entry>

<entry>
    <title>Tri-Monitor Configuration With an HP Z600 Workstation and Dual Nvidia Quadro FX 1800&apos;s</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/12/tri-monitor-configuration-with-an-hp-z600-workstation-and-dual-nvidia-quadro-fx-1800s.html?rss" />
    <id>tag:mark.koli.ch,2010://1.246</id>

    <published>2010-12-03T22:50:00Z</published>
    <updated>2010-12-03T22:52:13Z</updated>

    <summary><![CDATA[For months, I've had a spare 20" HP LCD-2065 display sitting under my desk at the office.&nbsp; With a few extra cycles on my hands, I decided to take half-a-day and setup a truly bad ass developers workstation: three, 20-inch...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="hp" label="hp" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="linux" label="linux" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="nvidia" label="nvidia" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="ubuntu" label="ubuntu" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[For months, I've had a spare 20" HP LCD-2065 display sitting under my desk at the office.&nbsp; With a few extra cycles on my hands, I decided to take half-a-day and setup a truly bad ass developers workstation: three, 20-inch monitors, <a href="http://en.wikipedia.org/wiki/Xinerama">Xinerama'ed</a> to produce a single 4800x1200 pixel desktop (each display driving 1600x1200 @ 60 Hz).&nbsp; And, best of all, the HP Z600 Workstation powering this monster is running 64-bit 10.04 Ubuntu Linux.<br /><br /><span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://mark.koli.ch/2010/12/02/ubuntu-hp-z600-nvidia-fx-1800.jpg"><img alt="ubuntu-hp-z600-nvidia-fx-1800.jpg" src="http://mark.koli.ch/assets_c/2010/12/ubuntu-hp-z600-nvidia-fx-1800-thumb-400x248.jpg" class="mt-image-none" style="" height="248" width="400" /></a></span><br /><br />Not bad, eh?<br /><br />Here's how I did it ...]]>
        <![CDATA[My relatively bare bones HP Z600 Workstation shipped with a single mid-range <a href="http://www.nvidia.com/object/product_quadro_fx_1800_us.html">Nvidia Quadro FX 1800 PCIe card</a>.&nbsp; This really isn't a bad graphics card, but if you have one of these and you expect to drive three displays with a single FX 1800, you should stop right there.&nbsp; Turns out, even though the Quadro FX 1800 offers three display outs (2 <a href="http://en.wikipedia.org/wiki/DisplayPort">displayPorts</a> and 1 DVI) the card will not let you use all three outs at once.&nbsp; At most, in any configuration, you'll be able to drive only two displays with the FX 1800: either using both displayPorts, or one displayPort and the DVI out.<br /><br />So what if you want to drive 3-displays on a Z600?&nbsp; Well, you have two choices: get another Quadro FX 1800 and insert it into the 2nd PCIe x16 slot, or find another graphics card that's supported by the latest Nvidia driver.&nbsp; Lucky for me, I was able to get my hands on an extra Quadro FX 1800, popped it in, and bingo.&nbsp; Tri-monitor, biggest desktop I've ever used in my life, perfection.&nbsp; Not to mention the latest Nvidia drivers (<a href="http://www.nvidia.com/object/linux-display-amd64-260.19.21-driver.html">260.19.21</a> as of 12/2/10) work flawlessly with the FX 1800 on 64-bit Ubuntu 10.04.<br /><br />I will note, however, that before I snagged an extra FX 1800, I tried pairing a single FX 1800 with an older Nvidia GeForce 5200 PCI.&nbsp; Configuring 3-monitors with an FX 1800 (only supported by the latest Nvidia driver) and the GeForce 5200 (supported by the legacy GPU Nvidia driver) failed miserably.&nbsp; It turns out, you really can't mix-and-match new and legacy hardware that use different driver versions (well, maybe you can, but I tried for an hour and gave up in disgust).&nbsp; In short, after my experiments here, it feels like you'll have better luck using two identical cards, instead of trying to mix-and-match random graphics hardware.<br /><br />Good luck, and happy hacking!]]>
    </content>
</entry>

<entry>
    <title>JSON, XML, and stray Ampersands with Amazon&apos;s SQS (Simple Queue Service)</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/10/json-xml-and-ampersands-with-amazons-sqs-simple-queuing-service.html?rss" />
    <id>tag:mark.koli.ch,2010://1.244</id>

    <published>2010-10-26T05:17:00Z</published>
    <updated>2010-10-26T05:15:08Z</updated>

    <summary><![CDATA[Amazon's SQS (Simple Queue Service) uses XML formatted payloads to push and pop messages to and from an SQS queue.&nbsp; In other words, the underlying request and response bodies are XML payloads, containing among other things, a message ID and...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="amazon" label="amazon" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="aws" label="aws" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="json" label="json" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="sqs" label="sqs" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="xml" label="xml" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[Amazon's <a href="http://aws.amazon.com/sqs/">SQS (Simple Queue Service)</a> uses <a href="http://docs.amazonwebservices.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/UnderstandingResponses.html">XML formatted payloads</a> to push and pop messages to and from an SQS queue.&nbsp; In other words, the underlying request and response bodies are XML payloads, containing among other things, a message ID and a message receipt handle.&nbsp; This means that the message body itself (the thing you're pushing onto the queue) ultimately has to be properly XML escaped to work right.&nbsp; Well it turns out, Amazon's own <a href="http://aws.amazon.com/sdkforjava">AWS library</a> is very flaky here, and does not do the right thing when unmarshalling an XML SQS response back into an Object.&nbsp; Essentially, it gets confused when it encounters an XML escaped ampersand in the message body.&nbsp; For example, this popped message (in pseudo XML) fails miserably to unmarshall:<br /><br /><pre class="prettyprint">&lt;sqs&gt;&lt;message&gt;{"music":"rock &amp;amp; roll"}&lt;/message&gt;&lt;/sqs&gt;<br /></pre><br />Note that the ampersand in the message body is properly XML escaped, however, Amazon's own AWS library returns this as the message body once unmarshalled:<br /><br /><pre class="prettyprint">{"music":"rock &amp;</pre><br />In other words, the message body ends abruptly at the ampersand, and needless to say, any reasonable JSON library will fail to parse this malformed block of nonsense.<br /><br /><b>Solution</b>: if you are going to use SQS, and there's a possibility the messages you're going to push onto a queue will have issues with XML escaping, it appears you should always base-64 encode the message body, then base-64 decode it when popping.&nbsp; Fortunately, the <a href="http://commons.apache.org/codec/apidocs/org/apache/commons/codec/binary/Base64.html">Apache Commons Codec library has a wonderful Base64 class</a> to handle this encoding and decoding mess for you.<br /><br />Bottom line, if you're encountering encoding issues with Amazon's SQS, better check if your messages are able to make it through the XML marshalling and unmarshalling process.&nbsp; If not, you'll need to base-64 encode and decode your messages to dance around bugs in Amazon's AWS library.<br /><br />Good luck, and hope this helps.<br />]]>
        
    </content>
</entry>

<entry>
    <title>Better Mobile Device Detection with a Spring 3 Interceptor</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/10/spring-3-mobile-device-detection-with-an-interceptor.html?rss" />
    <id>tag:mark.koli.ch,2010://1.243</id>

    <published>2010-10-19T04:05:00Z</published>
    <updated>2010-10-27T02:08:46Z</updated>

    <summary><![CDATA[The other day on some Java/JSP tutorial web-site I saw the worst example ever for detecting and properly rendering a mobile capable version of a web-site.&nbsp; Yes, I'm pointing at you Roseindia.&nbsp; In every JSP of this example, their mobile...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="howto" label="howto" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="spring" label="spring" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[The other day on some Java/JSP tutorial web-site I saw the worst example ever for detecting and properly rendering a mobile capable version of a web-site.&nbsp; Yes, I'm pointing at you <a href="http://www.roseindia.net/">Roseindia</a>.&nbsp; In every JSP of this example, their mobile User-Agent detection involved one big if/else block:<br /><br /><pre class="prettyprint">&lt;%<br /><br />String userAgent = request.getHeader("User-Agent");<br />if(userAgent.contains("iPhone")) {<br />  %&gt; Mobile site! &lt;%<br />} else {<br />  %&gt; Regular site! &lt;%<br />}<br /><br />%&gt;<br /></pre><br />This wins the worst coding example of the year award.&nbsp; Here's why this is terrible and you should never use this example:<br /><br /><ol><li>Not all requests contain a User-Agent header.&nbsp; In fact, the User-Agent header is purely optional.&nbsp; In the code above, if the request does not contain a User-Agent you'll see a nice <a href="http://download.oracle.com/javase/6/docs/api/java/lang/NullPointerException.html">NullPointerException</a> thrown at userAgent.contains() given that the userAgent is null.<br /><br /></li><li>Not every mobile device is an iPhone.&nbsp; What about Blackberry, Andriod or Palm clients?&nbsp; Blindly assuming that every mobile user is on an iPhone, or other similar device, is horrendously ignorant.<br /><br /></li><li>Many great frameworks exist, like Spring 3 MVC, that allow you to separate your web-application business and display logic.&nbsp; This in mind, combing both into a single JSP is a bad idea for a number of reasons.&nbsp; In an ideal world, your mobile device detection would occur in an interceptor that triggers your MVC framework to render one view for mobile devices, and another view for all others.</li></ol><br />This is a fairly common requirement: users visiting your site in a standard web-browser see one view, and users on a mobile device (like a Palm) see a "mobile version" of the same view.&nbsp; So, here's a way to implement better mobile device detection in your web-application using a Spring 3 <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/portlet/handler/HandlerInterceptorAdapter.html">HandlerInterceptorAdapter</a> ...]]>
        <![CDATA[Note that I assume you are familiar with Spring 3 MVC, and have a working Spring 3 application already up and running.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">1 - Configure Spring</span><br /><br />First, you'll need to make the necessary adjustments to your Spring MVC configuration which usually involves tweaking your <b>mvc.xml</b> configuration file.&nbsp; Regardless of where your MVC XML configuration is, you'll be defining an MVC interceptor for your application like so ...<br /><br /><pre class="prettyprint">&lt;mvc:interceptors&gt;<br />  <br />  &lt;mvc:interceptor&gt;<br />    &lt;mvc:mapping path="/somepath**" /&gt;<br />&nbsp;&nbsp;&nbsp; &lt;mvc:mapping path="/anotherpath**" /&gt;<br />&nbsp;&nbsp;&nbsp; &lt;bean class="com.kolich.spring.interceptors.MobileInterceptor"<br />        init-method="init"&gt;<br />        &lt;property name="mobileUserAgents"&gt;<br />          &lt;list value-type="java.lang.String"&gt;<br />            &lt;value&gt;.*(webos|palm|treo).*&lt;/value&gt;<br />            &lt;value&gt;.*(andriod).*&lt;/value&gt;<br />            &lt;value&gt;.*(kindle|pocket|o2|vodaphone|wap|midp|psp).*&lt;/value&gt;<br />            &lt;value&gt;.*(iphone|ipod).*&lt;/value&gt;<br />            &lt;value&gt;.*(blackberry|opera mini).*&lt;/value&gt;<br />          &lt;/list&gt;<br />        &lt;/property&gt;<br />    &lt;/bean&gt;<br />  &lt;/mvc:interceptor&gt;<br /><br />&lt;/mvc:interceptors&gt;</pre><br />This bean, which I will discuss next, extends Spring's abstract <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/servlet/handler/HandlerInterceptorAdapter.html">HandlerInterceptorAdapter</a>.&nbsp; We will build this interceptor so that it's called right before a Spring view is rendered, giving the interceptor a chance to modify the final view name as necessary.&nbsp; Also, note that this bean defines a List of regular expressions that the interceptor will use to determine if the client is a mobile device.&nbsp; You can add or remove regular expressions to this List depending on which mobile devices (a.k.a., User-Agent's) you plan to support.<br /><br />If you are not familiar with Spring interceptors, you might like to <a href="http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/htmlsingle/spring-framework-reference.html#mvc-handlermapping-interceptor">read up on them here</a>.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">2 - The MobileInterceptor</span><br /><br />Without further ado, here's my MobileInterceptor ...<br /><br /><pre class="prettyprint">/**
 * Copyright (c) 2010 Mark S. Kolich
 * http://mark.koli.ch
 *
 * Permission is hereby granted, free of charge, to any person
 * obtaining a copy of this software and associated documentation
 * files (the "Software"), to deal in the Software without
 * restriction, including without limitation the rights to use,
 * copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following
 * conditions:
 *
 * The above copyright notice and this permission notice shall be
 * included in all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
 * OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
 * HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
 * WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 * OTHER DEALINGS IN THE SOFTWARE.
 */

package com.kolich.spring.interceptors;<br /><br />import java.util.ArrayList;<br />import java.util.List;<br />import java.util.regex.Matcher;<br />import java.util.regex.Pattern;<br />import java.util.regex.PatternSyntaxException;<br /><br />import javax.servlet.http.HttpServletRequest;<br />import javax.servlet.http.HttpServletResponse;<br /><br />import org.apache.log4j.Logger;<br />import org.springframework.web.servlet.ModelAndView;<br />import org.springframework.web.servlet.handler.HandlerInterceptorAdapter;<br /><br />public class MobileInterceptor extends HandlerInterceptorAdapter {<br />  <br />  /**<br />   * The name of the mobile view that the viewer is re-directed to<br />   * in the event that a mobile device is detected.<br />   */<br />  private static final String MOBILE_VIEWER_VIEW_NAME = "mobile";<br />  <br />  /**<br />   * The User-Agent Http header.<br />   */<br />  private static final String USER_AGENT_HEADER = "User-Agent";<br />  <br />  private List&lt;String&gt; mobileAgents_;<br />  private List&lt;Pattern&gt; uaPatterns_;<br />  <br />  public void init() {<br />    uaPatterns_ = new ArrayList&lt;Pattern&gt;();<br />    // Pre-compile the user-agent patterns as specified in mvc.xml<br />    for(final String ua : mobileAgents_) {<br />      try {<br />        uaPatterns_.add(Pattern.compile(ua, Pattern.CASE_INSENSITIVE));<br />      } catch (PatternSyntaxException e) {<br />        // Ignore the pattern, if it failed to compile<br />        // for whatever reason.<br />      }<br />    }<br />  }<br />  <br />  @Override<br />  public void postHandle(HttpServletRequest request,<br />    HttpServletResponse response, Object handler,<br />    ModelAndView mav) throws Exception {<br />    final String userAgent = request.getHeader(USER_AGENT_HEADER);<br />    if(userAgent != null) {<br />      // If the User-Agent matches a mobile device, then we set<br />      // the view name to the mobile view JSP so that a mobile<br />      // JSP is rendered instead of a normal view.<br />      if(isMobile(userAgent)) {<br />        final String view = mav.getViewName();<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // If the incoming view was "homepage" then this interceptor<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // changes the view name to "homepage-mobile" which, depending<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // on your Spring configuration would probably resolve to<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // "homepage-mobile.jsp" to render a mobile version of your<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // site.<br />        mav.setViewName(view + "-" + MOBILE_VIEWER_VIEW_NAME);<br />      }<br />    }<br />  }<br />  <br />  /**<br />   * Returns true of the given User-Agent string matches a suspected<br />   * mobile device.<br />   * @param agent<br />   * @return<br />   */<br />  private final boolean isMobile(final String agent) {<br />    boolean mobile = false;<br />    for(final Pattern p : uaPatterns_) {<br />      final Matcher m = p.matcher(agent);<br />      if(m.find()) {<br />        mobile = true;<br />        break;<br />      }<br />    }<br />    return mobile;<br />  }<br />  <br />  public void setMobileUserAgents(List&lt;String&gt; agents) {<br />    mobileAgents_ = agents;<br />  }<br />  <br />}</pre><br />As you probably noticed the real meat of this interceptor bean is inside of the postHandle() method, which examines the User-Agent HTTP request header (if any), checks if it's a mobile device, and if so slightly changes the resulting view name so that a mobile version of the view is rendered instead of the normal version.&nbsp; According to the Spring documentation, the postHandle() method is "called after HandlerAdapter actually invoked the handler, but before the DispatcherServlet renders the view."&nbsp; In our case, this is perfect.<br /><br />Inside of postHandle() my MobileInterceptor retrieves the resolved view name, then if the User-Agent matches that of a known mobile device, it changes the view name by appending "-mobile" to end of it.&nbsp; For example, say you have a view named "about" that is rendered by "about.jsp".&nbsp; This interceptor would change the resulting view name to "about-mobile" which would be rendered by "about-mobile.jsp" (assuming you are using a standard <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/servlet/view/InternalResourceViewResolver.html">InternalResourceViewResolver</a> to resolve view names to JSP's).&nbsp; In other words, this means you can put all of your mobile display logic into about-mobile.jsp, while about.jsp is left in tact for non-mobile clients; you keep your mobile and non-mobile display logic separate in two individual files.&nbsp; Of course, I don't have to tell you that keeping these separate will make your life as a developer a LOT easier in the long run.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">3 - Putting it all Together</span><br /><br />Putting everything together, the &lt;mvc:interceptor&gt; XML configuration tells Spring to call my interceptor bean whenever it encounters a specific path.&nbsp; In this case, I told Spring to watch for the paths /somepath and /anotherpath based on the &lt;mvc:mapping&gt;'s you see above in the XML.&nbsp; When Spring handles a request for /somepath or /anotherpath it will call the interceptor at the appropriate point in the chain based on the methods overridden by my bean.&nbsp; In this case, I've overridden the <b>postHandle()</b> method such that Spring will call my interceptor bean to do what it needs to do once the view has been resolved and it's ready to render up some content.&nbsp; Of course, you could also override <b>preHandle()</b> if you needed the interceptor to be called before a view is selected, and so on.&nbsp; Again, take a peek at <a href="http://static.springsource.org/spring/docs/3.0.x/javadoc-api/org/springframework/web/servlet/handler/HandlerInterceptorAdapter.html">HandlerInterceptorAdapter</a> for all of the gory details.<br /><br />Enjoy!]]>
    </content>
</entry>

<entry>
    <title>Understanding the HTTP Vary Header and Caching Proxies (Squid, etc.)</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/09/understanding-the-http-vary-header-and-caching-proxies-squid-etc.html?rss" />
    <id>tag:mark.koli.ch,2010://1.242</id>

    <published>2010-09-25T07:10:00Z</published>
    <updated>2010-09-25T07:25:56Z</updated>

    <summary><![CDATA[I never paid much attention to the HTTP Vary header.&nbsp; In fact, I've been fortunate enough to avoid it for this long and never really had to care much about it.&nbsp; Well, it turns out when you're configuring a high-performance...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="apache" label="apache" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="http" label="http" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="mod_deflate" label="mod_deflate" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="proxies" label="proxies" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="squid" label="squid" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[I never paid much attention to the <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.44">HTTP Vary header</a>.&nbsp; In fact, I've been fortunate enough to avoid it for this long and never really had to care much about it.&nbsp; Well, it turns out when you're configuring a high-performance <a href="http://en.wikipedia.org/wiki/Reverse_proxy">reverse proxy</a>, understanding the Vary header and what it means to your reverse proxy caching policies is absolutely crucial.<br /><br />Here's an interesting problem I recently solved that dealt with Squid, Apache, and that elusive Vary response header ...<br />]]>
        <![CDATA[<br /><span style="font-size: 1.1em; font-weight: bold;">1 - The Vary Basics</span><br /><br />Popular caching proxies, like <a href="http://www.squid-cache.org/">Squid</a>, usually generate a <a href="http://en.wikipedia.org/wiki/Hash_function">hash</a> of the request from a number of inputs including the URI and the <b>contents of the Vary response header</b>.&nbsp; When a caching proxy receives a request for a resource, it gathers these inputs, generates a hash, then checks its cache to see if it already has a resource sitting on disk, or in memory, that matches the computed hash.&nbsp; This is how Squid, and other caching proxies, fundamentally know if they have a cache HIT or MISS (e.g., can Squid return the content it has cached or does it need to revalidate the request against the destination server).<br /><br />That in mind, you can probably see how the Vary header is quite important when a caching proxy is looking for a cache HIT or MISS.&nbsp; The Vary header is a way for the web-server to tell any intermediaries (caching proxies) what they should use, if necessary, to figure out if the requested resource is fresh or stale.&nbsp; Sample Vary headers include:<br /><br /><pre class="prettyprint">Vary: Accept-Encoding<br />Vary: Accept-Encoding,User-Agent<br />Vary: X-Some-Custom-Header,Host<br />Vary: *<br /></pre><br />According to the HTTP spec, "the Vary field value indicates the set of request-header fields that fully determines, while the response is fresh, whether a cache is permitted to use the response to reply to a subsequent request without revalidation."&nbsp; Yep, that's pretty important (I discovered this the hard way).<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">2 - The Caching Problem</span><br /><br />I configured Squid to act as a round-robin load balancer and caching proxy, sitting in front of about four Apache web-servers.&nbsp; Each Apache web-server was running a copy of my web-application, which I intended to have Squid cache where possible.&nbsp; Certain requests, were for large JSON objects, and I explicitly configured Squid to cache requests ending in .json for 24-hours.<br /><br />I opened a web-browser and visited a URL I expected to be cached (should have already been in the cache from a previous request, notice the HIT) ...<br /><br /><pre class="prettyprint">GET /path/big.json HTTP/1.1<br />Host: app.kolich.local<br />User-Agent: Firefox<br /><br />HTTP/1.0 200 OK<br />Date: Fri, 24 Sep 2010 23:09:32 GMT<br />Content-Type: application/json;charset=UTF-8<br />Content-Language: en-US<br />Vary: Accept-Encoding,User-Agent<br />Age: 1235<br />X-Cache: HIT from cache.kolich.local<br />X-Cache-Lookup: HIT from cache.kolich.local:80<br />Content-Length: 25090<br />Connection: close<br /></pre><br />Ok, looks good!&nbsp; I opened a 2nd web-browser on a different machine (hint: with a different User-Agent) and tried again.&nbsp; This time, notice the X-Cache: MISS ...<br /><br /><pre class="prettyprint">GET /path/big.json HTTP/1.1<br />Host: app.kolich.local<br />User-Agent: Chrome<br /><br />HTTP/1.0 200 OK<br />Date: Fri, 24 Sep 2010 23:11:45 GMT<br />Content-Type: application/json;charset=UTF-8<br />Content-Language: en-US<br />Vary: Accept-Encoding,User-Agent<br />Age: 4<br />X-Cache: MISS from cache.kolich.local<br />X-Cache-Lookup: MISS from cache.kolich.local:80<br />Content-Length: 25090<br />Connection: close<br /></pre><br />Wow, look at that.&nbsp; I requested exactly the same resource, just from a different browser, and I saw a cache MISS.&nbsp; This is obviously not what I want, I need the same cached resource to be served up from the cache regardless of who's making the request.&nbsp; If left alone, this is only caching a response per User-Agent, not globally per resource.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">3 - Solution: Check Your Vary Headers</span><br /><br />Remember how I said the contents of the Vary header are important for caching proxies?<br /><br />In both requests above, note the User-Agent request headers and the contents of the Vary response headers.&nbsp; Although each request was for exactly the same resource, Squid determined that they were very different as far as its cache was concerned.&nbsp; How did this happen?&nbsp; Well, take a peek at a Vary response header:<br /><br /><pre class="prettyprint">Vary: Accept-Encoding,User-Agent<br /></pre><br />This tells Squid that the request URI, the Accept-Encoding request header, and the User-Agent request header should be included in a hash when determining if an object is available in its cache, or not.&nbsp; Obviously, any reasonable hash of (URI, Accept-Encoding, "Firefox") should not match the hash of (URI, Accept-Encoding, "Chrome").&nbsp; Hence why Squid seemed to think the request was for different objects!<br /><br />To fix this, I located the source of the annoying "User-Agent" addition to my Vary response header, which happened to come from <a href="http://mark.koli.ch/2009/04/howto-use-apache-mod-deflate-to-compress-web-content-obsessed-with-speed-of-kolichcommobi.html">Apache's very own mod_deflate module</a>.&nbsp; The recommended mod_deflate configuration involves appending "User-Agent" to the Vary response header on any response that is not compressed by mod_deflate.&nbsp; I don't really see why this is necessary, but the Apache folks seemed to think this was important.&nbsp; Here's the relevant lines from the Apache suggested mod_deflate configuration:<br /><br /><pre class="prettyprint">SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|ico)$ no-gzip dont-vary<br />Header append Vary User-Agent env=!dont-vary<br /></pre><br />In any event, I removed the 2nd line above, restarted Apache and Squid began caching beautifully regardless of which client issued the request.&nbsp; Essentially, I told Squid to stop caring about the User-Agent by removing "User-Agent" from my Vary response header, and problem solved!<br /><br />The joys of HTTP.<br /><br />Cheers.<br />]]>
    </content>
</entry>

<entry>
    <title>Recursively Deleting Large Amazon S3 Buckets</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/09/recursively-deleting-large-amazon-s3-buckets.html?rss" />
    <id>tag:mark.koli.ch,2010://1.241</id>

    <published>2010-09-17T15:54:00Z</published>
    <updated>2010-09-17T15:59:20Z</updated>

    <summary><![CDATA[My first experience using Amazon Web Services for a production quality project was quite fun, and deeply interesting.&nbsp; I've played with AWS a bit on my own time, but I recently had a chance to really sink my teeth into...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="amazon" label="amazon" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="aws" label="aws" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[My first experience using <a href="http://aws.amazon.com/">Amazon Web Services</a> for a production quality project was quite fun, and deeply interesting.&nbsp; I've played with AWS a bit on my own time, but I recently had a chance to really sink my teeth into it and implement production level code that uses AWS as a real platform for an upcoming web, and mobile application.<br /><br />Perhaps the most interesting, and frustrating, part of this project involved storing hundreds of thousands of objects in an <a href="http://aws.amazon.com/s3/">AWS S3</a> bucket.&nbsp; If you're not familiar with <a href="http://aws.amazon.com/s3/">S3</a>, it's the AWS equivalent to an online storage web-service.&nbsp; The concept is simple: you create an S3 "bucket" then shove "objects" into the bucket, creating folders where necessary.&nbsp; Of course, you can also update and delete objects.&nbsp; If it helps, think of S3 as a pseudo online file-system that's theoretically capable of storing an unlimited amount of data.&nbsp; Yes, I'm talking <a href="http://en.wikipedia.org/wiki/Exabyte">Exabytes</a> of data ... theoretically ... if you're willing to pay Amazon for that much storage.<br /><br />In any event, I created a new S3 bucket and eventually placed hundreds of thousands of objects into it.&nbsp; S3 handled this with ease.&nbsp; The problem, however, was when it came time to delete this bucket and all objects inside of it.&nbsp; Turns out, there is no native <a href="http://docs.amazonwebservices.com/AmazonS3/latest/API/">S3 API</a> call that recursively deletes an S3 bucket, or renames it for that matter.&nbsp; I guess Amazon leaves it up to the developer to implement such functionality?<br /><br />That said, if you need to recursively delete a very large S3 bucket, you really have 2 options: use a tool like <a href="http://code.google.com/p/s3funnel/">s3funnel</a> or write your own tool that efficiently deletes multiple objects <b>concurrently</b>.&nbsp; Note that I say <b>concurrently</b>, otherwise you'll waste a lot of time sitting around waiting for a single-threaded delete to remove objects one at a time, which is horribly inefficient.&nbsp; Well this sounds like a perfect problem for a thread pool and wouldn't you guess it, even a <a href="http://mark.koli.ch/2010/04/understanding-javas-countdownlatch.html">CountDownLatch</a>!<br /><br />Continue reading for the code ...]]>
        <![CDATA[The idea here is you'll want to spawn multiple threads from a controlled thread pool where each thread is responsible for deleting a single object.&nbsp; This way, you can delete 20, 30, 100 objects at a time.&nbsp; Yay for threads!<br /><br />Here's the <b>pseudo</b> code.&nbsp; Note that I say <b>pseudo</b> code because it's not a complete implementation.&nbsp; This examples assumes you have an AWS S3 implementation (a library) that's able to list objects in a bucket, delete buckets, and delete objects.<br /><br /><pre class="prettyprint">package com.kolich.aws.s3.util;<br /><br />import java.io.UnsupportedEncodingException;<br />import java.net.URLEncoder;<br />import java.util.List;<br />import java.util.concurrent.CountDownLatch;<br />import java.util.concurrent.ExecutorService;<br />import java.util.concurrent.Executors;<br /><br />import com.amazonaws.services.s3.model.S3ObjectSummary;<br /><br />public class RecursiveS3BucketDelete {<br /><br />  private static final String AWS_ACCESS_KEY_PROPERTY = "aws.key";<br />  private static final String AWS_SECRET_PROPERTY = "aws.secret";<br />  <br />  /**<br />   * The -Daws.key and -Daws.secret system properties should<br />   * be set like this:<br />   * -Daws.key=AK7895IH1234X2GW12IQ<br />&nbsp;&nbsp; * -Daws.secret=1234567890123456789012345678901234456789<br />   */<br />  <br />&nbsp; // Set up a new thread pool to delete 20 objects at a time.<br />  private static final ExecutorService pool__ =<br />        Executors.newFixedThreadPool(20);<br />  <br />  public static void main(String[] args) {<br />    <br />    final String accessKey = System.getProperty(AWS_ACCESS_KEY_PROPERTY);<br />    final String secret = System.getProperty(AWS_SECRET_PROPERTY);<br />    if(accessKey == null || secret == null) {<br />      throw new IllegalArgumentException("You're missing the " +<br />          "-Daws.key and -Daws.secret required VM properties.");<br />    }<br />    <br />    final String bucketName;<br />    if(args.length &lt; 1) {<br />      throw new IllegalArgumentException("Missing required " +<br />          "program argument: bucket name.");<br />    }<br />    bucketName = args[0];<br />    <br />    // ... setup your S3 client here.<br />    <br />    List&lt;S3ObjectSummary&gt; objects = null;<br />    do {<br />      objects = s3.listObjects(bucketName).getObjectSummaries();<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // Create a new CountDownLatch with a size of how many objects<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // we fetched.&nbsp; Each worker thread will decrement the latch on<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // completion; the parent waits until all workers are finished<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // before starting a new batch of delete worker threads.<br />      final CountDownLatch latch = new CountDownLatch(objects.size());<br />      for(final S3ObjectSummary object : objects) {<br />        pool__.execute(new Runnable() {<br />          @Override<br />          public void run() {<br />            try {<br />              s3.deleteObject(bucketName,<br />                URLEncoder.encode(object.getKey(), "UTF-8"));<br />            } catch (Exception e) {<br />              System.err.println("&gt;&gt;&gt;&gt; FAILED to delete object: (" +<br />                bucketName + ", " + object.getKey()+ ")");<br />            } finally {<br />              latch.countDown();<br />            }<br />          }<br />        });<br />      }<br />      // Wait here until the current set of threads<br />      // are done processing.&nbsp; This prevents us from shoving too<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // many threads into the thread pool; it's a little more<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // controlled this way.<br />      try {<br />        System.out.println("Waiting for threads to finish ...");<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // This blocks the parent until all spawned children<br />&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; // have finished.<br />        latch.await();<br />      } catch (InterruptedException e) { }<br />    } while(objects != null &amp;&amp; !objects.isEmpty());<br />    <br />    pool__.shutdown();<br />    <br />    // Finally, delete the bucket itself.<br />    try {<br />      s3.deleteBucket(bucketName);<br />    } catch (Exception e) {<br />      System.err.println("Failed to ultimately delete bucket: " +<br />          bucketName);<br />    }<br /><br />  }<br /><br />}<br /></pre><br />Additional notes, and warnings:<br /><br /><ul><li>If you're not familiar with using a CountDownLatch, you can find my <a href="http://mark.koli.ch/2010/04/understanding-javas-countdownlatch.html">detailed blog post on it here</a>.<br /><br /></li><li>If you're going to delete multiple objects at a time, you should confirm the S3 library you're using is thread safe.&nbsp; Many S3 libraries I've seen rely on the popular <a href="http://hc.apache.org/httpcomponents-client/">Apache Commons HttpClient</a> to handle the underlying HTTP communication work with S3.&nbsp; However, you should note that HttpClient isn't thread safe by default, unless you've explicitly set it up to use a <a href="http://hc.apache.org/httpcomponents-client/httpclient/apidocs/org/apache/http/impl/conn/tsccm/ThreadSafeClientConnManager.html">ThreadSafeClientConnManager</a>.</li></ul><br />Enjoy.]]>
    </content>
</entry>

<entry>
    <title>Integrating Ant and Google&apos;s Closure Compiler</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/08/integrating-ant-and-googles-closure-compiler.html?rss" />
    <id>tag:mark.koli.ch,2010://1.240</id>

    <published>2010-08-21T18:45:00Z</published>
    <updated>2010-08-23T16:11:23Z</updated>

    <summary><![CDATA[With a little spare time on my hands, I dug into Google's Closure Compiler, an open-source tool that minifies and optimizes JavaScript for better performance.&nbsp; Unlike other JavaScript compression tools like JSMin, Packer or the YUI compressor, Google's Closure Compiler...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="ant" label="ant" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="google" label="google" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="javascript" label="javascript" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[With a little spare time on my hands, I dug into <a href="http://code.google.com/closure/compiler/">Google's Closure Compiler</a>, an open-source tool that minifies and optimizes JavaScript for better performance.&nbsp; Unlike other JavaScript compression tools like <a href="http://www.crockford.com/javascript/jsmin.html">JSMin</a>, <a href="http://dean.edwards.name/packer/">Packer</a> or the <a href="http://developer.yahoo.com/yui/compressor/">YUI compressor</a>, Google's Closure Compiler converts your decent JavaScript into "better" JavaScript.&nbsp; It does so by searching for and pointing out obvious JavaScript pitfalls, deleting dead code, and optimizing what's left to make your web-applications more efficient.&nbsp; This all sounds great, so I gave it a whirl and eventually integrated it into my <a href="http://ant.apache.org/">Ant build</a> for a new project.&nbsp; Heck, I <a href="http://code.google.com/p/closure-compiler/issues/detail?id=221">even found a bug</a> and <a href="http://code.google.com/p/closure-compiler/issues/detail?id=220">suggested an enhancement</a>.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">1 - $jQuery Users be Warned</span><br /><br />Before you run off and have a field day with the Closure Compiler, you should know that optimizing jQuery with the <a href="http://code.google.com/closure/compiler/docs/api-tutorial3.html">compiler's ADVANCED_OPTIMIZATIONS</a> mode failed miserably.&nbsp; Even with a proper <a href="http://code.google.com/closure/compiler/docs/api-tutorial3.html#externs">JS externs</a> file, I could not get jQuery, or my JavaScript that uses jQuery, to compile under ADVANCED_OPTIMIZATIONS mode.&nbsp; Ok, actually it compiled fine, but when I tried to run this JavaScript in a browser: welcome to JavaScript Error City, population one.<br /><br />Eventually, I gave up and reverted back to SIMPLE_OPTIMIZATIONS mode instead. &nbsp;This works much better. &nbsp;If you know something I don't about compiling jQuery in advanced mode, please let me know.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">2 - Get 'er done</span><br /><br />First things first, you'll need to <a href="http://code.google.com/p/closure-compiler/downloads/list">download the Closure Compiler JAR</a> and integrate it into your Ant build file by defining a new Ant task:<br /><br /><pre class="prettyprint">&lt;property name="jscompjar"<br />  location="${lib.dir}/google-closure-compiler.jar" /&gt;<br /><br />&lt;taskdef name="jscomp"<br />  classname="com.google.javascript.jscomp.ant.CompileTask"<br />  classpath="${jscompjar}"/&gt;<br /></pre><br />Yay, now you have a &lt;jscomp&gt; Ant task.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">3 - Concatenate</span><br /><br />Next, you'll need Ant to concatenate each of your JavaScript files into a single file.&nbsp; Technically, you don't have to do this, but I found it much easier and I like having all of my JavaScript compressed into a single resource:<br /><br /><pre class="prettyprint">&lt;concat destfile="${build.dir}/js/project.js"&gt;<br />  &lt;filelist dir="${source.js.dir}"&gt;<br />    &lt;file name="underscore-1.0.4.min.js" /&gt;<br />    &lt;file name="jquery-1.4.2.min.js" /&gt;<br />    &lt;file name="jquery-ui-1.8.4.min.js" /&gt;<br />    &lt;file name="project.util.js" /&gt;<br />    &lt;file name="project.js" /&gt;<br />  &lt;/filelist&gt;<br />&lt;/concat&gt;</pre><br />Note that I'm using a <a href="http://ant.apache.org/manual/Types/filelist.html">FileList</a> instead of a <a href="http://ant.apache.org/manual/Types/fileset.html">FileSet</a>. &nbsp;This is because the order in which the resources are concatenated is important; they should be concatenated in the order in which they are required, and using a FileList ensures proper order.&nbsp; More on that <a href="http://mark.koli.ch/2009/12/a-better-way-to-include-javascript-and-css-in-your-web-apps-break-proxy-caching-too.html">here</a>.<br /><br />Also, note that if you eventually plan on compiling your JavaScript with ADVANCED_OPTIMIZATIONS enabled, you'll need to concatenate your JavaScript files together anyways.&nbsp; Further, using a single JavaScript resource across your entire site has obvious performance advantages.&nbsp; HTTP operations are usually expensive (<a href="http://about.digg.com/blog/technical-strategy-mdiggcom">especially on mobile devices</a>), so having a web-browser download one large JavaScript file with everything in it is usually better than having it download multiple smaller files.&nbsp; Of course, you <a href="http://blog.getify.com/2009/11/labjs-why-not-just-concat">could even take it a step further and use a tool like LABjs</a> to load your JavaScript dynamically, as needed. <br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">4 - Compile</span><br /><br />Ok, now that your JavaScript files have been concatenated into a single resource, let's use the &lt;jscomp&gt; Ant task we defined earlier to compile it:<br /><br /><pre class="prettyprint">&lt;jscomp compilationLevel="simple" warning="quiet"<br />  debug="false" output="${release.js.dir}/project.min.js"&gt;<br />  &lt;sources dir="${build.dir}/js"&gt;<br />    &lt;file name="project.js" /&gt;<br />  &lt;/sources&gt;<br />&lt;/jscomp&gt;<br /></pre><br />Assuming your code compiled cleanly, you'll have a Closure Compiler optimized JavaScript file sitting around ready for deployment!<br /><br />Enjoy.<br />]]>
        
    </content>
</entry>

<entry>
    <title>Spring 3 and Spring Security: Setting your Own Custom /j_spring_security_check Filter Processes URL</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/07/spring-3-and-spring-security-setting-your-own-custom-j-spring-security-check-filter-processes-url.html?rss" />
    <id>tag:mark.koli.ch,2010://1.239</id>

    <published>2010-07-25T02:20:00Z</published>
    <updated>2010-07-25T06:07:24Z</updated>

    <summary><![CDATA[While working on a new personal project, I decided to pick up and dig into Spring 3 MVC and Spring Security.&nbsp; I've touched both of these technologies here and there in a number of other projects, but this new opportunity...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="spring" label="spring" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="springsecurity" label="spring-security" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[While working on a new personal project, I decided to pick up and dig into <a href="http://static.springsource.org/spring/docs/3.0.x/spring-framework-reference/htmlsingle/spring-framework-reference.html">Spring 3 MVC</a> and <a href="http://static.springsource.org/spring-security/site/index.html">Spring Security</a>.&nbsp; I've touched both of these technologies here and there in a number of other projects, but this new opportunity has really opened the door for a deep dive into Spring.&nbsp; I know the Ruby fanatics out there would say I'm crazy, but once I got all of the awful XML configuration mess out of the way, I'm really enjoying Spring's stability and reliability.&nbsp; I know Ruby and Rails is hip and what not, but Spring 3 has really made an impression on me.&nbsp; It's hard to beat such a mature enterprise framework.<br /><br />I setup a few Spring 3 controllers, and integrated Spring Security into my web-app.&nbsp; All went great and so I added a simple form-based login to my Spring Security XML configuration.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">Problem: Overriding UsernamePasswordAuthenticationFilter</span><br /><br />When setting up a form-based login via a default Spring Security &lt;http:security&gt; configuration, Spring auto generates and configures a <a href="http://static.springsource.org/spring-security/site/docs/3.0.x/apidocs/org/springframework/security/web/authentication/UsernamePasswordAuthenticationFilter.html">UsernamePasswordAuthenticationFilter</a> bean.&nbsp; This filter, by default, responds to the URL /j_spring_security_check when processing a login POST from your web-form.&nbsp; First, I want to override Spring Security's default login process URL to /login instead of /j_spring_security_check.&nbsp; Second, I've configured a Spring 3 controller to display my login web-form when a user visits /login.<br /><br />That said, here's the underlying problem with Spring Security's default <a href="http://static.springsource.org/spring-security/site/docs/3.0.x/apidocs/org/springframework/security/web/authentication/UsernamePasswordAuthenticationFilter.html">UsernamePasswordAuthenticationFilter</a>: I want it to accept and process POST's to /login, but a GET or any HTTP method to /login should be forwarded to the next filter in the chain.&nbsp; Not surprisingly, you cannot do this with Spring Security's default <a href="http://static.springsource.org/spring-security/site/docs/3.0.x/apidocs/org/springframework/security/web/authentication/UsernamePasswordAuthenticationFilter.html">UsernamePasswordAuthenticationFilter</a> because it does not @Override the doFilter() method of <a href="http://static.springsource.org/spring-security/site/docs/3.0.x/apidocs/org/springframework/security/web/authentication/AbstractAuthenticationProcessingFilter.html">AbstractAuthenticationProcessingFilter</a>.&nbsp; In short, there's no way to get and check the incoming HTTP request method and re-route it using the default <a href="http://static.springsource.org/spring-security/site/docs/3.0.x/apidocs/org/springframework/security/web/authentication/UsernamePasswordAuthenticationFilter.html">UsernamePasswordAuthenticationFilter</a>.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">Solution: Write your own Spring Security Authentication Filter</span><br /><br />If you want a Spring controller to process GET requests to /login, but Spring Security to intercept and process a POST to /login, then you'll need to write your own Spring Security authentication filter.&nbsp; Here's the idea:<br /><br /><pre class="prettyprint">public class MyFilter extends AbstractAuthenticationProcessingFilter {<br /><br />  private static final String DEFAULT_FILTER_PROCESSES_URL = "/login";<br />  private static final String POST = "POST";<br /><br />  public MyFilter () {<br />    super(DEFAULT_FILTER_PROCESSES_URL);<br />  }<br /><br />  @Override<br />  public Authentication attemptAuthentication(HttpServletRequest request,<br />    HttpServletResponse response) throws AuthenticationException,<br />    IOException, ServletException {<br />    // You'll need to fill in the gaps here.  See the source of<br />    // UsernamePasswordAuthenticationFilter for a working implementation<br />    // you can leverage.<br />  }<br /><br />  @Override<br />  public void doFilter(ServletRequest req, ServletResponse res,<br />    FilterChain chain) throws IOException, ServletException {<br />    final HttpServletRequest request = (HttpServletRequest) req;<br />    final HttpServletResponse response = (HttpServletResponse) res;<br />    if(request.getMethod().equals(POST)) {<br />      // If the incoming request is a POST, then we send it up<br />      // to the AbstractAuthenticationProcessingFilter.<br />      super.doFilter(request, response, chain);<br />    } else {<br />      // If it's a GET, we ignore this request and send it<br />      // to the next filter in the chain.  In this case, that<br />      // pretty much means the request will hit the /login<br />      // controller which will process the request to show the<br />      // login page.<br />      chain.doFilter(request, response);<br />    }<br />  }<br /><br />}<br /></pre><br />Note the good stuff inside of <b>doFilter()</b>.&nbsp; If the incoming request method is a POST, then we send it up to our AbstractAuthenticationProcessingFilter to actually process the login.&nbsp; If it's a GET, or any other HTTP request method for that matter, we simply send it to the next filter in the chain.<br /><br />Finally, remember that you'll need to define your own <b>FORM_LOGIN_FILTER</b> inside of your &lt;security:http&gt; Spring Security XML configuration to override the default /j_spring_security_check URL:<br /><br /><pre class="prettyprint">&lt;security:http auto-config="false" use-expressions="true"<br />  entry-point-ref="LoginUrlAuthenticationEntryPoint"&gt;<br />  &lt;security:custom-filter position="FORM_LOGIN_FILTER" ref="MyFilter" /&gt;<br />&lt;/security:http&gt;<br /><br />&lt;bean id="LoginUrlAuthenticationEntryPoint"<br />  class="org.springframework.security.web.authentication.LoginUrlAuthenticationEntryPoint"&gt;<br />  &lt;property name="loginFormUrl" value="/login" /&gt;<br />&lt;/bean&gt;</pre><br />Enjoy!]]>
        
    </content>
</entry>

<entry>
    <title>Extended Thoughts on Custom Tiny URL Engines</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/05/extended-thoughts-on-custom-tiny-url-engines.html?rss" />
    <id>tag:mark.koli.ch,2010://1.237</id>

    <published>2010-05-31T23:35:00Z</published>
    <updated>2010-06-01T02:07:19Z</updated>

    <summary><![CDATA[A blog reader recently contacted me via email and asked, "hey, how does your koli.ch tiny URL thing work?"&nbsp; Well, I would be happy to explain.&nbsp; As previously discussed here, and here, I'm not using Apache's mod_rewrite RewriteMap engine.&nbsp; No,...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="apache" label="apache" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="base36" label="base36" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="http" label="http" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="php" label="php" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[A blog reader recently contacted me via email and asked, "hey, how does your koli.ch tiny URL thing work?"&nbsp; Well, I would be happy to explain.&nbsp; As previously discussed <a href="http://mark.koli.ch/2009/03/howto-make-your-own-tiny-url-shortener-service.html">here</a>, and <a href="http://mark.koli.ch/2009/03/update-howto-make-your-own-tiny-url-alias-service-tinyurl-bitly-twurlcc-etc-with-apache-rewritemap.html">here</a>, I'm not using <a href="http://httpd.apache.org/docs/2.2/mod/mod_rewrite.html#rewritemap">Apache's mod_rewrite RewriteMap</a> engine.&nbsp; No, koli.ch tiny URL's are using an interesting combination of Apache's mod_rewrite RewriteRule directive and <a href="https://onyx.koli.ch/">Onyx</a>, which is basically <a href="http://mark.koli.ch/2009/10/onyx-my-custom-solution-to-the-digital-clutter-problem.html">PHP talking to a MySQL database</a>.<br /><br />So, let's follow <a href="http://koli.ch/x3dh">http://koli.ch/x3dh</a> through the redirection process.&nbsp; The "3dh" you see on the end of each URL is a <a href="http://mark.koli.ch/2009/10/base36-encoding-for-tiny-urls-with-php.html">base-36 encoded number</a>, that maps to a unique bookmark resource stored inside of <a href="https://onyx.koli.ch/">Onyx</a>.<br /><br /><span style="font-size: 1.1em; font-weight: bold;">Step 1 - koli.ch to mark.koli.ch</span><br /><br />First, you may have noticed that http://koli.ch always redirects to http://mark.koli.ch by default.&nbsp; This is accomplished using Apache's mod_rewrite in my virtual host configuration ...<br /><br /><pre class="prettyprint">RewriteCond %{HTTP_HOST} !^mark\.koli\.ch [NC]<br />RewriteRule ^/(.*)$ http://mark.koli.ch/$1 [R=301,L]</pre><br />You'll notice that the Perl compatible regular expression in the RewriteRule directive captures the URL-path content after the first slash, allowing me to gracefully forward URL's like <a href="http://koli.ch/about-mark.html">http://koli.ch/about-mark.html</a> to <a href="http://mark.koli.ch/about-mark.html">http://mark.koli.ch/about-mark.html</a>.&nbsp; Consequently, tiny URL's like http://koli.ch/x3dh are forwarded to http://mark.koli.ch/x3dh.&nbsp; And, this redirect is a <a href="http://en.wikipedia.org/wiki/HTTP_301">301 Moved Permanently</a> instead of a default 302 Found redirect.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">Step 2 - mark.koli.ch to onyx.koli.ch</span><br /><br />Next, in the same virtual host configuration, I use mod_rewrite's RewriteRule directive to match URL-paths that represent a tiny URL.&nbsp; As far as <a href="https://onyx.koli.ch/">Onyx</a> is concerned, all of my tiny URL's all start with "x" and contain at least one, but no more than ten, word characters following the "x" ...<br /><br /><pre class="prettyprint">RewriteCond %{REQUEST_URI} !\.(htm?l)$ [NC]<br />RewriteRule ^/x(\w{1,10})$ https://onyx.koli.ch/x$1 [L]</pre><br />Notice the regular expression group, I'm only capturing the word-characters following the "x" in the request URI, then appending them onto <a href="https://onyx.koli.ch/x">https://onyx.koli.ch/x</a> before handing it off to Onyx.&nbsp; Note that I have not specified the redirect response code in the RewriteRule options.&nbsp; In this case, Apache will <a href="http://en.wikipedia.org/wiki/HTTP_302">302 Found</a> redirect to Onyx, which tells search engines and other bots that this redirect is much less permanent than a <a href="http://en.wikipedia.org/wiki/HTTP_301">301 Moved Permanently</a>.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">Step 3 - onyx.koli.ch to ?</span><br /><br />Once the browser hits <a href="https://onyx.koli.ch/x">https://onyx.koli.ch/x</a>, I use RewriteRule one last time to call a redirection controller I wrote in PHP which actually takes the tiny URL, sanitizes the request URI then looks it up in a MySQL database to retrieve the final URL to redirect to.&nbsp; This is part of Onyx.&nbsp; Here's the pseudo code ... <br /><br /><pre class="prettyprint">&lt;?php<br /><br />$uri = (isset($_SERVER['REQUEST_URI'])) ? $_SERVER['REQUEST_URI'] : null;<br />if(empty($uri)) {<br />  throw new InternalErrorException();<br />}<br /><br />// Strip the leading / from the front of the URI<br />$uri = preg_replace("/^\//","",$uri);<br />@list(, $resource) = preg_split("/[\/\?]/", $uri, -1, PREG_SPLIT_NO_EMPTY);<br />if(!empty($resource)) {<br />  $resource = preg_replace("/\D/","",$resource);<br />} else {<br />  throw new BadRequestException();<br />}<br /><br />// ... do some MySQL things here, look up the $resource in the DB.<br />$url = ... some URL from the database<br /><br />// Redirect the user to their destination, via an HTTP 302 Found<br />header(HTTP_302_FOUND);<br /><br />// Unset the Pragma HTTP response header; to avoid<br />// redirection issues on IE.<br />header(PRAGMA.": ");<br />header(CACHE_CONTROL.": no-store, no-cache");<br /><br />header(LOCATION_HEADER.': '.$url);<br /><br />?&gt;<br /></pre><br />That's it.&nbsp; Cheers.]]>
        
    </content>
</entry>

<entry>
    <title>MySQL Triggers and SUPER Privileges: &quot;Access denied; you need the SUPER privilege for this operation.&quot;</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/05/mysql-triggers-and-super-privileges-access-denied-you-need-the-super-privilege-for-this-operation.html?rss" />
    <id>tag:mark.koli.ch,2010://1.236</id>

    <published>2010-05-27T19:20:00Z</published>
    <updated>2010-05-27T19:33:41Z</updated>

    <summary><![CDATA[I just discovered that dealing with MySQL triggers, in many instances, is quite painful.&nbsp; For example, here's a trigger that deletes a bunch of rows in a table on every INSERT:delimiter |CREATE TRIGGER delete_expired_tweets AFTER INSERT ON tweets FOR EACH...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="mysql" label="mysql" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="security" label="security" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[I just discovered that dealing with MySQL triggers, in many instances, is quite painful.&nbsp; For example, here's a trigger that deletes a bunch of rows in a table on every INSERT:<br /><br /><pre class="prettyprint">delimiter |<br />CREATE TRIGGER delete_expired_tweets AFTER INSERT ON tweets<br />  FOR EACH ROW BEGIN<br />    DELETE FROM tweets WHERE DATEDIFF(NOW(), created_at) &gt; 365;<br />  END;<br />|<br /><br />delimiter ;<br /></pre><br />Ok, let's load this trigger into MySQL:<br /><br /><pre class="prettyprint">#/&gt; mysql -h myhost -u normaluser -p mydatabase<br />Enter password: **********<br />Welcome to the MySQL monitor.  Commands end with ; or \g.<br />Your MySQL connection id is 10<br />Server version: 5.0.41-community-nt MySQL Community Edition (GPL)<br /><br />Type 'help;' or '\h' for help. Type '\c' to clear the buffer.<br /><br />mysql&gt; source trigger.sql<br />ERROR 1227 (42000): Access denied; you need the SUPER privilege<br />    for this operation</pre><br />Access denied?&nbsp; I dug into it, and confirmed that <strong>you can only add triggers if your user account has the SUPER privilege enabled</strong>.&nbsp; You're probably thinking, "No kidding Sherlock, that's what the error message says."&nbsp; Yes, I know that's what the error message says.&nbsp; But here's the problem.&nbsp; Normal database users created using "GRANT ALL PRIVILEGES ON database.* TO..." will <strong>not have the SUPER privilege</strong> assigned to them by default.&nbsp; As <a href="http://dev.mysql.com/doc/refman/5.0/en/privileges-provided.html#priv_super">described here</a>, the SUPER privilege in MySQL let's the account do some things that normal database users, in most environments, should not be able to do (like kill database threads, modify global system variables, etc.).&nbsp; As a result, it's a very bad idea to grant the SUPER privilege to normal database users, even if they just need the SUPER privilege to load a trigger.&nbsp; You know better than that!<br /><br />Even worse, suppose you GRANT SUPER PRIVILEGES to a single user, on a single database.&nbsp; Well, that still won't be enough to load a trigger.&nbsp; Unfortunately, loading triggers requires SUPER PRIVILEGES at the global level (e.g., GRANT SUPER PRIVILEGES ON *.*).&nbsp; Again, it's a very bad ideal to grant normal database users the SUPER privilege.<br /><br />So how exactly am I supposed to load this trigger?&nbsp; Well as far as I can tell, assuming I refuse to give myself SUPER PRIVILEGES for the reasons I just explained, I have two options:<br /><br /><ol><li>Don't use triggers, and find another way to cleanup rows in my table.</li><li>Log into the database as root/admin and load the trigger on behalf of the normal user.&nbsp; If I wasn't the owner of this database server, this would probably involve asking my database administrator to load the trigger for me.<br /></li></ol><br />This is just one of many common annoyances with MySQL.&nbsp; Sucks, I know.<br />]]>
        
    </content>
</entry>

<entry>
    <title>OAuth and the Twitter API: Generate a one-time access token and token secret</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/05/oauth-and-the-twitter-api-generate-a-one-time-access-token-and-token-secret.html?rss" />
    <id>tag:mark.koli.ch,2010://1.235</id>

    <published>2010-05-23T19:05:00Z</published>
    <updated>2010-10-15T05:55:11Z</updated>

    <summary><![CDATA[This is not a Twitter and OAuth tutorial.&nbsp; This post does not talk about how to use the Twitter API, nor does it offer any examples.&nbsp; For that, you should go elsewhere.&nbsp; This post simply provides a high-level overview of...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="api" label="api" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="php" label="php" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="twitter" label="twitter" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[<span style="font-size: 1.1em; font-weight: bold;">This is not a Twitter and OAuth tutorial.&nbsp; This post does not talk about how to use the Twitter API, nor does it offer any examples.&nbsp; For that, you should go elsewhere.&nbsp; This post simply provides a high-level overview of using an OAuth "one-time access token and secret" when you need to write a quick piece of code that fetches read-only data from the Twitter API.</span><br /><br />You may have heard that <a href="http://groups.google.com/group/twitter-api-announce/browse_thread/thread/f71eb68600996af8?pli=1">Twitter plans to stop supporting HTTP Basic Authentication on June 30, 2010</a>.&nbsp; This means that starting on June 30th, to use <a href="http://apiwiki.twitter.com/Twitter-API-Documentation">Twitter's API</a>, your application must support <a href="http://oauth.net/">OAuth</a>.&nbsp; OAuth is a nice step up from basic authentication but it makes developing web or desktop applications that communicate with Twitter, slightly more painful.&nbsp; Well, painful isn't the right word, but you definitely have to jump through more hoops to get things to work.&nbsp; Gone are the days of simply sending a username and password to the API.<br /><br />In response to this change, Twitter API proxy services like <a href="http://www.supertweet.net/">SuperTweet</a> have popped up.&nbsp; Turns out, if you know what you're doing with OAuth, SuperTweet and other API proxy services are entirely unnecessary, not to mention unsafe.&nbsp; You're better off upgrading your applications to use OAuth the right way, instead of making them rely on potentially insecure third-party proxy services.&nbsp; And again, it's not difficult, just a bit annoying.<br /><br /><span style="font-size: 1.1em; font-weight: bold;">Scenario</span><br /><br />You're a developer, and you need to write some code that pulls in Tweets from one or more users.&nbsp; Maybe you also need to pull down a list of followers for each of these users.&nbsp; Not surprisingly, it's entirely unreasonable to ask each of them to authenticate your application using OAuth.&nbsp; You just want to write code that pulls down their public timeline, followers, etc. avoiding the whole OAuth dance with each user, every time.<br /><br /><span style="font-size: 1.1em; font-weight: bold;">Solution</span><br /><br />Register a new application on Twitter.&nbsp; Then, dig into your application control panel and find your new "single access token" and "single access token secret" for the application you just registered.<br /><br />As <a href="http://dev.twitter.com/pages/oauth_single_token">described here</a>, "Twitter offers the ability for you to retrieve a single access token (complete with oauth_token_secret) from application detail pages found in your application control panel.&nbsp; This is ideal for applications migrating to OAuth with single-user use cases ... By using a single access token, you don't need to implement the entire OAuth token acquisition dance. Instead, you can pick up from the point where you are working with an access token to make signed requests for Twitter resources."<br /><br />This token and token secret is as close as you'll get to a 
username/password equivalent in OAuth.&nbsp; In other words, once you have this one-time token and token secret for your application, you can issue signed OAuth requests against the Twitter API just like you would with a basic username and password.&nbsp; If you want to think about it this way, the token is like your username and the token secret is like your password.&nbsp; Don't share them.&nbsp; Once you have these credentials, you can pull in Tweets for any public user, get their followers, read the public timeline, etc.<br /><br />Here are <a href="http://dev.twitter.com/pages/oauth_single_token">several examples in a number of popular languages showing how you can use this one-time token and token secret</a> in your project.<br /><br />Yay for OAuth.]]>
        
    </content>
</entry>

<entry>
    <title>Formatting a Java Date into a Specific TimeZone and Conversion Between TimeZone&apos;s</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/05/formatting-a-java-date-into-a-specific-timezone.html?rss" />
    <id>tag:mark.koli.ch,2010://1.234</id>

    <published>2010-05-15T18:51:00Z</published>
    <updated>2010-05-15T20:33:22Z</updated>

    <summary><![CDATA[Date objects in Java, and probably most other robust languages, simply represent a snapshot of a point in time.&nbsp; In other words, java.util.Date knows nothing about the time zone you're referring to when instantiating or manipulating a Date object.&nbsp; Fact...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="code" label="code" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="howto" label="howto" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[Date objects in Java, and probably most other robust languages, simply represent a snapshot of a point in time.&nbsp; In other words, <a href="http://java.sun.com/javase/6/docs/api/java/util/Date.html">java.util.Date</a> knows nothing about the time zone you're referring to when instantiating or manipulating a Date object.&nbsp; Fact is, java.util.Date does not have to care about your time zone, because internally a Date is really nothing more than a count of the number of milliseconds since the standard base time known as "<a href="http://en.wikipedia.org/wiki/Unix_time">the epoch</a>", namely January 1, 1970, 00:00:00 GMT.<br /><br />If it helps, think about it this way: X milliseconds since the epoch is X milliseconds since the epoch in the US-Pacific time zone, X milliseconds since the epoch in GMT-0 (London), X milliseconds since the epoch in India, etc.&nbsp; In short, when it's 1273947282085 milliseconds since the epoch, it's 1273947282085 milliseconds since the epoch everywhere in the world at the same time regardless of what time zone you're sitting in.&nbsp; And since Java's util.Date is simply a snapshot of the number of milliseconds at a specific point in time, you can see why Date doesn't care about your time zone.&nbsp; It's irrelevant.<br /><br />But Mark, <b>how do I convert a java.util.Date into a different time zone</b>?&nbsp; You can't, and that question makes no sense.&nbsp; That's like asking me to "take a picture of the sound."&nbsp; Here's some <a href="http://en.wikipedia.org/wiki/Shit#Shortening_of_bullshit">B.S.</a> code that you should not use, but I've put it here for illustrative purposes:<br /><br /><pre class="prettyprint">// Do NOT use this, it does nothing and makes no sense.<br />public static final Date convertIntoTimeZone(Date date, TimeZone tz) {<br />  final Calendar cal = Calendar.getInstance();<br />  cal.setTime(date);<br />  cal.setTimeZone(tz);<br />  return cal.getTime();<br />}</pre><br />You can't convert a Date into a different time zone, but you can use Java's handy <a href="http://java.sun.com/javase/6/docs/api/java/text/DateFormat.html">DateFormat</a> class to format a Date into the time zone of your choice.&nbsp; To put it differently, let Date do its thing.&nbsp; Then, <b>when you're ready to display or print out a String representation of Date, that's when you tell DateFormat what time zone you want it in</b>.&nbsp; So, here's some code that makes sense, and actually works:<br /><br /><pre class="prettyprint">final Date currentTime = new Date();<br /><br />final SimpleDateFormat sdf =<br />        new SimpleDateFormat("EEE, MMM d, yyyy hh:mm:ss a z");<br /><br />// Give it to me in US-Pacific time.<br />sdf.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));<br />System.out.println("LA time: " + sdf.format(currentTime));<br /><br />// Give it to me in GMT-0 time.<br />sdf.setTimeZone(TimeZone.getTimeZone("GMT"));<br />System.out.println("GMT time: " + sdf.format(currentTime));<br /><br />// Or maybe Zagreb local time.<br />sdf.setTimeZone(TimeZone.getTimeZone("Europe/Zagreb"));<br />System.out.println("Zagreb time: " + sdf.format(currentTime));<br /><br />// Even 10 hours and 10 minutes ahead of GMT<br />sdf.setTimeZone(TimeZone.getTimeZone("GMT+0010"));<br />System.out.println("10/10 ahead time: " + sdf.format(currentTime));<br /></pre><br />Yay for Dates and time zones!&nbsp; Enjoy.]]>
        
    </content>
</entry>

<entry>
    <title>HTTP Digest Access Authentication using MD5 and HttpClient 4</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/05/http-digest-access-authentication-with-md5-and-httpclient-4.html?rss" />
    <id>tag:mark.koli.ch,2010://1.233</id>

    <published>2010-05-04T20:30:00Z</published>
    <updated>2010-05-04T20:53:58Z</updated>

    <summary><![CDATA[Dealing with HTTP's Digest authentication mechanism isn't too bad once you have the basic building blocks in place.&nbsp; Luckily HttpClient 4 can automatically solve many types of authentication challenges for you, if used correctly.&nbsp; Using HttpClient 4, I built an...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="code" label="code" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="http" label="http" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="httpclient" label="httpclient" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="md5" label="md5" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="soap" label="soap" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[Dealing with HTTP's Digest authentication mechanism isn't too bad once you have the basic building blocks in place.&nbsp; Luckily <a href="http://hc.apache.org/httpcomponents-client/index.html">HttpClient 4</a> can automatically solve many types of authentication challenges for you, if used correctly.&nbsp; Using <a href="http://hc.apache.org/httpcomponents-client/index.html">HttpClient 4</a>, I built an app that authenticates against a SOAP based web-service requiring WWW-Authenticate Digest authentication.&nbsp; In a nutshell, the fundamental principal behind HTTP Digest authentication is simple:<br /><br /><ul><li>The client asks for a page that requires authentication.<br /></li><li>The server responds with an HTTP 401 response code, providing the authentication realm and a randomly-generated, single-use value called a "nonce".&nbsp; The authentication "challenge" itself is encapsulated inside of the WWW-Authenticate HTTP response header.<br /></li><li>The client "solves" the authentication challenge and a solution is sent back to the web-server via the HTTP Authorization header on a subsequent request.&nbsp; The solution usually contains some type of MD5 hashed mess of your username, password, and "nonce".<br /></li><li>Assuming the solution is acceptable the server responds with a successful type response, usually an HTTP 200 OK.<br /></li></ul><br />Here's a sample with a bit of pseudo code mixed in (so, you get the idea):<br /><br /><pre class="prettyprint">// A org.apache.http.impl.auth.DigestScheme instance is<br />// what will process the challenge from the web-server<br />final DigestScheme md5Auth = new DigestScheme();<br /><br />// This should return an HTTP 401 Unauthorized with<br />// a challenge to solve.<br />final HttpResponse authResponse =<br />        doPost(url, postBody, contentType);<br /><br />// Validate that we got an HTTP 401 back<br />if(authResponse.getStatusLine().getStatusCode() ==<br />      HttpStatus.SC_UNAUTHORIZED) {<br />  if(authResponse.containsHeader("WWW-Authenticate")) {<br />    // Get the challenge.<br />    final Header challenge =<br />            authResponse.getHeaders("WWW-Authenticate")[0];<br />    // Solve it.<br />    md5Auth.processChallenge(challenge);<br />    // Generate a solution Authentication header using your<br />    // username and password.<br />    final Header solution = md5Auth.authenticate(<br />            new UsernamePasswordCredentials(username, password),<br />            new BasicHttpRequest(HttpPost.METHOD_NAME,<br />                  new URL(url).getPath()));<br />    // Do another POST, but this time include the solution<br />    // Authentication header as generated by HttpClient.<br />    final HttpResponse goodResponse =<br />            doPost(url, postBody, contentType, solution);<br />    // ... do something useful with goodResponse, which assuming<br />    // your credentials were valid, should contain the data you<br />    // requested.<br />  } else {<br />    throw new Error("Web-service responded with Http 401, " +<br />      "but didn't send us a usable WWW-Authenticate header.");<br />  }<br />} else {<br />  throw new Error("Didn't get an Http 401 " +<br />      "like we were expecting.");<br />}</pre><br />Enjoy.<br />]]>
        
    </content>
</entry>

<entry>
    <title>Pingdom Incorrectly Reports Page Load Time</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/04/pingdom-incorrectly-reports-page-load-time.html?rss" />
    <id>tag:mark.koli.ch,2010://1.232</id>

    <published>2010-04-24T20:00:00Z</published>
    <updated>2010-04-24T20:03:16Z</updated>

    <summary><![CDATA[I recently discovered that Pingdom, in many instances, does not correctly report page-load time.&nbsp; As it turns out, when you ask Pingdom to tell you how fast your site loads it will incorrectly download all images and other resources referenced...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="commonsense" label="commonsense" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="compression" label="compression" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="css" label="css" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[I recently discovered that <a href="http://www.pingdom.com/">Pingdom</a>, in many instances, does not correctly report page-load time.&nbsp; As it turns out, when you ask Pingdom to tell you how fast your site loads it will incorrectly download all images and other resources referenced in your CSS, regardless of whether they are actually used on your site or not.&nbsp; Of course this isn't ideal, because Pingdom will claim the page you're testing takes longer to load than it really does!&nbsp; By comparison, smart web-browsers do not blindly download every image referenced in the CSS, on every page.&nbsp; Instead, they download the resources as needed.<br /><br />For folks who really care about site optimization and performance, 
this is a big deal.&nbsp; Especially if you're hired to help a client optimize the page-load time of their web-site, but services like Pingdom lie and report junk.<br /><br />Here's an example; imagine this is your homepage:<br /><br /><pre class="prettyprint">&lt;html&gt;<br />&lt;head&gt;<br /> &lt;title&gt;My Homepage&lt;/title&gt;<br /> &lt;style type="text/css"&gt;<br />  #header {<br />    background: black url(/header.png) no-repeat top left;<br />  }<br />  #homepage {<br />    background: white url(/homepage.png) no-repeat top left;<br />  }<br />  #aboutus {<br />    background: white url(/about-us.png) no-repeat top left;<br />  }<br />  #contact {<br />    background: white url(/contact.png) no-repeat top left;<br />  }<br />  #footer {<br />    background: black url(/footer.png) no-repeat top left;<br />  }<br /> &lt;/style&gt;<br />&lt;/head&gt;<br /> &lt;body&gt;<br />  &lt;div id="header"&gt;header&lt;/div&gt;<br />  &lt;div id="homepage"&gt;content&lt;/div&gt;<br />  &lt;div id="footer"&gt;footer&lt;/div&gt;<br /> &lt;/body&gt;<br />&lt;/html&gt;<br /></pre><br />Ok, so you've got a header and a footer that will appear on every page.&nbsp; In addition, you've got a few other CSS selectors for the homepage, for your "about us" page, and for your "contact us" page.&nbsp; In a real web-browser, when you visit this page the browser will load the following images as referenced in your CSS:<br /><br /><ul><li>header.png</li><li>homepage.png</li><li>footer.png</li></ul>However, when you test this same page using Pingdom, Pingdom will claim your site is loading the following resources:<br /><br /><ul><li>header.png</li><li>homepage.png</li><li>about-us.png</li><li>contact.png</li><li>footer.png</li></ul>Wait a second, that's not accurate at all!&nbsp; On this hypothetical homepage, I'm definitely not using #aboutus or #contact anywhere, so why should the resources associated with these selectors be factored into my page load time?&nbsp; In reality, they shouldn't be because that's not how real web-browsers work.<br /><br />To further validate my findings, I used the Rackspace Cloud and allocated a new machine with a running web-server.&nbsp; I loaded this hypothetical homepage HTML onto it, and asked Pingdom to tell me how fast it loads:<br /><br /><span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://mark.koli.ch/2010/04/24/pingdom-lies-snap1.jpg"><img alt="pingdom-lies-snap1.jpg" src="http://mark.koli.ch/assets_c/2010/04/pingdom-lies-snap1-thumb-400x202.jpg" class="mt-image-none" style="" height="202" width="400" /></a></span><br /><br />Wrong!&nbsp; I'm not using aboutus.png or contactus.png anywhere on the homepage, so why should Pingdom factor the load time of those images into my test results?&nbsp; In contrast, using Firefox and <a href="http://code.google.com/p/httpfox/">HttpFox</a>, I verified that real web-browsers only load the resources they need to properly render the page:<br /><br /><span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://mark.koli.ch/2010/04/24/pingdom-lies-browser-is-better.jpg"><img alt="pingdom-lies-browser-is-better.jpg" src="http://mark.koli.ch/assets_c/2010/04/pingdom-lies-browser-is-better-thumb-400x200.jpg" class="mt-image-none" style="" height="200" width="400" /></a></span><br /><br />Here you can see the correct resources are loaded in Firefox: header.png, homepage.png, and footer.png.<br /><br />Bottom line, you really shouldn't use Pingdom and other external performance analysis tools as "bibles" for improving your page-load time.&nbsp; Clearly they're only tools to help you identify obvious problems, and are definitely not a complete solution.&nbsp; For more accurate results, I would suggest using tools like <a href="http://developer.yahoo.com/yslow/">YSlow</a> and <a href="http://code.google.com/speed/page-speed/">Google's Page Speed</a>.<br /><br />Cheers.<br />]]>
        
    </content>
</entry>

<entry>
    <title>Understanding Java&apos;s CountDownLatch and CyclicBarrier</title>
    <link rel="alternate" type="text/html" href="http://mark.koli.ch/2010/04/understanding-javas-countdownlatch.html?rss" />
    <id>tag:mark.koli.ch,2010://1.231</id>

    <published>2010-04-09T18:20:00Z</published>
    <updated>2010-04-14T02:39:33Z</updated>

    <summary><![CDATA[While working on some nifty multi-threaded Java recently, a colleague pointed me to a few really useful Java classes: CountDownLatch and CyclicBarrier.&nbsp; My code was quite typical, a parent worker thread spawns a bunch of children to do real work,...]]></summary>
    <author>
        <name>Mark Kolich</name>
        <uri>http://mark.koli.ch</uri>
    </author>
    
    <category term="code" label="code" scheme="http://www.sixapart.com/ns/types#tag" />
    <category term="java" label="java" scheme="http://www.sixapart.com/ns/types#tag" />
    
    <content type="html" xml:lang="en-US" xml:base="http://mark.koli.ch/">
        <![CDATA[<span class="mt-enclosure mt-enclosure-image" style="display: inline;"><a href="http://mark.koli.ch/2010/04/09/countdownlatch-demo-screenshot.png"><img alt="countdownlatch-demo-screenshot.png" src="http://mark.koli.ch/assets_c/2010/04/countdownlatch-demo-screenshot-thumb-200x208.png" class="mt-image-left" style="margin: 0pt 20px 20px 0pt; float: left;" height="208" width="200" /></a></span>While working on some nifty multi-threaded Java recently, a colleague pointed me to a few really useful Java classes: <a href="http://java.sun.com/javase/6/docs/api/java/util/concurrent/CountDownLatch.html">CountDownLatch</a> and <a href="http://java.sun.com/javase/6/docs/api/java/util/concurrent/CyclicBarrier.html">CyclicBarrier</a>.&nbsp; My code was quite typical, a parent worker thread spawns a bunch of children to do real work, and needs to wait for the children to finish before continuing.&nbsp; The catch though, is that the child worker threads may or may not finish successfully, and in all likelihood will finish at different times.&nbsp; Even so, the parent thread must wait until all of its children have finished because the parent can only make forward progress once the children are complete.&nbsp; I <a href="http://mark.koli.ch/2010/04/09/kolich.com-countdownlatch-swing-example.zip">whipped up a little demo</a> (screen shot left) that spawns five worker threads which update a JProgressBar at a random interval.&nbsp; The demo finishes once each progress bar hits 100%.]]>
        <![CDATA[<br /><span style="font-size: 1.1em; font-weight: bold;">1 - CountDownLatch</span><br /><br />Meet <a href="http://java.sun.com/javase/6/docs/api/java/util/concurrent/CountDownLatch.html">CountDownLatch</a>.&nbsp; As described <a href="http://java.sun.com/javase/6/docs/api/java/util/concurrent/CountDownLatch.html">in the Java 6 API docs</a>, a CountDownLatch is "a synchronization aid that allows one or more threads to wait until a set of operations being performed in other threads completes."&nbsp; In other words, the developer says new CountDownLatch(N) which waits for N threads to finish before the latch is "released" allowing the calling thread to make forward progress.&nbsp; Couldn't be more perfect here.&nbsp; To make my life a little easier, I wrote a few wrapper classes that encapsulate a CountDownLatch which allow me to easily synchronize on a List&lt;BaseWorker&gt;, a list of worker threads:<br /><br /><ul><li><a href="http://mark.koli.ch/2010/04/09/ThreadRunner.java">ThreadRunner.java</a> -- A class that accepts a List&lt;BaseWorker&gt; (a List of BaseWorker's), creates a new CountDownLatch(list.size()), starts each BasedWorker then allows the developer to await() on the runner for all BaseWorker's to finish.<br /><br /></li><li><a href="http://mark.koli.ch/2010/04/09/BaseWorker.java">BaseWorker.java</a> -- An abstract class that represents each worker thread, and defines a set of methods each BaseWorker must implement to be used with a ThreadRunner.<br /></li></ul><br />So, using these wrappers, let's create a new worker:<br /><br /><pre class="prettyprint">public class MyWorker extends BaseWorker {<br /><br />  private int someField_;<br /><br />  public MyWorker(int worker) {<br />    super();<br />    someField_ = worker;<br />    // ...<br />  }<br /><br />  @Override<br />  public void myRun() throws Exception {<br />    // ...<br />  }<br /><br />  @Override<br />  public String getWorkerName() {<br />    return String.format("%s #%s",<br />        getClass().getSimpleName(), someField_);<br />  }<br /><br />}<br /></pre><br />Now let's setup a new ThreadRunner that will take a bunch of BaseWorker's, start them, then wait for all to finish:<br /><br /><pre class="prettyprint">public class MyRunner {<br /><br />  private static final List&lt;BaseWorker&gt; workers__;<br />  static {<br />    workers__ = new ArrayList&lt;BaseWorker&gt;();<br />    workers__.add(new MyWorker(1));<br />    workers__.add(new MyWorker(2));<br />    workers__.add(new MyWorker(3));<br />  }<br /><br />  public static void main(String[] args) {<br />    final ThreadRunner runner = new ThreadRunner(workers__);<br />    // Start all of the threads in this runner.<br />    runner.start();<br />    // Wait for all of the threads to finish.<br />    runner.await();<br />    // Did all of our workers complete without error?<br />    if(runner.wasSuccessful()) {<br />      System.out.println("All workers finished cleanly.");<br />    } else {<br />      System.out.println("Not all workers finished cleanly.");<br />    }<br />  }<br /><br />}<br /></pre><br />In this example, I built a List of BaseWorker's, gave the list to the ThreadRunner and asked the runner to start them.&nbsp; Upon calling runner.await(), the ThreadRunner blocks waiting for all of the workers to finish.&nbsp; Note that my concept of "finish" here means either successfully, or unsuccessfully (an Exception or Error case).&nbsp; Subsequently, I call runner.wasSuccessful() to check if all of the workers finished cleanly, basically asking the runner did all of your workers finish without throwing any Exception's or Error's?<br /><br />If you're interested, you can <a href="http://mark.koli.ch/2010/04/09/kolich.com-countdownlatch-swing-example.zip">download my complete ThreadRunner demo/example</a> that further demonstrates the usage of these wrapper classes using Swing and several JProgressBar's.<br /><br /><br /><span style="font-size: 1.1em; font-weight: bold;">2 - CyclicBarrier</span><br /><br />A <a href="http://java.sun.com/javase/6/docs/api/java/util/concurrent/CyclicBarrier.html">CyclicBarrier</a> is similar to that of a CountDownLatch, except that a <a href="http://java.sun.com/javase/6/docs/api/java/util/concurrent/CyclicBarrier.html">CyclicBarrier</a> is "a synchronization aid that allows a set of threads to all wait for each other to reach a common barrier point."&nbsp; Like a CountDownLatch, a CyclicBarrier can be used to synchronize a number of threads.&nbsp; But instead of exiting upon completion, theads using a CyclicBarrier await() for all other threads in the pool to finish.&nbsp; Here's a usage example of a CyclicBarrier built around my BaseWorker class:<br /><br /><pre class="prettyprint">public class MyCyclicWorker extends BaseWorker {<br /><br />  private CyclicBarrier barrier_;<br /><br />  public MyWorker(CyclicBarrier barrier) {<br />    super();<br />    barrier_ = barrier;<br />    // ...<br />  }<br /><br />  @Override<br />  public void myRun() throws Exception {<br />    // ...<br />    // Wait here for all other threads<br />    // in the CyclicBarrier to finish.<br />    barrier_.await();<br />  }<br /><br />  @Override<br />  public String getWorkerName() {<br />    return getClass().getSimpleName();<br />  }<br /><br />} <br /></pre><br />Here's the class that starts up a bunch of these MyCyclicWorkers, then runs a single "cleanup" thread once all of the workers are done:<br /><br /><pre class="prettyprint">public class CyclicExample {<br /><br />  private static final int CYCLIC_THREADS = 5;<br /><br />  public static void main(String[] args) {<br />    final CyclicBarrier barrier =<br />            new CyclicBarrier(CYCLIC_THREADS,<br />              new Runnable() {<br />                @Override<br />                public void run() {<br />                  // Cleanup thread, or completion thread.<br />                  // Called when all of the worker threads<br />                  // are finished.<br />                  // ...<br />                }<br />              });<br />    for(int i=0; i &lt; CYCLIC_THREADS; ++i) {<br />      new MyCyclicWorker(barrier).start();<br />    }<br />    // ...<br />  }<br /><br />}<br /></pre><br />Enjoy!<br />]]>
    </content>
</entry>

</feed>

