Mark S. Kolich

I’ve been poking at a fun side project lately, exploring how to compress/uncompress arbitrary streams flowing between two sockets. I ended up with something that’s a little hacky, but surprisingly works quite well.

Introducing Boildown.

From a remote location (usually from work or on the road), I SSH home quite regularly and port-forward to several services behind NAT on my home network: SSH, remote desktop, web-cams, etc. I was curious to see if I could write something general that compresses traffic flowing between two sockets in an attempt to improve the overall “remote experience”. That is, compress the bidirectional traffic flowing over-the-wire to see if I could make things “faster”.

Orthogonally, I kinda wanted an excuse to play with LZF and Snappy.

How it works

Boildown listens on a local port, compresses (or decompresses) incoming traffic, and forwards the result to its destination. It’s like SSH port-forwarding, but the bidirectional network traffic flowing through Boildown is automatically compressed or decompressed, depending on how it’s configured. In essence, Boildown provides a compressed “pipe” connecting two nodes on a network.

Boildown is entirely protocol agnostic — it knows nothing about the protocol of the data flowing through it, and works transparently with any protocol that can be expressed over TCP/IP. The most common being HTTP (port 80), HTTPS (port 443), and SSH (port 22). This was key for me, because I wanted to build something general — a tool that isn’t protocol or application specific, and an app I could just stick between a sender and a receiver on a network and (ideally) see some sort of performance benefit with compression.

And so, Boildown v1 supports the following framed or “block” codecs:

• ZLIB
• LZF — provided by ning/compress
• Google Snappy — provided by xerial/snappy-java

Usage

There’s two sides (or “modes”) to Boildown:

• Compressor — listens on a local port, compresses outgoing traffic, and forwards the compressed data to another host.
• Decompressor — listens on a local port, decompresses incoming traffic, and forwards the original (uncompressed) result to its destination.

Assuming you’d want to SSH to remote:22, here’s how you’d create a compressed pipe using Boildown for an SSH session between localhost:10022 and remote:22:

+--------- [localhost] ---------+                               +----------- [remote] ------------+
| --compress 10022:remote:10022 | <---- (compressed pipe) ----> | --decompress 10022:localhost:22 |
+-------------------------------+                               +---------------------------------+

A Boildown compressor listens at localhost:10022 and forwards compressed traffic to the decompressor listening at remote:10022. Any bytes received by the decompressor at remote:10022 are decompressed and forwarded to the SSH server daemon listening locally on localhost:22. Of course, traffic flowing the other way, remote:22 back to localhost:10022, is compressed and decompressed in the same way.

Hence, a bidirectional, compressed network pipe.

On localhost

Start a compressor on localhost:10022, forwarding compressed traffic to remote:10022:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --compress 10022:remote:10022 --zlib

On remote

Start a decompressor on remote:10022, forwarding decompressed traffic to localhost:22:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --decompress 10022:localhost:22 --zlib

Connect-the-dots

On localhost, start a new SSH session, funneling traffic through the Boildown managed compressed pipe:

ssh -p 10022 localhost

Compression codecs

Specify --zlib, --snappy, or --lzf on the command line to use any of the 3 supported compression codecs.

Note, both sides of the pipe need to be using the same codec (obviously).

Thread pool

The compressor and decompressor implementations run within threads. The size of the internal thread pool used by Boildown can be controlled with the --poolSize N argument, where N is the maximum number of desired threads in the pool.

By default, if --poolSize is omitted, the internal thread pool is sized to match the number of available cores.

On-the-wire

Seeing what’s happening on-the-wire, over the Boildown compressed pipe, is quite easy with nc (netcat), telnet and tcpdump.

Spin up a compressor listening at localhost:20000 that forwards compressed traffic to localhost:30000:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --compress 20000:localhost:30000 --zlib &

Spin up a decompressor listening at localhost:30000 that forwards uncompressed traffic back to localhost:30001:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --decompress 30000:localhost:30001 --zlib &

In a separate terminal, spin up an instance of tcpdump that dumps traffic on port 30000. On Mac OS X:

sudo /usr/sbin/tcpdump -i lo0 -nnvvXXSs 1514 port 30000

In another terminal, launch nc to open up a socket and listen on port 30001 (where the decompressed/original bytes will be forwarded to):

nc -l 30001

And finally, in yet another terminal window, launch telnet and connect to localhost:20000:

telnet localhost 20000

Magic

Click to enlarge.

In the left panel, we’re using telnet to connect to the Boildown compressor listening at localhost:20000. Anything typed into this telnet session is routed through Boildown, compressed, and forwarded to localhost:30000.

The middle panel, we’re running nc which is listening at localhost:30001. This is the decompressed side. Anything from the telnet session at localhost:20000 is seen here, and consequently, anything we type into this session is forwarded (and compressed) back to localhost:20000.

In the right panel, notice the bidirectional compressed traffic captured by tcpdump flowing over localhost:30000. The astute reader will notice the Z? header in the tcpdump output given we’re running Boildown with --zlib.

Next steps

• Java NIO — eventually I want to explore how to use Java’s non-blocking I/O paradigm in lieu of threads to manage data flowing over-the-wire, similar to Jetty’s NIO org.eclipse.jetty.server.ServerConnector.
• Specify Multiple Compressors/Decompressors — as of now you can only specify a single --compress or --decompress route on the command line, but I’d eventually like to rework the app to support an arbitrary number of routes similiar to SSH’s -L.

Open Source

Boildown is free on GitHub and licensed under the popular MIT License.

Issues and pull requests welcome.

I’ve been getting back into Maven lately, converting the build system behind several of my personal projects on GitHub into something a little more sane and well-travelled. For reasons yet-to-be formally discussed, I’ve embarked on a mass migration away from SBT — albeit, I still have a number of published projects backed by SBT.

SBT Rant

Although I’m still using it sparingly, SBT has left a bitter taste in my mouth. The long-and-short of it is that I’m tired of everything in SBT being a complicated puzzle backed by poor documentation — I just want to get stuff done. I wish I had the countless hours of my life back that I spent figuring out how to accomplish very specific (yet seemingly common) tasks with SBT. Build definitions do not need to be written in Turing-complete languages, and in my humble opinion, SBT is a perfect example of what not to do.

</rant>

Maven

I was refactoring a personal project to use Maven the other day, and stumbled across a need to “add a local JAR to my classpath”. That is, I have a .jar file on disk from many moons ago, that is not in any public Maven repository yet I need to add it to the compile scope of my project.

Bad: The system Scope

A quick search of the Interwebz clearly calls out a worst practice: using the Maven system scope.

The system scope was designed to deal with “system” related files — files sitting in some fixed location, like Java’s core rt.jar. To discourage bad behavior, the Maven contributors intentionally refused to make pathname expansion work correctly in the context of the <systemPath> tag in the system scope. In other words, ${basedir}/lib/foo.jar below will not resolve:

<dependencies>
    <!-- WRONG: DON'T DO THIS -->
    <dependency>
        <groupId>com.foo</groupId>
        <artifactId>bar</artifactId>
        <version>1.0</version>
        <scope>system</scope>
        <systemPath>${basedir}/lib/bar-1.0.jar</systemPath>
    </dependency>
</dependencies>

Don’t do this.

Good: Use a Local Repository

The best practice is to “publish” the .jar file to a local Maven repository nested within the project. Yes, you read that correctly, publish the .jar to a ~/.m2 like repo within your project that is checked into SCM!

Here’s how…

On disk, your project probably looks something like this:

project/
  src/main/java
  src/main/resources
  src/test/java
  pom.xml

1) Create a lib directory in your project root — this lib directory will act as a local Maven repository within the project.

cd project
mkdir lib

2) Download the .jar file to disk, and use mvn to publish the .jar to the lib directory.

In this example, I’m publishing the Gagawa library I wrote and open-sourced many years ago.

mvn org.apache.maven.plugins:maven-install-plugin:2.5.2:install-file \
  -Dfile=~/Desktop/gagawa-1.0.1.jar \
  -DgroupId=com.hp \
  -DartifactId=gagawa \
  -Dversion=1.0.1 \
  -Dpackaging=jar \
  -DlocalRepositoryPath=lib

If all went well, you can find your artifact published inside of lib.

project$ find lib
lib
lib/com
lib/com/hp
lib/com/hp/gagawa
lib/com/hp/gagawa/1.0.1
lib/com/hp/gagawa/1.0.1/gagawa-1.0.1.jar
lib/com/hp/gagawa/1.0.1/gagawa-1.0.1.pom
lib/com/hp/gagawa/maven-metadata-local.xml

Note the structure here mimics what you’d find in ~/.m2.

3) Now, in your pom.xml, declare the lib directory in your project a Maven repository.

<repositories>
    <repository>
        <id>my-local-repo</id>
        <url>file://${basedir}/lib</url>
    </repository>
</repositories>

4) And lastly, in your pom.xml declare a dependency on the local .jar like you would for any other classpath dependency.

<dependencies>
    <dependency>
        <groupId>com.hp.gagawa</groupId>
        <artifactId>gagawa</artifactId>
        <version>1.0.1</version>
    </dependency>
</dependencies>

At runtime, Maven will consult the local repo at ${basedir}/lib in addition to ~/.m2 and any other remote repositories you have defined.

Ship it!

I hate seeing compiler warnings in code, and anyone who argues that ignoring them is a fine software engineering policy, should be swiftly relieved of their position. Warnings call out mistakes, and occasional blatant stupidity that should not be ignored — heck just Google “pay attention to compiler warnings” for some fun anecdotes. Folks in software are generally helpful, and compiler writers don’t inject annoying warnings because they are mean-spirited. Instead, people want to help, and consequently, compiler warnings are there to help. I’ve personally worked on several stacks with literally thousands of compiler warnings peppered throughout the code — it’s miraculous that some of those applications worked at all.

To combat warning hell, I’ve made it a personal best practice to do two things:

1. In mature code bases, never introduce more warnings and never ignore existing warnings I happen to stumble across. Ever. If I see a warning in an area of code that I’m working on, I’ll clean it up — no excuses.
2. In new projects, starting from scratch, set my build tool to immediately fail the build on any warning. In other words, treat warnings as compilation errors!

The latter is surprisingly easy and very effective at forcibly setting a high bar before entropy can gain a foothold.

Here’s how, with a few popular build tools:

Ant

If you’re still using Ant, set a series of <compilerarg> tags in your <javac> tasks. Of course, this goes in your build.xml:

<javac srcdir="${src.dir}" destdir="${classes.dir}" classpathref="libraries">
  <compilerarg value="-Xlint:all"/>
  <compilerarg value="-Xlint:-processing"/>
  <compilerarg value="-Xlint:-serial"/>
  <compilerarg value="-Werror"/>
</javac> 

Maven

Using the maven-compiler-plugin add a few <compilerArgs> to your configuration within your pom.xml:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>3.3</version>
  <configuration>
    <source>1.8</source>
    <target>1.8</target>
    <compilerArgs>
      <arg>-Xlint:all</arg>
      <arg>-Xlint:-processing</arg>
      <arg>-Xlint:-serial</arg>
      <arg>-Werror</arg>
    </compilerArgs>
  </configuration>
</plugin>

Gradle

To fail the build on any compiler warning, in main source and in test source, set this in your build.gradle:

tasks.withType(JavaCompile) {
  options.compilerArgs << "-Xlint:all" << "-Xlint:-processing" << "-Xlint:-serial" << "-Werror"
}

SBT

In the unlikely event that you’re building a pure Java project or Java source with SBT, set this in your project/Build.scala:

lazy val projectSettings = Defaults.coreDefaultSettings ++ Seq(
  scalacOptions ++= Seq(
    "-deprecation", "-unchecked", "-feature", "-Xlint", "-Xfatal-warnings", "-encoding", "utf8"
  ),
  javacOptions ++= Seq(
    "-Xlint:all,-processing,-serial", "-Werror", "-encoding", "utf8", "-g"
  )
)

The Scala compiler scalac equivalent to -Werror is -Xfatal-warnings (apparently).

A few notes

The magic is in -Werror which is documented here. When set, -Werror terminates compilation when warnings are found.

I’m also passing -Xlint:-processing which disables any annotation processor warnings from JARs on the compile classpath. And lastly, -Xlint:-serial disables any warnings complaining of Serializable classes that do not have an explicit serialVersionUID field. Ok, yes, certainly one could argue that ignoring complaints about a missing serialVersionUID field is dangerous, but I’ll let you be the judge.

Cheers!

With summer looming on the horizon and the California drought becoming an ever increasing problem, water conservation is undoubtedly a concern if not a necessity. Faced with a mandatory 25% water consumption reduction imposed by the State of California, my wife and I decided to seriously look at how we could further reduce our water usage — and for the water we must use, how we could reuse as much as possible where it made sense. Unfortunately, as a couple who already self-imposed deep water restrictions in mid-2014, the State’s latest mandatory 25% reduction on top of what we were already doing to conserve made the government’s ask feel all the more egregious.

And so, here we are.

Looking around the house for ways to even further conserve and reuse water, we stumbled across the ole’ washing machine. A classic Kenmore 500, this bad boy is clearly a big consumer of household water — fill up, wash, rinse, repeat. Studying the machine and its operation, my wife and I noticed just how much relatively clean greywater we were pumping down the drain during each wash cycle.

The light bulbs above both of our heads went on almost in unison — lets capture that water for the yard!

A quick trip to the hardware store, and bingo: ten minutes and $11 later we had the parts needed to send our washing machine greywater water into a garden hose.

Parts

Your washing machine, and garden hose, will almost certainly be different so you should get a tape measure and identify the right drain hose dimensions for your washing machine.

Here’s what we used:

1. Flexible Rubber Pipe Coupler with Clamps
   
   
   
2. PVC Schedule 40, Slip x NPT Female Adapter
   
   
   
3. PVC Schedule 40, NPT Male x MHT Male Adapter
   
   
   
   
4. Garden Hose
   
   
   

Assembly

With two hands, a wrench, and a gentle but firm touch you should be able to assemble these parts like so:

Some PVC glue for the slip to NPT adapter, and vinyl plumbers tape for the threads is good too, if you’re into that sort of thing.

Winning

And there you have it: a relatively cheap and clever solution to capture greywater water from your washing machine to be used in/around your yard.

Detergent

We did some digging and from what we can tell, mild detergents are not a problem for most ornamental plants and trees — of course, environmentally friendly detergents that degrade easily are best. And, it should go without saying, but chlorine bleach/borax is harmful for any living thing so you should avoid draining the water from bleached loads onto your yard.

Our plants love it — my wife and I have been watering our yard using this method for several weeks now, and thus far, everything is healthy. Despite our fears, not a single plant has keeled over and died (yet) being watered with washing machine greywater.

Caveats

1. I take no responsibility and hold no liability if, for whatever reason, you break your washing machine or kill the plants in your yard using greywater from your washing machine. Do your own homework, any mistakes are on you.
2. Some cities/municipalities have laws regarding greywater usage in/around homes. Before you try any of this, you should consult with your city officials to make sure it’s OK to water your yard with greywater.
3. It should go without saying, but the water that comes out of your washing machine is non-potable. Meaning you cannot, and should not, drink it. Keep thirsty pets and animals away too.
4. Don’t try to store greywater — in a 50-gallon drum, garbage can, etc. It’s gross. Apparently if you want to store greywater, there are advanced systems you can buy that will let you do exactly that. Storing untreated greywater from a washing machine in a 50-gallon drum is just asking for trouble.

Happy watering!