SBT: Recursive sbt.IO.listFiles

96b0fbe1fdf322ad86c2349d52e94d20329f15d6

Thu 26 Jun 2014 11:31:04 -0800

Annoyingly, SBT’s very own sbt.IO util Object doesn’t provide a mechanism to recursively list files in a directory.

As of SBT 0.13.5, the three listFiles functions it does implement are only somewhat useful for complex builds.

  • def listFiles(dir: File): Array[File]
  • def listFiles(dir: File, filter: java.io.FileFilter): Array[File]
  • def listFiles(filter: java.io.FileFilter)(dir: File): Array[File]

Meh.

Perhaps more frustrating is that sbt.IO is an Object (a singleton) which by its very nature in Scala means it cannot be extended. So, even if I wanted to extend sbt.IO and override to make it recursive, I can’t.

So, here’s how one can recursively list files in a directory leveraging SBT’s sbt.IO.listFiles:

trait IOHelpers {
  def listFilesRecursively(dir: File): Seq[File] = {
    val list = IO.listFiles(dir)
    list.filter(_.isFile) ++ list.filter(_.isDirectory).flatMap(listFilesRecursively)
  }
}

Functional programming for-the-win!

Cheers.

Manually Throttle the Bandwidth of a Linux Network Interface

ccf6dea1113d4479376e643cc0ddffac20715c35

Sun 08 Jun 2014 11:52:32 -0800

In complex service oriented application stacks, some bugs only manifest themselves on congested or slow networking interfaces. Consider a web-service running on a generic Linux box with a single networking interface, eth0. If eth0 is busy enough to completely saturate its networking link, a web-service running on the host behind that link may experience odd behavior when things “slowdown”.

For instance, established client connections timeout but the service fails to gracefully cleanup after itself leaving these connections open — this is a classic source of connection leaks, which on the JVM usually results in the dreaded IOException: Too many open files problem.

So, in development, if one wants to see how a service behaves behind a slow networking interface with extra latency:

  • Download large files in a loop to artificially saturate your networking link
  • Or, more appropriately, figure out how to shape networking traffic on an interface of your choice

A quick search for “how to artificially slow down a Linux networking interface” produced a number of interesting results. Folks mostly discussed 3rd party tools like Wondershaper and Dummynet. Other suggestions involved proxying all HTTP/HTTPS traffic through Apache’s mod_bw — yuck!

Fortunately, most Linux distros ship with the tc command which is used to configure Traffic Control in the Linux kernel.

On my Ubuntu 12.04 box I’ve got a single gigabit networking interface, eth0.

Let’s slow ’er down!

Add latency, slowing ping times

Without throttling, ping times to another local node on my home network are less than 0.2ms on average.

[mark@ubuntu]~$ ping regatta
PING regatta.kolich.local (1.0.0.2) 56(84) bytes of data.
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=1 ttl=64 time=0.118 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=2 ttl=64 time=0.193 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=3 ttl=64 time=0.181 ms

So, lets use tc to add 500ms of latency to all network traffic.

[mark@ubuntu]~$ sudo tc qdisc add dev eth0 root netem delay 500ms

Now, trying ping again note time=500 ms as desired.

[mark@ubuntu]~$ ping regatta
PING regatta.kolich.local (1.0.0.2) 56(84) bytes of data.
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=1 ttl=64 time=500 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=2 ttl=64 time=500 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=3 ttl=64 time=500 ms

Using tc we’ve added a delay of 500ms to all traffic. This will slow short connections, but once a connection gets past the TCP Slow-start window we’re back to full speed. That is, the connection may start slow — as shaped by our tc delay tweak — but once things are started TCP will ramp up and eventually hit full speed again.

Throttling a sustained maximum rate

So, let’s configure a sustained maximum rate using tc. In other words, lets configure Linux to never allow eth0 to use more than 1kbps regardless of port or application.

[mark@ubuntu]~$ sudo tc qdisc add dev eth0 handle 1: root htb default 11
[mark@ubuntu]~$ sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 1kbps
[mark@ubuntu]~$ sudo tc class add dev eth0 parent 1:1 classid 1:11 htb rate 1kbps

Looks good, now lets download a large .iso file using wget to prove to ourselves that our sustained maximum rate throttling is actually working.

[mark@ubuntu]~$ wget http://mirrors.kernel.org/.../CentOS-6.5-x86_64-bin-DVD1.iso -O /dev/null
HTTP request sent, awaiting response... 200 OK
Length: 4467982336 (4.2G) [application/octet-stream]
Saving to: `/dev/null'
 13% [==>                                   ] 580,837,703     10.5K/s

Note the download isn’t going to hover exactly at 1.0K/sec — the actual download speed as reported by wget is an average over time. In short, you’ll see numbers closer to an even 1.0K/sec the longer the transfer. In this example, I didn’t wait to download an entire 4.2GB file, so the 10.5K/s you see above is just wget averaging the transfer speed over the short time I left wget running.

Clearing all tc rules

Now that we’re done, simply delete all traffic control throttling rules to return to normal.

[mark@ubuntu]~$ sudo tc qdisc del dev eth0 root

Cheers!

Bolt: A Wrapper around Java's ReentrantReadWriteLock

09df17fb2d779355c5a8121409741543bd982ef4

Thu 27 Feb 2014 20:49:40 -0800

Concurrency is difficult, and generally tough to get right. Fortunately, there are tools that can somewhat ease this pain. For instance, take Java’s ReentrantReadWriteLock — a useful and foundational class that helps any highly concurrent Java application manage a set of readers and writers that need to access a critical block of code. When using a ReentrantReadWriteLock you can have any number of simultaneous readers, but the write lock is exclusive. In other words:

  • If any thread holds the write lock, all readers are forced to wait (or fail hard) until the thread that holds the write lock releases the lock.
  • If the write lock is not held, any number of readers are allowed to access the protected critical block concurrently — and any incoming writers are forced to wait (or fail hard) until all readers are done.

In short, this is the classic ReadWriteLock paradigm.

This is great, except that a vanilla ReentrantReadWriteLock is missing a few key features:

  1. Conditionally wait, or fail immediately, if the desired lock is not available. In other words, let me define upfront what I want to do if the lock I want to “grab” is not available — fail now, or wait indefinitely?
  2. And, execute a callback function only upon successful execution of a transaction. Here, we define a transaction to mean successfully acquiring the lock, doing work (without failure), and releasing the lock.

I wanted these features, so I implemented Bolt — a very tiny wrapper around Java’s ReentrantReadWriteLock with better wait, cleaner fail, and transactional callback support.

LockableEntity

Using Bolt, any entity or object you want to protect should implement the LockableEntity interface.

import com.kolich.bolt.LockableEntity;
import java.util.concurrent.locks.ReadWriteLock;

public final class Foobar implements LockableEntity {

  private final ReadWriteLock lock_;

  public Foobar() {
    lock_ = new ReadWriteLock();
  }

  @Override
  public ReadWriteLock getLock() {
    return lock_;
  }

}

Now, let’s create an instance of this example entity which we will use to protect a critical section of code within a transaction.

public static final Foobar foo = new Foobar();

This instance, foo, is used below throughout my examples.

Read Lock, Fail Immediately

First, let’s grab a shared read lock on foo, but fail immediately with a LockConflictException if the write lock is already acquired by another thread.

new ReentrantReadWriteEntityLock<T>(foo) {
  @Override
  public T transaction() throws Exception {
    // ... do read only work.
    return baz;
  }
}.read(false); // Fail immediately if read lock is not available

Note that read asks for a shared reader lock — the lock will be granted if and only if there are no threads holding a write lock on foo. There very well may be other reader threads.

Read Lock, Block/Wait Forever

Next, let’s grab a shared read lock on foo, but block/wait forever for the read lock to become available. Execute the success callback if and only if the transaction method finished cleanly without exception.

Note the implementation of the success method is completely optional.

new ReentrantReadWriteEntityLock<T>(foo) {
  @Override
  public T transaction() throws Exception {
    // ... do read only work.
    return baz;
  }
  @Override
  public T success(final T t) throws Exception {
    // Only called if transaction() finished cleanly without exception
    return t;
  }
}.read(); // Wait forever

It is very important to note that the underlying lock is held, while the success method is called. That is, the acquired lock isn’t released until the transaction and success method are finished.

Write Lock, Fail Immediately

Grab an exclusive write lock on foo, or fail immediately with a LockConflictException if a write or read lock is already acquired by another thread. Further, execute the success callback method if and only if the transaction method finished cleanly without exception.

new ReentrantReadWriteEntityLock<T>(foo) {
  @Override
  public T transaction() throws Exception {
    // ... do read or write work, safely.
    return baz;
  }
  @Override
  public T success(final T t) throws Exception {
    // Only called if transaction() finished cleanly without exception
    return t;
  }
}.write(); // Fail immediately if write lock not available

Write Lock, Block/Wait Forever

Grab an exclusive write lock on foo, or block/wait forever for all readers to finish.

new ReentrantReadWriteEntityLock<T>(foo) {
  @Override
  public T transaction() throws Exception {
    // ... do read or write work, safely.
    return baz;
  }
}.write(true); // Wait forever

An Example

The Havalo project makes extensive real-world use of this locking mechanism, as a way to manage shared entities that may be concurrently accessed by any number of threads. Havalo is a lightweight key-value store written in Java. Internally, it maintains a collection of repositories and objects, and uses Bolt to conditionally gate access to these objects in local memory.

GitHub

Bolt is free, and open source, on GitHub:

https://github.com/markkolich/kolich-bolt

Pull requests welcome.

Better HTTP Request Body Handling with Curacao

08f781f224841a49a977716374ef56be8bb562ae

Wed 19 Feb 2014 19:47:11 -0800

When discussing HTTP, standard conjecture these days is that the server side is obligated to read and buffer request bodies in their entirety. Although not formally required, many antiquated server side frameworks will synchronously read and buffer POST and PUT bodies into local memory (up to some configurable maximum) before invoking the application layer business logic.

You should note that just because an HTTP client wants to send a request body, does not mean the server side is required to read it. Furthermore, it’s especially bad form if your server side web-layer blindly buffers a POST or PUT body into memory, regardless of whether or not the underlying application business logic actually needs it.

But Someone Has to Read the Bytes!

Actually, no.

For a moment, imagine I’m the server and you’re the client. To initiate an HTTP transaction with me, you open a socket bound to some port, and send me a request. Your request headers on-the-wire might look something like this:

POST /foobar HTTP/1.1
Host: localhost
Content-Type: application/x-www-form-urlencoded
Content-Length: 1073741824

As the server, once I’ve received your request headers, I’m now tasked with deciding what I want to do. Note, at this point, I haven’t read any data from you other than the headers even though you’ve said you want to send me a 1GB payload (as specified by your Content-Length header).

Let’s then say I decide that 1073741824 bytes is just too large, and it exceeds the amount of data I’m willing to process. As such, without reading any data from you, I immediately respond with this:

HTTP/1.1 413 Request Entity Too Large
Content-Length: 0
Connection: close

In other words, I received your request, but you want to send me too much data. Without reading anything else from you on-the-wire, I rejected your request. So you see, on my end I may have had a pointer to an open InputStream from which I could have consumed to read your request body, but I chose not to. In short, this means that you never sent your 1GB payload over-the-wire — I determined that I didn’t want to read it, and therefore, no data was sent.

This is completely legal.

Bad

In general, the worst way to approach this on the server-side is to blindly buffer the entire request body into memory, regardless of its size and whether the application layer needs the body or not. This is essentially playing Russian roulette with your application stack. You’re going to try and buffer the entire request body into memory, regardless of its size and whether or not it’s needed, and hope you don’t fall over in the process.

In Java, assuming you have an open InputStream representing a pointer to the bytes on-the-wire:

import org.apache.commons.io.IOUtils;

final InputStream in = getInputStream();
final ByteArrayOutputStream out = new ByteArrayOutputStream();

IOUtils.copyLarge(in, out); // Ouch

This JVM is guaranteed to fall over if its heap size is N bytes, and it receives a single request whose Content-Length is just N+1 bytes. It will likely fall over with much less than N+1 bytes well before an excessively large request is ever received, given other classes and resources are already loaded and held in local memory.

Better

A slightly improved approach to this problem is to buffer the request body up to a fixed size. Again, this is buffering the request body in memory whether it’s needed or not by the application layer — that is, on any POST or PUT request that contains a Content-Length, it will be buffered locally up to some configured maximum.

import org.apache.commons.io.IOUtils;
import com.google.common.io.ByteStreams; // From Google Guava

final InputStream in = getInputStream();

// Wrap input, limit to 1MB
final InputStream wrapped = ByteStreams.limit(in, 1048576L);

final ByteArrayOutputStream out = new ByteArrayOutputStream();
IOUtils.copyLarge(wrapped, out);

This is a bit better — assuming you limit each request processing thread in your application to only buffer 1MB, you’ll use at most N * K bytes where N is the maximum number of request bytes to buffer per thread and K is the total number of request handler threads.

Now, with a fixed ceiling, we can tune our heap size and number of request threads to be sure that request bodies will never consume enough of the heap to cause our JVM to crash.

Best

The right way to approach this problem is twofold. Only buffer request bodies:

  • up to a configurable maximum size
  • and, only if needed by the application layer

In theory, any POST or PUT request can contain a request body with some Content-Length. But, that doesn’t mean the web-layer has to read it.

Consider the case where a POST request is used to toggle a virtual ON/OFF switch.

POST /switch/toggle HTTP/1.1
Host: localhost
Content-Type: application/x-www-form-urlencoded
Content-Length: 1073741824

In this instance, everything the application layer needs to make a decision and complete the operation is already provided (hint: it’s in the URI /switch/toggle). The excessive payload you see in the Content-Length here can be completely ignored — there’s no reason any web-layer should attempt to read and locally buffer this 1GB payload.

Curacao

If using an intelligent web-layer toolkit like Curacao, addressing this problem is trivial. That is, your application doesn’t have to manage decisions around whether or not to buffer the request body, Curacao will do it for you as appropriate.

First, in your application.conf configuration file, define a reasonable value for curacao.mappers.request.max-request-body-size:

curacao {
  mappers.request.max-request-body-size = 1m
}

This will instruct Curacao to only ever buffer 1MB of the request body into memory, as needed. If unspecified, the default is 512KB.

Next, define a controller:

@Controller
public final class SwitchController {

  @POST("/switch/toggle")
  public final int toggle() {
    // ... toggle switch, return raw status code.
    return 200;
  }

  @POST("/switch/set")
  public final String set(@RequestBody final String op) {
    final String result;
    if("ON".equals(op)) {
      // ... turn on.
      result = "ON";
    } else if("OFF".equals(op)) {
      // ... turn off.
      result = "OFF";
    } else {
      // ... unknown state, no-op.
      result = "UNKNOWN";
    }
    return result;
  }

}

This example controller defines two methods.

  1. First, the toggle() method will be called on receipt of a POST:/switch/toggle request. Note the toggle() method has no arguments — when Curacao processes the POST request, no request body will be read from the client because the application layer is not asking for it.

  2. Second, the set() method will be called on receipt of a POST:/switch/set request. You’ll note the set() method has a single argument: @RequestBody final String op. This is Curacao’s cue to read and buffer the POST request body into memory, up to the configured maximum of 1MB.

But wait, we can make this even better!

We know that the set() method is built to only ever care about values 3-bytes or less: ON or OFF. Knowing this, we can further tighten the processing of the request body. For instance, we can pass a maxSizeInBytes argument into the @RequestBody annotation, to further limit the number of bytes Curacao will attempt to read before invoking this Java method:

@POST("/switch/set")
public final String set(@RequestBody(maxSizeInBytes=3L) final String op) {
  // ...
}

Here, instead of buffering up to the globally configured maximum of 1MB, just for this one request mapping Curacao will only accept a Content-Length that is 3-bytes or less. In the event that a client omits a Content-Length request header, Curacao will fallback gracefully and only read at most 3-bytes, then stop.

Pretty neat, eh?

Curacao is free, and open source, on GitHub:

https://github.com/markkolich/curacao

Pull requests welcome.