Content tagged: scala

Fail your Build on Java Compiler Warnings

2c8127e956e463d7dd061b870cc9fce3d256f974

Sat Sep 12 16:37:41 2015 -0700

I hate seeing compiler warnings in code, and anyone who argues that ignoring them is a fine software engineering policy, should be swiftly relieved of their position. Warnings call out mistakes, and occasional blatant stupidity that should not be ignored — heck just Google “pay attention to compiler warnings” for some fun anecdotes. Folks in software are generally helpful, and compiler writers don’t inject annoying warnings because they are mean-spirited. Instead, people want to help, and consequently, compiler warnings are there to help. I’ve personally worked on several stacks with literally thousands of compiler warnings peppered throughout the code — it’s miraculous that some of those applications worked at all.

To combat warning hell, I’ve made it a personal best practice to do two things:

  1. In mature code bases, never introduce more warnings and never ignore existing warnings I happen to stumble across. Ever. If I see a warning in an area of code that I’m working on, I’ll clean it up — no excuses.
  2. In new projects, starting from scratch, set my build tool to immediately fail the build on any warning. In other words, treat warnings as compilation errors!

The latter is surprisingly easy and very effective at forcibly setting a high bar before entropy can gain a foothold.

Here’s how, with a few popular build tools:

Ant

If you’re still using Ant, set a series of <compilerarg> tags in your <javac> tasks. Of course, this goes in your build.xml:

<javac srcdir="${src.dir}" destdir="${classes.dir}" classpathref="libraries">
  <compilerarg value="-Xlint:all"/>
  <compilerarg value="-Xlint:-processing"/>
  <compilerarg value="-Xlint:-serial"/>
  <compilerarg value="-Werror"/>
</javac> 

Maven

Using the maven-compiler-plugin add a few <compilerArgs> to your configuration within your pom.xml:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-compiler-plugin</artifactId>
  <version>3.3</version>
  <configuration>
    <source>1.8</source>
    <target>1.8</target>
    <compilerArgs>
      <arg>-Xlint:all</arg>
      <arg>-Xlint:-processing</arg>
      <arg>-Xlint:-serial</arg>
      <arg>-Werror</arg>
    </compilerArgs>
  </configuration>
</plugin>

Gradle

To fail the build on any compiler warning, in main source and in test source, set this in your build.gradle:

tasks.withType(JavaCompile) {
  options.compilerArgs << "-Xlint:all" << "-Xlint:-processing" << "-Xlint:-serial" << "-Werror"
}

SBT

In the unlikely event that you’re building a pure Java project or Java source with SBT, set this in your project/Build.scala:

lazy val projectSettings = Defaults.coreDefaultSettings ++ Seq(
  scalacOptions ++= Seq(
    "-deprecation", "-unchecked", "-feature", "-Xlint", "-Xfatal-warnings", "-encoding", "utf8"
  ),
  javacOptions ++= Seq(
    "-Xlint:all,-processing,-serial", "-Werror", "-encoding", "utf8", "-g"
  )
)

The Scala compiler scalac equivalent to -Werror is -Xfatal-warnings (apparently).

A few notes

The magic is in -Werror which is documented here. When set, -Werror terminates compilation when warnings are found.

I’m also passing -Xlint:-processing which disables any annotation processor warnings from JARs on the compile classpath. And lastly, -Xlint:-serial disables any warnings complaining of Serializable classes that do not have an explicit serialVersionUID field. Ok, yes, certainly one could argue that ignoring complaints about a missing serialVersionUID field is dangerous, but I’ll let you be the judge.

Cheers!

SBT: Recursive sbt.IO.listFiles

5e292bcf846dd291a57409813d677d0c43d1020d

Thu Jun 26 12:31:04 2014 -0700

Annoyingly, SBT’s very own sbt.IO util Object doesn’t provide a mechanism to recursively list files in a directory.

As of SBT 0.13.5, the three listFiles functions it does implement are only somewhat useful for complex builds.

  • def listFiles(dir: File): Array[File]
  • def listFiles(dir: File, filter: java.io.FileFilter): Array[File]
  • def listFiles(filter: java.io.FileFilter)(dir: File): Array[File]

Meh.

Perhaps more frustrating is that sbt.IO is an Object (a singleton) which by its very nature in Scala means it cannot be extended. So, even if I wanted to extend sbt.IO and override to make it recursive, I can’t.

So, here’s how one can recursively list files in a directory leveraging SBT’s sbt.IO.listFiles:

trait IOHelpers {
  def listFilesRecursively(dir: File): Seq[File] = {
    val list = IO.listFiles(dir)
    list.filter(_.isFile) ++ list.filter(_.isDirectory).flatMap(listFilesRecursively)
  }
}

Functional programming for-the-win!

Followup 2015

I submitted my idea for a recursive listFiles function as an enhancement request to the folks that maintain SBT:

https://github.com/sbt/sbt/issues/1789

It was quickly closed as “won’t fix” but they did point out that you can use sbt.Path (a.k.a., Paths) to achieve the same behavior without any hacks:

scala> import sbt._, Path._
import sbt._
import Path._

scala> val base = new java.io.File(".")
base: java.io.File = .

scala> (base ** (-DirectoryFilter)).get
res1: Seq[java.io.File] = ArrayBuffer(./.gitattributes,...

Cheers.

Thoughts on Using Raw java.util.UUID's in Your Web-Application or Web-Service: Check your UUID Length's Too

47dc940a4a615e48319720c7ee351344ba788395

Mon May 13 19:15:00 2013 -0700

You’re probably familiar with UUID’s — those ubiquitous universally unique identifiers used in just about every modern web-application or web-service. And, if you’re a developer living on the JVM, you’re probably close friends with java.util.UUID whether you like it or not.

Generally speaking, UUID’s are a convenient way to represent some unique object or entity inside of an application. After all, they’re supposed to be “universally unique” and random enough such that an application can, in theory, generate “random” UUID’s forever without any collisions. In other words, UUID’s are represented by a 128-bit number under-the-hood, so the total number of possible UUID’s is immense — 340,282,366,920,938,463,463,374,607,431,768,211,456 unique UUID’s to be exact.

No application today could possibly have more than 340,282,366,920,938,463,463,374,607,431,768,211,456 users, or need to store more than 340,282,366,920,938,463,463,374,607,431,768,211,456 objects, right?

So what’s the big deal?

In their canonical form, UUID’s are represented by 32 hexadecimal digits, displayed in five groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters. For example:

scala> import java.util.UUID

scala> UUID.randomUUID
res0: 09bf989f-5b24-47bc-871e-1e824d4f4c60

Again, note that UUID’s are typically represented by 32-hexadecimal digits, with a canoncial form string length of 36 (including the hyphens).

scala> UUID.randomUUID.toString.length
res1: Int = 36

And therein lies the rub.

Given that UUID’s are represented by a series of hexadecimal digits, it occurs to me that appending a long string of leading zeros, or even omitting a leading zero (if present), still results in a valid UUID. For example, 0x0000000A is equivalent to 0x0A, or even 0xA.

That said, these UUID’s are logically identical:

9bf989f-5b24-47bc-871e-1e824d4f4c60
09bf989f-5b24-47bc-871e-1e824d4f4c60
00000000000000000000000000000009bf989f-5b24-47bc-871e-1e824d4f4c60

At least, according to java.util.UUID they are!

OK, so, what’s the problem?

Well, consider this: if you use UUID’s in the paths to resources in your web-service or web-application, you need to make sure your application (or the framework you’re using) does the right thing with egregiously long, or slightly short, UUID’s represented in a URI as a String.

For example, take this request:

GET:/api/object/{uuid}

Within the business logic, many web-applications (and frameworks) do something like the following:

val id = UUID.fromString({uuid})

In theory, this can lead to a number of wonderful exploits, including buffer overflow attacks and other awesome denial of service breakdowns in your web-service or web-application.

What’s a developer to do?

Well, the long and short of it is that at the end of the day, you also have to check the length of incoming UUID’s that you plan to “do something useful” with in your application. If the incoming UUID is longer than 36-characters or shorter than 36-characters, you’re wasting your time.

So, here’s a quick regular expression that does the right thing as far as “syntactically” correct UUID’s go:

^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$

And now, we can use the Scala interactive interpreter to verify our new regular expression:

scala> val r =
     | "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$".r

scala> r.findFirstIn("000000000000000009bf989f-5b24-47bc-871e-1e824d4f4c60")
res0: Option[String] = None

scala> r.findFirstIn("09bf989f-5b24-47bc-871e-1e824d4f4c60")
res1: Option[String] = Some(09bf989f-5b24-47bc-871e-1e824d4f4c60)

Yay! Note that on the first call to findFirstIn, a None (no match) was returned. On the second invocation with a UUID of the correct length, Some(uuid) was returned given the input String was syntactically correct and of a perfect length.

So, in the end, not a huge deal but it’s good to keep in mind that when dealing with UUID’s you cannot rely on java.util.UUID alone to parse and verify an incoming identifier. In the end, you’ve got to use your own UUID verification regular expression. Or, better yet, use the verification mechanisms provided by your web-service or web-application framework (if one exists) to verify the length of incoming UUID’s.

Enjoy.