Content tagged: security

Thoughts on Using Raw java.util.UUID's in Your Web-Application or Web-Service: Check your UUID Length's Too


Mon May 13 19:15:00 2013 -0700

You’re probably familiar with UUID’s — those ubiquitous universally unique identifiers used in just about every modern web-application or web-service. And, if you’re a developer living on the JVM, you’re probably close friends with java.util.UUID whether you like it or not.

Generally speaking, UUID’s are a convenient way to represent some unique object or entity inside of an application. After all, they’re supposed to be “universally unique” and random enough such that an application can, in theory, generate “random” UUID’s forever without any collisions. In other words, UUID’s are represented by a 128-bit number under-the-hood, so the total number of possible UUID’s is immense — 340,282,366,920,938,463,463,374,607,431,768,211,456 unique UUID’s to be exact.

No application today could possibly have more than 340,282,366,920,938,463,463,374,607,431,768,211,456 users, or need to store more than 340,282,366,920,938,463,463,374,607,431,768,211,456 objects, right?

So what’s the big deal?

In their canonical form, UUID’s are represented by 32 hexadecimal digits, displayed in five groups separated by hyphens, in the form 8-4-4-4-12 for a total of 36 characters. For example:

scala> import java.util.UUID

scala> UUID.randomUUID
res0: 09bf989f-5b24-47bc-871e-1e824d4f4c60

Again, note that UUID’s are typically represented by 32-hexadecimal digits, with a canoncial form string length of 36 (including the hyphens).

scala> UUID.randomUUID.toString.length
res1: Int = 36

And therein lies the rub.

Given that UUID’s are represented by a series of hexadecimal digits, it occurs to me that appending a long string of leading zeros, or even omitting a leading zero (if present), still results in a valid UUID. For example, 0x0000000A is equivalent to 0x0A, or even 0xA.

That said, these UUID’s are logically identical:


At least, according to java.util.UUID they are!

OK, so, what’s the problem?

Well, consider this: if you use UUID’s in the paths to resources in your web-service or web-application, you need to make sure your application (or the framework you’re using) does the right thing with egregiously long, or slightly short, UUID’s represented in a URI as a String.

For example, take this request:


Within the business logic, many web-applications (and frameworks) do something like the following:

val id = UUID.fromString({uuid})

In theory, this can lead to a number of wonderful exploits, including buffer overflow attacks and other awesome denial of service breakdowns in your web-service or web-application.

What’s a developer to do?

Well, the long and short of it is that at the end of the day, you also have to check the length of incoming UUID’s that you plan to “do something useful” with in your application. If the incoming UUID is longer than 36-characters or shorter than 36-characters, you’re wasting your time.

So, here’s a quick regular expression that does the right thing as far as “syntactically” correct UUID’s go:


And now, we can use the Scala interactive interpreter to verify our new regular expression:

scala> val r =
     | "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$".r

scala> r.findFirstIn("000000000000000009bf989f-5b24-47bc-871e-1e824d4f4c60")
res0: Option[String] = None

scala> r.findFirstIn("09bf989f-5b24-47bc-871e-1e824d4f4c60")
res1: Option[String] = Some(09bf989f-5b24-47bc-871e-1e824d4f4c60)

Yay! Note that on the first call to findFirstIn, a None (no match) was returned. On the second invocation with a UUID of the correct length, Some(uuid) was returned given the input String was syntactically correct and of a perfect length.

So, in the end, not a huge deal but it’s good to keep in mind that when dealing with UUID’s you cannot rely on java.util.UUID alone to parse and verify an incoming identifier. In the end, you’ve got to use your own UUID verification regular expression. Or, better yet, use the verification mechanisms provided by your web-service or web-application framework (if one exists) to verify the length of incoming UUID’s.


Configuring Apache to Tunnel SSH Through an HTTP Web-Proxy with Proxytunnel


Sat Dec 31 13:00:00 2011 -0800

Here’s the situation, I’m often on a network that does not allow outbound traffic on port 22. Meaning, I cannot directly “SSH out” from that network to my Linux box at home. Fair enough. However, this network does allow outbound traffic on ports 80, 443, and 8443 via a web-proxy. That said, if I want to “SSH out” from this network to my Linux box at home, I can do so with a little tweaking of my remote Apache server and my local SSH client.

Here’s how …


First, you’ll need to configure your Apache web-server to accept traffic on a port that’s acceptable to the web-proxy. In my case, I don’t have anything running on port 8443, and the web-proxy allows traffic through port 8443, so that’s perfect. Apache will be configured to listen on 8443, and act as a “proxy” between an SSH client and an SSH server (usually the SSH server running on the box you’re trying to connect to).

Second, on the client side, you’ll be using something like Proxytunnel to punch a hole through the web-proxy allowing your SSH client to connect to an SSH server of your choice.

Putting it all together, the basic flow is …

  1. Your local SSH client uses Proxytunnel to connect to
  2. connects to Apache running at yourwebserver:8443
  3. Your Apache server, acting as yet another proxy, connects to yoursshserver:22
  4. It works!

Install and Configure Proxytunnel

If you’re on Ubuntu, you can install Proxytunnel with the following command:

#/> sudo apt-get install proxytunnel

Once installed, edit your ~/.ssh/config file to instruct your SSH client to use Proxytunnel when connecting to the destination host:

## ~/.ssh/config

  ProtocolKeepAlives 30
  ProxyCommand /usr/bin/proxytunnel \
    -p \
    -r -d %h:%p \
    -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)"

In this example, when my SSH client makes an attempt to connect to, it will spawn Proxytunnel which will then route my connection through and on to This seems really convoluted, but it actually works quite well.

Note that I’m spoofing a somewhat real User-Agent to prevent suspicion from the system administrators running If you’re a system administrator that runs such a web-proxy, please accept my apologies for making your life even more difficult.

Configure mod_proxy on Apache

Now that you’ve got the client part figured out, you’ll need to configure Apache’s mod_proxy module to proxy traffic between yourwebserver:8443 and yoursshserver:22. In all likelihood, your web-server and SSH server are the same box. At least, in my home, they are.

Oh, and I assume you already have Apache and mod_proxy installed, and working. There are ton of other tutorials and nice blog posts online about how to install and setup Apache if you don’t already have it installed and functional.

In my Apache virtual host configuration, I’ve added another V-host listening on port 8443 that will only accept CONNECT requests bound for on port 22:

## Load the required modules.
LoadModule proxy_http_module modules/
LoadModule proxy_connect_module modules/

## Listen on port 8443 (in addition to other ports like 80 or 443)
Listen 8443

<VirtualHost *:8443>

  ServerName youwebserver:8443
  DocumentRoot /some/path/maybe/not/required

  ## Only ever allow incoming HTTP CONNECT requests.
  ## Explicitly deny other request types like GET, POST, etc.
  ## This tells Apache to return a 403 Forbidden if this virtual
  ## host receives anything other than an HTTP CONNECT.
  RewriteEngine On
  RewriteRule ^/(.*)$ - [F,L]

  ## Setup proxying between youwebserver:8443 and yoursshserver:22

  ProxyRequests On
  ProxyBadHeader Ignore
  ProxyVia Full

  ## IMPORTANT: The AllowCONNECT directive specifies a list
  ## of port numbers to which the proxy CONNECT method may
  ## connect.  For security, only allow CONNECT requests
  ## bound for port 22.
  AllowCONNECT 22

  ## IMPORTANT: By default, deny everyone.  If you don't do this
  ## others will be able to connect to port 22 on any host.
  <Proxy *>
    Order deny,allow
    Deny from all

  ## Now, only allow CONNECT requests bound for
  ## Should be replaced with or the hostname
  ## of whatever SSH server you're trying to connect to.  Note
  ## that ProxyMatch takes a regular expression, so you can do
  ## things like (kolich\.com|anothersshserver\.com) if you want
  ## to allow connections to multiple destinations.
  <ProxyMatch (kolich\.com)>
    Order allow,deny
    Allow from all

  ## Logging, always a good idea.
  LogLevel warn
  ErrorLog logs/yourwebserver-proxy_error_log
  CustomLog logs/yourwebserver-proxy_request_log combined


Once you get everything integrated, restart Apache and you should be golden.

Under the Hood

To prove that everything works, let’s try a few things.

First I’m going to telnet to Then, I’m going to tell it to connect to Finally, I’m going to tell Apache on to connect to This is exactly the same flow used by Proxytunnel under the hood.

(mark@ubuntu)~> telnet 3128
Connected to (
Escape character is '^]'.

HTTP/1.0 200 Connection Established


HTTP/1.0 200 Connection Established
Proxy-agent: Apache


Sweet! Notice the raw “SSH-2.0-OpenSSH_4.3” response from the SSH server, indicating a successful connection. Now, If I was a real SSH client, I’d continue the handshake and away we go.

So, from a real SSH client with Proxytunnel enabled …

(mark@ubuntu)~> ssh
Via -> ->'s password:

Last login: Sat Dec 31 12:53:22 2011 from gateway.kolich.local

It works! Notice the intermediate “Via -> ->” output from Proxytunnel telling me what it’s doing to connect. And of course, look at that beautiful shell prompt.

SSH through a web-proxy, I love it.


MySQL Triggers and SUPER Privileges: "Access denied; you need the SUPER privilege for this operation."


Thu May 27 13:20:00 2010 -0700

I just discovered that dealing with MySQL triggers, in many instances, is quite painful. For example, here’s a trigger that deletes a bunch of rows in a table on every INSERT:

delimiter |
CREATE TRIGGER delete_expired_tweets AFTER INSERT ON tweets
    DELETE FROM tweets WHERE DATEDIFF(NOW(), created_at) > 365;

delimiter ;

Ok, let’s load this trigger into MySQL:

#/> mysql -h myhost -u normaluser -p mydatabase
Enter password: **********
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 10
Server version: 5.0.41-community-nt MySQL Community Edition (GPL)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> source trigger.sql
ERROR 1227 (42000): Access denied; you need the SUPER privilege
    for this operation

Access denied? I dug into it, and confirmed that you can only add triggers if your user account has the SUPER privilege enabled. You’re probably thinking, “No kidding Sherlock, that’s what the error message says.” Yes, I know that’s what the error message says. But here’s the problem. Normal database users created using GRANT ALL PRIVILEGES ON database.* TO... will not have the SUPER privilege assigned to them by default. As described here, the SUPER privilege in MySQL let’s the account do some things that normal database users, in most environments, should not be able to do (like kill database threads, modify global system variables, etc.). As a result, it’s a very bad idea to grant the SUPER privilege to normal database users, even if they just need the SUPER privilege to load a trigger. You know better than that!

Even worse, suppose you GRANT SUPER PRIVILEGES to a single user, on a single database. Well, that still won’t be enough to load a trigger. Unfortunately, loading triggers requires SUPER PRIVILEGES at the global level (e.g., GRANT SUPER PRIVILEGES ON *.*). Again, it’s a very bad ideal to grant normal database users the SUPER privilege.

So how exactly am I supposed to load this trigger? Well as far as I can tell, assuming I refuse to give myself SUPER PRIVILEGES for the reasons I just explained, I have two options:

  1. Don’t use triggers, and find another way to cleanup rows in my table.
  2. Log into the database as root/admin and load the trigger on behalf of the normal user. If I wasn’t the owner of this database server, this would probably involve asking my database administrator to load the trigger for me.

Just one of many common annoyances with MySQL.

HTTP Digest Access Authentication using MD5 and HttpClient 4


Tue May 04 14:30:00 2010 -0700

Dealing with HTTP’s Digest authentication mechanism isn’t too bad once you have the basic building blocks in place. Luckily HttpClient 4 can automatically solve many types of authentication challenges for you, if used correctly. Using HttpClient 4, I built an app that authenticates against a SOAP based web-service requiring WWW-Authenticate Digest authentication. In a nutshell, the fundamental principal behind HTTP Digest authentication is simple:

  • The client asks for a page that requires authentication.
  • The server responds with an HTTP 401 response code, providing the authentication realm and a randomly-generated, single-use value called a “nonce”. The authentication “challenge” itself is encapsulated inside of the WWW-Authenticate HTTP response header.
  • The client “solves” the authentication challenge and a solution is sent back to the web-server via the HTTP Authorization header on a subsequent request. The solution usually contains some type of MD5 hashed mess of your username, password, and “nonce”.
  • Assuming the solution is acceptable the server responds with a successful type response, usually an HTTP 200 OK.

Here’s a sample with a bit of pseudo code mixed in (so, you get the idea):

// A org.apache.http.impl.auth.DigestScheme instance is
// what will process the challenge from the web-server
final DigestScheme md5Auth = new DigestScheme();

// This should return an HTTP 401 Unauthorized with
// a challenge to solve.
final HttpResponse authResponse = doPost(url, postBody, contentType);

// Validate that we got an HTTP 401 back
if(authResponse.getStatusLine().getStatusCode() == HttpStatus.SC_UNAUTHORIZED) {
  if(authResponse.containsHeader("WWW-Authenticate")) {
    // Get the challenge.
    final Header challenge = authResponse.getHeaders("WWW-Authenticate")[0];
    // Solve it.
    // Generate a solution Authentication header using your
    // username and password.
    final Header solution = md5Auth.authenticate(
      new UsernamePasswordCredentials(username, password),
      new BasicHttpRequest(HttpPost.METHOD_NAME,
          new URL(url).getPath()));
    // Do another POST, but this time include the solution
    // Authentication header as generated by HttpClient.
    final HttpResponse goodResponse =
      doPost(url, postBody, contentType, solution);
    // ... do something useful with goodResponse, which assuming
    // your credentials were valid, should contain the data you
    // requested.
  } else {
    throw new Error("Web-service responded with Http 401, " +
      "but didn't send us a usable WWW-Authenticate header.");
} else {
  throw new Error("Didn't get an Http 401 " +
    "like we were expecting.");


Setting Up Your Own SVN Server (Using Apache and mod_dav_svn)


Tue Mar 16 15:30:00 2010 -0700

Setting up your own SVN source control server is surprisingly easy. At home, I recently setup an SVN server in a CentOS 5.4 virtual machine with Apache 2.2 and mod_dav_svn. With a little work, I had a secure and fully functional SVN server up and running in about 20 minutes.

Note that this HOWTO is specific to CentOS/RHEL/Fedora. The location of configuration files, and other tools, might be different depending on your Linux distro. For the most part though, everything should be pretty similar and you should be able to figure it out.

Install Apache, Subversion, mod_dav_svn and mod_ssl

On CentOS, installing the Apache web-server, Subversion, and the Apache mod_dav_svn and mod_ssl modules are a snap with yum:

#(root)/> yum -y install httpd subversion mod_dav_svn mod_ssl openssl

If you’re on Ubuntu you can probably install the required packages using apt-get install. Note that you need to install mod_ssl if you plan on securing your SVN server with HTTPS. If you don’t care about HTTPS, then you can ignore mod_ssl and skip to “Configure mod_ssl and Setup HTTPS” below.

Create your SVN Root Directory Structure

On my SVN server, I created a new SVN root at /svn. From here on out, all of my SVN repositories will live under /svn/repos:

#(root)/> mkdir -p /svn/repos
#(root)/> chown -R apache:apache /svn

Once done, you’re ready to create your first SVN repository.

Create your First Repository

Using the svnadmin command, create a repository under /svn/repos. For the sake of this example, the repository I’m creating is named myproject. Of course, you can name your own repository whatever you’d like. Oh, and you can create as many repositories as you’d like under /svn.

#(root)/> cd /svn/repos
#(root)/svn/repos> svnadmin create --fs-type fsfs myproject
#(root)/svn/repos> chown -R apache:apache myproject
#(root)/svn/repos> chmod -R g+w myproject
#(root)/svn/repos> chmod g+s myproject/db

You’ll notice that the svnadmin command created a new directory named myproject/. If you look inside myproject/ you’ll see a bunch of SVN repository data and configuration files.

#(root)/svn/repos> ll myproject
total 28
drwxrwxr-x 2 apache apache 4096 Mar 13 11:49 conf
drwxrwxr-x 2 apache apache 4096 Mar 13 12:01 dav
drwxrwsr-x 5 apache apache 4096 Mar 13 12:23 db
-r--rw-r-- 1 apache apache 2    Mar 13 11:49 format
drwxrwxr-x 2 apache apache 4096 Mar 13 11:49 hooks
drwxrwxr-x 2 apache apache 4096 Mar 13 11:49 locks
-rw-rw-r-- 1 apache apache 229  Mar 13 11:49 README.txt

Great, looks like our new SVN repository was setup correctly!

Configure mod_dav_svn

Now that you have an SVN repository setup and ready to go, let’s configure Apache and the mod_dav_svn module. Open /etc/httpd/conf.d/subversion.conf in your favorite text editor, and tweak the configuration to match your installation. My subversion.conf file looks like this:

LoadModule dav_svn_module     modules/
LoadModule authz_svn_module   modules/

<Location /svn>

   DAV svn
   SVNParentPath /svn/repos

   # Require SSL connection for password protection.

   AuthType Basic
   AuthName "Marks SVN Server"
   AuthUserFile /svn/repos/users
   Require valid-user


First, note that when you install mod_dav_svn using yum, the installation process will create a standard cookie cutter template /etc/httpd/conf.d/subversion.conf for you. This template has a LimitExcept directive in it, and a few other things. For security, I think it’s best to require a user to authenticate before they are able to issue any request. Hence, why I removed the LimitExcept directive and did my own thing. If you want your SVN server to be read-only for anonymous users, and read-write for authenticated users, then my subversion.conf file is not for you. My subversion.conf file shown above allows no anonymous access; all users must authenticate (enter a valid username and password) before they can do anything with the SVN server.

Second, note that I have enabled the SSLRequireSSL directive. This triggers mod_dav_svn to reject all non-HTTPS requests. This ensures that any communication between the server and my SVN client will be sent via HTTPS; usernames, passwords, and source code will be reasonably secured. I’ll show you how to setup HTTPS here in a moment. Note that if you don’t want to enable HTTPS on your SVN server, then you can comment out or remove the SSLRequireSSL line in your subversion.conf configuration file.

Finally, note that my AuthUserFile is /svn/repos/users. This is a standard Apache htpasswd file that we’ll create in the next step.

Create your SVN Users File

Create your SVN users file using the htpasswd command. This is the file that stores a list of usernames and passwords declaring who is allowed to access your SVN server.

#(root)/> htpasswd -c /svn/repos/users mark

Replace “mark” above with your desired username. Repeat this command for however many users you need to add access.

Configure mod_ssl and Setup HTTPS

If you have decided to make your SVN sever an HTTPS only server, we’ll need to setup Apache’s HTTPS configuration. This involves tweaking /etc/httpd/conf.d/ssl.conf and creating a new self-signed SSL certificate. For your convenience, I’ve included the same set of instructions below. Note that my SVN server is named svn.kolich.local — yours will obviously be different. Whatever it is, make sure that you enter the correct server name when openssl prompts you for a “Common Name” in your certificate. The “Common Name” in your SSL certificate should match the fully qualified name of your SVN server.

Note that if you have an SSL certificate signed by a legitimate Certificate Authority (Network Solutions, Verisign, Thawte) you shouldn’t need to generate a new SSL key and self-signed certificate. You can simply use the one issued to you by your CA.

First, create a new SSL key with the openssl command:

#(root)/> mkdir /etc/httpd/ssl
#(root)/> cd /etc/httpd/ssl
#(root)/etc/httpd/ssl> openssl genrsa 4096 > svn.kolich.local.key

Now that you have a private key, create a self-signed certificate:

#/etc/httpd/ssl> openssl req -new -key svn.kolich.local.key -x509 \
    -days 1095 -out svn.kolich.local.crt

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [GB]:US
State or Province Name (full name) [Berkshire]:California
Locality Name (eg, city) [Newbury]:My Town
Organization Name (eg, company) [My Company Ltd]:Mark Kolich
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:svn.kolich.local
Email Address []:

Finally, edit /etc/httpd/conf.d/ssl.conf to point to your newly generated SSL key and SSL certificate. This involves updating the SSLCertificateFile and SSLCertificateKeyFile directives accordingly:

## SSL Virtual Host Context

<VirtualHost _default_:443>
 ## Required, see
 SSLEngine On
 SSLCertificateFile /etc/httpd/ssl/svn.kolich.local.crt
 SSLCertificateKeyFile /etc/httpd/ssl/svn.kolich.local.key

Note that you should not place your SSL private key and certificate in a location accessible by the web-server. Usually placing them under /etc/httpd is sufficient. It would be less desirable and quite insecure to place them under /var/www/html for example.

Configure HTTP to HTTPS Redirection

If you’ve bothered to setup HTTPS in the previous step, you probably want Apache to gracefully redirect clients from HTTP to HTTPS. If you don’t automatically redirect, and you have SSLRequireSSL enabled in your subversion.conf file, when clients try to communicate with your SVN server via HTTP they’ll see a 403 Forbidden error. Instead, let’s 301 Moved Permanently redirect them to HTTPS. Open /etc/httpd/conf/httpd.conf in your favorite text editor, jump to the bottom of the file, and edit your VirtualHost configuration. Mine is as follows:

NameVirtualHost *:80

<VirtualHost *:80>

  DocumentRoot /var/www/html
  ServerName svn.kolich.local
  ServerAlias svn

  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^/(.*)$ https://svn.kolich.local/$1 [R=301,L]

  ErrorLog logs/svn.kolich.local-error_log
  CustomLog logs/svn.kolich.local-access_log common


Save it, and you’re done. Now when a client tries to communicate with my SVN server via HTTP, it’ll see a 301 Moved Permanently redirect to HTTPS. If my SVN client is smart enough, it will gracefully follow this redirect to HTTPS, and all is well. Of course, you’ll need to change the HTTPS URL shown above in the RewriteRule directive to match your server hostname (your SVN server is not svn.kolich.local).

Start Apache, and Enjoy

That’s it! Start Apache and checkout your new repository.

#(root)/> /etc/init.d/httpd start

On another machine, try to checkout the repository:

#(mark)~> svn co http://svn.kolich.local/svn/myproject
svn: PROPFIND request failed on '/svn/myproject'
svn: PROPFIND of '/svn/myproject': 301 Moved Permanently (http://svn.kolich.local)

Yep, HTTP to HTTPS redirection is working as expected. Unfortunately my SVN client isn’t smart enough to follow the redirect on its own. Oh well, change that repository URL to HTTPS, and try again:

#(mark)~> svn co https://svn.kolich.local/svn/myproject
Error validating server certificate for 'https://svn.kolich.local:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: svn.kolich.local
 - Valid: from Mar 15 20:17:38 2010 GMT until Mar 14 20:17:38 2013 GMT
 - Issuer: Mark Kolich, California, US
 - Fingerprint: ff:ee:b6:9c:d8:d7:78:3b:ce:9e:09:dd:4a:99:93:11:3e:12:07:85
(R)eject, accept (t)emporarily or accept (p)ermanently? p
Authentication realm: <https://svn.kolich.local:443> Marks SVN Server
Password for 'mark': ...
A    myproject
Checked out revision 0.

It worked! Note that the “Error validating server certificate” warning is because I’m using a self-signed SSL certificate. When SVN asks if you want to accept the certificate, if you permanently accept it you will not be prompted about this again. If you use an SSL certificate issued by a real Certificate Authority like Network Solutions, Verisign, or Thawte, you shouldn’t see this warning.

Time to start hacking — cheers!

UAC Prompt From Java: CreateProcess error=740, The requested operation requires elevation (ShellExecuteEx Runas Example)


Fri Dec 18 09:35:00 2009 -0800

I just finished an absolutely monstrous project at work that involved quite a bit of Java and a little Visual C++. The latter part of this project involved writing some VC++ that interactively upgraded a piece of Java based software installed on a PC. This sounds relatively mundane, but frankly, it wasn’t. I ended up spending almost of week of engineering effort, writing code that gracefully understands how to deal with Window’s User Account Control (UAC). If you’re not familiar with UAC, it’s that incredibly annoying security mechanism in Vista, and Win7, that prompts the user for confirmation if a piece of software attempts to make any changes to a protected location on the computer. I appreciate what Microsoft was trying to accomplish with UAC, but it couldn’t be more painful for a developer to work with. Not to mention I couldn’t find any decent documentation from Microsoft that discussed how to properly integrate your software with UAC. I found a lot of marketing type documents and other nonsense through MSDN, and eventually gave up in disgust. All I wanted was a simple document, ideally one titled “this is how to open a UAC prompt in your application.”

In the end, I figured out how to deal with UAC by studying the source code of the Mozilla Firefox Updater (knowing that Firefox is open-source, so I can look at its code, and it always seems to update itself just fine on my development Win 7 and Vista boxes). This post is an attempt to document how I built an application that understands, and gracefully handles UAC.

So, you need to modify a file or directory in a protected location on the file system, eh? Like Program Files? Or, maybe you need to register a DLL or other library as an Administrator? In Vista, and Windows 7, your application can’t do any of these things without elevating itself to an Administrator, or privileged user. Your application running as a normal user, must programmatically elevate itself to an Administrator before it has permission to run these privileged operations. Your application needs to trigger Windows to display a UAC prompt. Welcome to the painful world of Windows User Account Control.

Here are several important points to remember about UAC (things I learned the hard way):

  • Unprivileged applications running as normal users, cannot simply fork another process to trigger a UAC prompt. Even if the requestedExecutionLevel of your application manifest is requireAdministrator. Using _spawnv(), exec(), etc. with a binary that has requireAdministrator set in its manifest will NOT work. For example, a Java app running as a normal user cannot spawn a process with privileged access; even in the process Java is trying to spawn has the correct `requireAdministrator manifest property.
  • A unprivileged application can only trigger a UAC prompt using the ShellExecute or ShellExecuteEx shell functions, provided via shell32.lib.
  • Setting requireAdministrator in your application manifest only appears to open a UAC prompt when a user double clicks on your executable in Windows Explorer.
  • If you are writing a Java application that needs to do something privileged on the computer, you should know you can’t directly do it in Java. If you follow good software engineering practices, your Java application (the JRE) will run as a normal user, the user that started the application. So, when you need to do something privileged your Java application must spawn another helper application (that you have to write) that understands UAC and prompts the user if necessary. If the user accepts the UAC prompt from your native helper application, then it can do what it needs to do in a privileged mode (e.g, moving files around under Program Files/, registering a DLL, etc.).

The Application Manifest

Several resources claim you can trigger a UAC by simply inserting an application manifest into the assembly of your application binary. An application manifest that triggers a UAC looks something like this:

<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <assemblyIdentity version="" processorArchitecture="X86"
       name="yourapp.exe" type="win32">
  <description>Some Application Description</description>
      <assemblyIdentity type="win32"
         version="" processorArchitecture="*"
         publicKeyToken="6595b64144ccf1df" language="*" />
  <ms_asmv3:trustInfo xmlns:ms_asmv3="urn:schemas-microsoft-com:asm.v3">
        <ms_asmv3:requestedExecutionLevel level="requireAdministrator"

Note the level=requireAdministrator attribute in this XML that I alluded to earlier. This works, but only when the user double-clicks your executable, or launches it from the Start Menu. This is hardly sufficient for an application that needs to become an Administrator when updating itself. You’ll need something more.

Basic UAC Flow

Ok, so before you dig into it, I thought it might be helpful to explain the basic flow of a UAC aware application and how everything fits together. Normally, your application runs as an unprivileged user. But, sometimes it needs to be an Administrator (to do whatever). So, here’s the basic idea, in pseudo code:

int main (int argc, char **argv) {

  HRESULT operation = tryToDoSomethingPrivileged();

  if (operation == ACCESS_DENIED && !alreadyElevated) {

    // Spawn a copy of ourselves, via ShellExecuteEx().
    // The "runas" verb is important because that's what
    // internally triggers Windows to open up a UAC prompt.
    HANDLE child = ShellExecuteEx(argc, argv, "runas");

    if (child) {
      // User accepted UAC prompt (gave permission).
      // The unprivileged parent should wait for
      // the privileged child to finish.
      WaitForSingleObject(child, INFINITE);
    else {
      // User rejected UAC prompt.
      return FAILURE;

    return SUCCESS;


  return SUCCESS;


On Windows XP, or other versions of Windows with UAC disabled, note that the user will simply run right through the code fragment without any prompt. However, if UAC is enabled, the privileged operation will be rejected meaning the application needs to spawn a copy of itself using ShellExecuteEx.

Here’s a quick code snippet showing the usage of ShellExecuteEx to open a UAC prompt:

memset(&sinfo, 0, sizeof(SHELLEXECUTEINFO));
sinfo.cbSize       = sizeof(SHELLEXECUTEINFO);
sinfo.fMask        = SEE_MASK_FLAG_DDEWAIT |
sinfo.hwnd         = NULL;
sinfo.lpFile       = argv[0];
sinfo.lpParameters = spawnCmdLine;
sinfo.lpVerb       = L"runas"; // <<-- this is what makes a UAC prompt show up
sinfo.nShow        = SW_SHOWMAXIMIZED;

// The only way to get a UAC prompt to show up
// is by calling ShellExecuteEx() with the correct
// SHELLEXECUTEINFO struct.  Non privlidged applications
// cannot open/start a UAC prompt by simply spawning
// a process that has the correct XML manifest.
BOOL result = ShellExecuteEx(&sinfo);

Note the runas lpVerb in the SHELLEXECUTEINFO struct — this is the verb that triggers windows to “shell execute” your application via UAC.

An Example

I whipped up a quick yet complete UAC example in Visual C++. You can download the entire VC++ project here. Or, download just the pre-compiled release binary if you want to experiment. Or, perhaps you’d prefer just the CPP source code. This VC++ example code was built using Visual C++ 2008 Express Edition.

The UAC demo, aptly named uac-example, should be run from a command prompt with a single argument:

C:\> uac-example.exe <working directory>

My UAC example works by attempting to create an empty file in the working directory specified by the command line argument. If the working directory happens to be a Windows protected location on the file system, like C:\Program Files, the example app will initially fail to create this empty file. In that case, it will re-spawn a copy of itself via a UAC prompt and try again as an Administrator. If the app successfully created the file when elevated, you will see the following success message:

If you reject the UAC prompt (clicked Deny), you’ll see this:

If your working directory isn’t a Windows protected directory, like %APPDATA% or your Desktop, you’ll immediately see this without a UAC prompt:

Here are some example working directories you might like to try:

C:\> uac-example.exe "C:\Program Files"
C:\> uac-example.exe "%APPDATA%"
C:\> uac-example.exe .

Please note that if you’re using a UAC capable computer, but UAC is turned off, no matter what directory you give to my example app it will always say “Worked (no UAC required)!” This is obviously because if UAC is turned off, Windows isn’t actively protecting any locations on the file system, so any running process can pretty much do whatever it wants.

Opening a UAC Prompt From Java

If you need to open a UAC prompt from Java, you should know that there is no way to do so without writing your own native app. Your Java code should call your UAC enabled native app to do “whatever it needs to do” in a privileged mode. Here’s an example using my uac-example app with Java’s ProcessBuilder:

import java.util.ArrayList;
import java.util.List;

public class UACTester {

  private static final String UAC_EXAMPLE_EXE = "uac-example.exe";

  public static void main(String[] args) {

    final File uacExample = new File(UAC_EXAMPLE_EXE);
    File workingDir;

    try {
      workingDir = new File(args[0]);
    } catch ( Exception e ) {
      workingDir = new File(".");

    try {
      // Build the command list to be given to the ProcessBuilder
      final List<String> cmdArgs = new ArrayList<String>();

      // Create a process, and start it.
      final ProcessBuilder p = new ProcessBuilder(cmdArgs); File("."));
    } catch (Throwable t) {



Note that if your Java app attempts to spawn a process as an Administrator, you’ll most likely see an exception like this:

CreateProcess error=740, The requested operation requires elevation

If the argument given to my sample Java app is a UAC protected directory, the native executable it calls will need to open a UAC prompt.


Links and Other Resources

Here are a few links and other resources I gathered while writing this blog post and integrating UAC support into a project at work:


Apache Tip: Deny TRACE and TRACK Requests with mod_rewrite


Sat Nov 14 10:41:46 2009 -0800

It’s long been rumored that exposing the HTTP TRACE and TRACK methods on your web-server can open the door to a number of miscellaneous vulnerabilities, including cookie thefts and other cross-site tracing attacks. Many resources out there claim you should configure you web-server to flat-out reject TRACE and TRACK requests, and I agree with them. Generally speaking, there’s really no good need (that I’ve found) that would require or make use of TRACE or TRACK. With that said, if you’re running Apache, it’s fairly easy to reject TRACE and TRACK using mod_rewrite:

RewriteRule ^/(.*)$ - [F,L]

You can prove to yourself that this works, by using a tool like curl to issue an HTTP TRACE and TRACK to your newly secured web-server. Use the -X option with curl to specify the HTTP request type:

#/> curl -v -X TRACE
* About to connect() to port 80 (#0)
*   Trying connected
* Connected to ( port 80 (#0)
> TRACE / HTTP/1.1
> User-Agent: Curl
> Host:
> Accept: */*
< HTTP/1.1 403 Forbidden
< Date: Sat, 14 Nov 2009 18:53:06 GMT
< Server: Apache
< Content-Length: 202
< Connection: close
< Content-Type: text/html; charset=iso-8859-1
<title>403 Forbidden</title>
<p>You don't have permission to access / on this server.</p>
* Closing connection #0

Yep, works nicely. One thing that slightly annoys me, however, is that the HTTP OPTIONS method still reports that my server supports TRACE, even though I clearly don’t anymore. A quick Google search reports that many other folks have had the same concern, with no clear resolution.

Apache, Don't Log Yourself (Don't Log Specific IP Address and User-Agents)


Sun Jun 07 11:30:00 2009 -0700

In some Apache web-server configurations, it might be useful avoid logging requests from specific IP addresses or User-Agent’s. For example, if you regularly check your own site from your home network you probably don’t want to record your own visit in your Apache access_log’s. For, I stopped logging requests from my home network since I was filling up my own log files with redundant junk.

httpd.conf Specific Configuration

Here’s how you can stop logging requests from a specific IP address, or a request for a specific resource (goes in your httpd.conf file):

## Dont log myself (requests from my own network), requests
## for my robots.txt file.
SetEnvIf Remote_Addr "192\.168\.1\." dontlog
SetEnvIf Request_URI "^/robots\.txt$" dontlog

## I don't use this, but I have it here as an example.  Why
## I avoid using this is explained below.
##SetEnvIfNoCase User-Agent "(msnbot|googlebot|slurp)" dontlog

## Normal logging directives.  Note the env!=dontlog at the
## end of CustomLog.
ErrorLog logs/
CustomLog logs/ combined env=!dontlog

Why Should I NOT Use SetEnvIfNoCase User-Agent to Avoid Logging Requests From Specific User-Agents?

You can use SetEnvIfNoCase User-Agent if you want to stop logging requests from specific User-Agent’s. However, I would recommend that you avoid using this feature because it is extremely easy to forge/fake the User-Agent header of an HTTP request. If a hacker tries to probe or attack your site disguised as the “GoogleBot”, and your Apache server is configured to not log requests from clients that claim they are the “GoogleBot”, you won’t see the probe attacks in your log files. In short, Apache will think the request is from the GoogleBot, when in fact, it could be a hacker or malicious user masquerading as a web-crawler.


The (Right?) Way to Parse, Filter, and Validate GET and POST Inputs From a Form in PHP (filter_input_array)


Sat Jun 06 11:00:00 2009 -0700

There are an endless number of ways to parse and validate GET or POST inputs to a PHP script. I’ve seen examples (and have written code) that use everything from array_key_exists to empty to preg_match. In any event, there doesn’t appear to be a standard way of parsing and validating form inputs from a GET or a POST; each developer and application seems to use its own validation mechanism. So, what is the PHP community preferred way to safely and securely validate submitted formvars? I don’t know. But, I think I personally use a pretty robust and secure method for validating GET and POST form inputs in PHP. So, consider this post an attempt to document and explain what I use in PHP to securely parse, filter, and validate formvars from a GET or POST.

For form validation, assuming you’re running PHP 5.2.0 or greater, you really can’t do much better than PHP’s very own filter_input_array function. If you’re not familiar with filter_input_array, you should look into it. From, filter_input_array, “…gets external variables and optionally filters them.” In other words, filter_input_array gives you a lot parsing and validation logic for free. It avoids the need to manually walk through each element of $_POST[], check if it exists, etc. And for an AJAX controller (a handler) written in PHP, filter_input_array can be very helpful.

In most cases, I find that a nice combo of filter_input_array and empty works quite nicely for most of my form validation needs.

The <form>

Here’s a reasonably complex <form> that should exercise filter_input_array. In this form, I have a bunch of text inputs, some check boxes, and a hidden field. One of the text inputs must be a valid email address, another must be a seven digit number (all digits, no non-digit characters), and the last one must be a number between 0 and 20. The hidden field must be a valid URL (maybe useful for a script redirect or something like that). The form will be submitted via a POST.

<form method="post" action="<?= $_SERVER['PHP_SELF']; ?>">

  <input type="text" name="email" />
  <input type="text" name="sevendigits" />
  <input type="text" name="zerototwenty" />

  <input type="checkbox" name="checkboxes[]" value="cb1" />
  <input type="checkbox" name="checkboxes[]" value="cb2" />
  <input type="checkbox" name="checkboxes[]" value="cb3" />

  <input type="hidden" name="validurl" value="" />

  <input type="submit" value="Submit" />


The Ole’ Fashioned (Common) Way to Parse Formvars

The common way to parse a form like this on the backend is to load each formvar from the $_POST[] array into separate variables and then check their value:

$email = $_`POST`['email'];
$sevendigits = $_`POST`['sevendigits'];
$zerototwenty = $_`POST`['zerototwenty'];
// ... and so on.

  // Empty email, show error or do something else.

    $sevendigits = intval($sevendigits);
  } else {
    // Not seven digits
} else {
  // Error, sevendigits wasn't submitted.

// ... and so on, you get the idea.

A lot of code, and time, is wasted upfront parsing, checking, and validating each of the formvars from the POST. Also, notice all of the nested and cumbersome if/else’s all over the place. We can definitely do better.

Using filter_input_array() Instead

Instead of checking each field one-by-one, in PHP 5.2.0 or later we can define a filter upfront that does most of this tedious work for us. The filter will return “…an array containing the values of the requested variables on success (if the input matches the required format), or FALSE on failure. An array value will be FALSE if the filter fails, or NULL if the variable is not set.”

Here’s some sample code that defines a filter and uses filter_input_array to validate the incoming variables from a POST:


// Works only in PHP 5.2.0 or later.

$filter = array(

   // This ensures that $_`POST`['email'] is actually a
   // valid email address.

   // This filter verifies that $_`POST`['sevendigits'] is
   // exactly a seven digit number using a regular expression.
   'sevendigits' => array('filter'  => FILTER_VALIDATE_REGEXP,
                          'options' => array('regexp' => '/^\d{7}$/')

   // Make sure that $_`POST`['zerototwenty'] is a number
   // 0 to 20 using the min_range and max_range specs.
   'zerototwenty' => array('filter'  => FILTER_VALIDATE_INT,
                           'options' => array('min_range' => 0,
                                              'max_range' => 20)

   // Verify that the incoming $_`POST`['checkboxes'] from
   // the checkbox list is actual an array like we expect.
   'checkboxes' => array('filter' => FILTER_VALIDATE_INT,
                         'flags'  => FILTER_REQUIRE_ARRAY,

   // Make sure that the hidden URL field is a valid
   // properly formatted URL.
   'validurl' => FILTER_VALIDATE_URL,

   // ----- A few other filter examples not included
   // ----- in the <form> sample above.  I just felt like
   // ----- experimenting with a few other filters.

   // This field must be a boolean type.  If mustbeboolean is
   // "true", "1", "TRUE" or some other value that represents
   // true then this will be true.  Otherwise, it will be false.
   'mustbeboolean' => array('filter' => FILTER_VALIDATE_BOOLEAN),

   // Encode a URL that we need encoded from $_`POST`['encodeurl']
   'encodeurl' => array('filter' => FILTER_SANITIZE_ENCODED),

   // This dosen't exist in the form, I'm just using it to show
   // what the result will be when an input doesn't exist.
   'doesnotexist' => FILTER_VALIDATE_INT


// Filter and sanitize the incoming $_`POST`[] with the filter above.
$inputs = filter_input_array( INPUT_`POST`, $filter );

// Here's an example of checking if $_`POST`['email'] made it
// past our FILTER_VALIDATE_EMAIL filter.
if( empty($inputs['email']) ) {
   echo "Empty or invalid email entered.";


As you can see, we define a set of filters upfront that validates the input from $_POST[]. The result is stored in $inputs, which we can then access like any other associative PHP array. BTW, additional filter constants like FILTER_SANITIZE_ENCODED, FILTER_VALIDATE_IP, etc. can be found on PHP’s Predefined Filter Constants page.

A More Complete Example

To test the example code above, I threw together some PHP which you can find here.

Using filter_input_array() with AJAX

In terms of usefulness, filter_input_array is perfect for an AJAX controller written in PHP. Imagine a controller expecting three or so formvars from a $_GET. Instead of checking for each one manually, one-by-one, you can define a filter upfront that checks and validates the incoming formvars. From there, a simple empty() check is sufficient to see if the variable matched the desired type, or format.

Here’s a sample AJAX controller illustrating the use of filter_input_array with a GET. Note that this script might be called with a request that looks like ../ajax/controller.php?idnumber=123&field=op&status=true:


// Only works in PHP 5.2.0 or later.

// A sample AJAX controller written in PHP that shows
// how to use filter_input_array().

$filter = array(

   // An ID number must be exactly 7-digits.
   'idnumber' => array('filter'  => FILTER_VALIDATE_REGEXP,
                       'options' => array('regexp' => '/^\d{7}$/')

   // A field name to update.
   'field' => array('filter'=> FILTER_SANITIZE_STRING),

   // Set the field to either true or false.
   'status' => array('filter' => FILTER_VALIDATE_BOOLEAN)


try {

   // Filter $_GET[]
   $inputs = filter_input_array( INPUT_GET, $filter );

   $id = null;
   if( empty($inputs['idnumber']) ) {
      throw new Exception("Invalid ID number!");
   } else {
      $id = $inputs['idnumber'];

   $field = null;
   if( empty($inputs['field']) ) {
      throw new Exception("Field to update cannot be empty!");
   } else {
      $field = $inputs['field'];

   $status = false;
   if( !empty($inputs['status']) ) {
      $status = $inputs['status'];

   // Do something here, probably with a database, to save/update
   // to set $field = $status for ID number $id

   echo "SUCCESS";

} catch ( Exception $e ) {
   echo "ERROR: " . $e->getMessage();



Whole Disk Backup and Recovery with dd, gzip, and p7zip


Tue May 26 21:31:00 2009 -0700

Several days ago I spent more than 5 hours setting up a fresh install of Windows Vista Enterprise on a desktop computer in my home office. This setup process involved tweaking the system configuration, installing and configuring all of my required software for development, checking out a massive SVN code repository, etc. Sadly, less than a week after I finished setting up this box, the SATA hard disk died. As it turns out, the disk flat out overheated due to poor airflow around the disk chassis in my PC (that’s a story/post/opinion piece for another day). In short, I lost everything on the drive. All of the blood, sweat and tears; for nothing.

I decided to never let this happen again (since it’s a huge time suck sitting around waiting for software to install) and began investigating free, yet solid and reliable backup solutions suitable for a home office. This post is an attempt to document what whole disk backup and recovery solution worked for me, using several freely available open-source tools.

It’s a known fact that more than half of all backups fail on recovery. As a result, I want something non-proprietary, works with any file system, is simple to use, and is pretty much guaranteed to work on all hardware. I also want to be sure that once backed up, my data is compressed using a very common data format (e.g., gzip). It would be a shame to use a proprietary tool that locks my data into some commercial proprietary format.


For the sake of this HOWTO, I’m going to assume you are familiar with “Linux on a CD” distributions like SystemRescueCD or Knoppix. Unfortunately, if you aren’t familiar with Linux, the command line, or how to use a Linux on a CD distro then this HOWTO is probably going to feel a bit over your head. BTW, this HOWTO assumes the drive you want to backup is at /dev/sda. Your block device DSF (device special file) might be different.

This HOWTO is provided to you “as is”, without warranty of any kind, express or implied. I am not responsible for data loss or hardware damage that occurs as a result of using these instructions. Use at your own risk.

Boot into Linux on a CD

Pop in your favorite Linux on a CD distro and boot your PC accordingly. For the sake of this HOWTO I’m going to assume you’re using SystemRescueCD. However, any decent Linux on a CD distribution should have all of the tools you’ll need.

Figure Out Where to Place the Backup

Before you do anything further, you should figure out where you are going to place your backups. Backups are usually quite big, so expect them to chew up a good 80-100 GB of storage in most cases. The better compression you use when making the backup, the less storage space you’ll need.

In my case, I decided to put the backup on a large RAID-1 (mirror) volume I have in my home datacenter. The mirrored volume is on another host, so I need to mount it locally using sshfs:

rescuecd#/> mkdir /mirror
rescuecd#/> sshfs mark@backup-host:/raid/backups /mirror

Once my mirror is mounted, I can read and write data to /mirror which will directly pipe it to the box connected to my RAID-1 volume via sshfs.

Determine the Appropriate Block Size

For a quicker backup, it can help to nail down the optimal block size of the disk device you are going to backup. Assuming you are going to backup /dev/sda, here’s how you can use the fdisk command to determine the best block size:

rescuecd#/> /sbin/fdisk -l /dev/sda | grep Units

Units = cylinders of 16065 * 512 = 8225280 bytes

Note the fdisk output says “cylinders of 16065 * 512”. This means that there are 512 bytes per block on the disk. You can significantly improve the speed of the backup by increasing the block size by a multiple of 2 to 4. In this case, an optimal block size might be 1k (512*2) or 2k (512*4). BTW, getting greedy and using a block size of 5k (512*10) or something excessive won’t help; eventually the system will bottleneck at the device itself and you won’t be able to squeeze out any additional performance from the backup process.

Backup the Partition Layout

Before you do anything, it’s always a good idea to backup the partition layout. When you create a whole disk backup, you don’t have to worry about partitions. However, it can be handy to have this partition information (in case you need to mount a specific partition in the backup as a file using the exact offset). Use the sfdisk command to backup the partition layout:

rescuecd#/> sfdisk -d /dev/sda > /mirror/backup-sda.sf

Once you’ve backed up the partition layout, you can cat /mirror/backup-sda.sf to verify that you’ve correctly saved the partition mapping.

Backup the Master Boot Record (MBR)

Again, you don’t need to explicitly do this since a whole disk backup includes the MBR, but it’s a good idea to snag the master boot record just in case. To backup the MBR, you can use the dd command:

rescuecd#/> dd if=/dev/sda of=/mirror/backup-sda.mbr count=1 bs=512

If you want to prove to yourself that you’ve successfully saved the MBR, you can run file /mirror/backup-sda.mbr to confirm you got what you needed:

rescuecd#/> file /mirror/backup-sda.mbr
backup-sda.mbr: x86 boot sector; partition 2: ID=0x83, active, starthead 1, \
  startsector 63, 2104452 sectors; partition 3: ID=0x82, starthead 0, \
  startsector 2104515, 4192965 sectors, code offset 0x48

Yep, the file command confirmed that we’ve successfully snagged the MBR of the disk. The MBR always sits on the first 512-bytes of any bootable disk.

Run the Backup

Now that you’ve saved everything you need from the disk, it’s time to make the backup. To create the backup, we’ll use the dd command in conjunction with gzip -9 for max compression. For dd, we’ll use an optimal block size of 1024 (as determined above).

Warning: this will take a long time so it’s probably best to let this run overnight. On my system at home, it took me 7+ hours to backup an entire 250 GB disk:

rescuecd#/> dd if=/dev/sda bs=1024 conv=noerror,sync | pv | gzip -c -9 > /mirror/backup-sda.gz

The conv=noerror,sync flag asks dd to keep going even if there are any read errors with the disk and to pad every input block with NULs to match your input block size. Note that I’m using the pv command to monitor the speed and progress of data flowing between dd and gzip. The pv command will tell me how much data I’ve processed, how long the backup has been running, and the approx speed of my backup; essentially it displays a progress bar on the console.

Re-compress with P7ZIP (if desired)

Gzip offers pretty decent compression, but if you want insanely awesome compression, you can use P7ZIP to compress your backups. After the dd to gzip backup is complete, you can re-compress backup-sda.gz using P7ZIP if you’d like to save a little storage space. If so, here’s how:

rescuecd#/> gunzip -c /mirror/backup-sda.gz | 7za a /mirror/backup-sda.7z -si

Again, be warned, this process will seem like it takes forever. However, using P7ZIP over Gzip, saved me about 5 GB on the compressed backup. Using gzip -9 alone, I compressed a 250 GB backup image down to about 31 GB. With P7ZIP, the same backup was only 26 GB. P7ZIP is interesting because it sacrifices CPU cycles for compression, using a more exhaustive and complete compression algorithm. If you want more information on P7ZIP, check out Wikipedia’s article on the 7z compression format.

Restore from Backup

Backups are useless unless you can actually restore your data. If you need to restore a P7ZIP compressed backup to /dev/sda, here’s how:

rescuecd#/> 7za x /mirror/backup-sda.7z -so | dd of=/dev/sda bs=1024

If you decided to skip P7ZIP compression, and need to restore a Gzip compressed backup to /dev/sda, here’s how:

rescuecd#/> gunzip -c /mirror/backup-sda.gz | pv | dd of=/dev/sda bs=1024

Compression Tip

For best performance, I strongly recommend zeroing out your disk before installing any OS’es on the drive you plan to backup. For example, I recently upgraded my Vista Enterprise box to Windows 7 Professional. I backed up Vista using the instructions in this post, then used the dd command to zero out the disk before installing Windows 7:

rescuecd#/> dd if=/dev/zero of=/dev/sda bs=1024 conv=noerror,sync

This writes zeroes to the entire disk, essentially eliminating any random or stray data that’s lingering at the end of the drive from a previous OS install. Then, once I install Windows 7 and back it up, the backup process will compress Windows 7, my data, and all installed applications. Eventually, it will hit the “rest of the disk”, which is all zeroes. As a result, the backup run time is reduced not to mention that the backup itself could be a fraction of the normal size (most compression algorithms LOVE large streams of similar patterns; they’re built for that, so if you give gzip, p7zip, or bzip2 some data then a huge stream of all zeros, expect some insanely good compression).

Using this technique, on one of my desktops, I compressed a 250GB disk with Windows 7 Professional installed down to only 12GB.

So, if you plan on re-installing your OS, then making a backup, you should always use /dev/zero to zero out your disk before doing anything.

Good luck!

Quick and Simple Tips to Better Secure your MySQL Server


Sat Apr 25 18:10:00 2009 -0700

Most MySQL server installations floating around on the Internet are blindly using the default server configuration. Even more shocking, you’d be surprised to find out how many of these MySQL server installations are left wide open, completely vulnerable to attacks. In this post, I’ll provide a few simple tips you can use to better secure your MySQL server. Note that these tips should be used as a basic starting point; they are not an end-all-be-all MySQL security solution. You still need to know what you’re doing, and if you’re unsure, find a consultant or friend who can help you lock down your server.

Change the Default Admin Username and Password

Sadly, many MySQL server installations are using the default admin username and password configuration. By default, the administrative username for MySQL is “root” and the password is empty (no-password). Anyone with root access to your MySQL server can log into your database, create new users, change or grant privileges, add or drop tables, etc. So, let’s change the root username to something less obvious, and while were at it, let’s change the root password too:

(root)/> mysql -u root mysql

mysql> UPDATE user SET user = "mydbadmin" WHERE user = "root";

Delete Default Users

On occasion, several default database user accounts are created when you install MySQL. You really shouldn’t need these for any reason, so let’s remove them:

(root)/> mysql -u mydbadmin -p mysql

mysql> DELETE FROM user WHERE NOT (host = "localhost" AND user = "mydbadmin");

Create Separate User Accounts for Each Application

Now, let’s create a single MySQL account per application that needs access to your database server. Some database administrators provide a single account that has global access to all MySQL databases and tables. This is bad for a number of reasons, but mainly because if one application using the database is compromised, a hacker can then access all other tables and databases running on the server.

For the sake of this example, say your MySQL database will be used by a blog and a guestbook. The blog and guestbook are separate web-applications, and therefore, should have their own MySQL username and password. The idea is that the blog MySQL account shouldn’t have access to the guestbook database and vice versa. If a hacker compromises the username and password used to access the blog database, they won’t be able to access the guestbook database. So, let’s create a separate database and user account for the blog and guestbook:

(root)/> mysql -u mydbadmin -p

mysql> CREATE DATABASE blog;
mysql> CREATE DATABASE guestbook;

mysql> CREATE USER 'blgu'@'localhost' IDENTIFIED BY 'somepass';
mysql> CREATE USER 'gbu'@'localhost' IDENTIFIED BY 'anotherpass';

mysql> GRANT ALL PRIVILEGES ON blog.* TO 'blgu'@'localhost' WITH GRANT OPTION;
mysql> GRANT ALL PRIVILEGES ON guestbook.* TO 'gbu'@'localhost' WITH GRANT OPTION;


Be sure to use good passwords when creating your new MySQL user accounts.

Delete the Sample Databases

As a general rule of thumb, you should delete or disable anything you don’t explicitly need. This includes the sample databases created by MySQL at install time. On most MySQL installations, the sample/test database is named “test”. Let’s remove it since it’s just another component we don’t need:

(root)/> mysql -u mydbadmin -p

mysql> DROP DATABASE test;
mysql> quit;


Generate Your Own Self-Signed SSL Certificates for Apache HTTPS


Sat Mar 21 23:01:56 2009 -0700

If you’d like to generate your own self-signed SSL certificates for use with Apache, the openssl command makes it easy.

At home, I run a few HTTPS dev Apache instances that use my own self-signed SSL certificates. Granted these certificates are not signed by a legitimate Certificate Authority (like Verisign, Thawte, or Network Solutions), but they get the job done if you want quick and cheap SSL security. Keep in mind that if you use a self-signed certificate, a web-browser will complain. You shouldn’t use these instructions to setup SSL in a real production environment, however, for development stuff at home, this is perfect.

Generate your own self-signed SSL certificates using the openssl command:

openssl genrsa 4096 >
openssl req -new -key -x509 -days 365 -out

The first command will generate a new private key with a specified size of 4096-bits.

The second command will produce a certificate worthy of inclusion into Apache.

Now that you’ve got a key (a .key file) and certificate (a .crt file), you can integrate them into Apache. This involves using the SSLCertificateFile and SSLCertificateKeyFile directives in your Apache configuration file that defines an HTTPS VirtualHost. You need to configure these directives to point to your certificate and key files, respectively. In my environment, this configuration goes into /etc/httpd/conf.d/ssl.conf:

## SSL Virtual Host Context

<VirtualHost _default_:443>
 SSLCertificateFile /path/to/crt/file/
 SSLCertificateKeyFile /path/to/key/file/

Remember, your private key (your .key file) is important. You should keep it in a secure/private place on your server, and certainly not in a public readable directory.

SHA1withRSA Digital Signing in Java: OpenSSL, PKCS#8


Fri Mar 20 11:42:42 2009 -0700

I’ve been going key crazy over the last several days working on digital signing in PHP, and now Java. It’s not hard, but what’s really confusing is all of the key generation and manipulation stuff before you even get to the code (the point where you want to actually use the key to sign some data).

  • PHP wants a plain RSA key in DER format.
  • Java seems to prefer a PKCS#8 base64 encoded RSA key in DER format with no password.

This stuff is all over the map.

This post is an attempt to document what worked for me. There are definitely other signing methods out there, but I finally got PKCS#8 with an RSA key in DER format to work in Java. Here’s how:

First, generate a new RSA key using openssl:

openssl genrsa -out key.pem 1024

Now, for Java, you need to convert the RSA key into a PKCS#8 encoded key in DER format:

openssl pkcs8 -topk8 -in key.pem -nocrypt -outform DER -out key.pkcs8

Now that you’ve got a PKCS#8 encoded key, you can easily use the PKCS8EncodedKeySpec class to parse the key for signing. Here’s my somewhat hackish code that signs a message (a String) using SHA1withRSA from Sun’s JSSE (Java’s Secure Socket Extension) framework:



 * Signs a byte array (a message) using a PKCS#8 encoded
 * RSA private key.
 * @author kolichko Mark S. Kolich
public class PKCS8RSASigner {

    private static final long MAX_KEY_SIZE_BYTES = 8192L;

    private static final String UTF_8 = "UTF-8";

    private static final String SHA1_WITH_RSA = "SHA1withRSA";
    private static final String SUN_JSSE = "SunJSSE";
    private static final String RSA = "RSA";

    private static final char HEX_DIGIT [] = {
        '0', '1', '2', '3', '4', '5', '6', '7',
        '8', '9', 'a', 'b', 'c', 'd', 'e', 'f'

    private File pkcsKeyFile_;
    private byte [] keyFileBytes_;
    private Signature dsa_;
    private KeyFactory keyFactory_;
    private PrivateKey privateKey_;

    public PKCS8RSASigner ( File pkcsKeyFile ) {

        try {
            this.pkcsKeyFile_ = pkcsKeyFile;
            this.dsa_ = Signature.getInstance(SHA1_WITH_RSA, SUN_JSSE);
            this.keyFactory_ = KeyFactory.getInstance(RSA, SUN_JSSE);

        } catch ( Exception e ) {
            // Wrap it, so every where that you use PKCS8RSASigner
            // you don't have to wrap the constructor in a try/catch.
            // But the caller should catch Error's though.
            throw new Error(e);


     * Given a message, generate a signature based on this
     * PKCS#8 private key.
     * @param message
     * @return
     * @throws Exception
    public byte [] getSignature ( byte [] message ) throws Exception {

        this.dsa_.update( message );
        return this.dsa_.sign();


     * Setup this PKCS8RSASigner.  Load the key file into
     * memory, and init the key factory accordingly.
     * @throws IOException
    private void init ( ) throws Exception {

        FileInputStream is = null;

        if ( !this.pkcsKeyFile_.exists() ) {
            throw new FileNotFoundException( "RSA key file not found!" );

        // Get the size, in bytes, of the key file.
        final long length = this.pkcsKeyFile_.length();

        if ( length > MAX_KEY_SIZE_BYTES ) {
            throw new IOException( "Key file is too big!" );

        try {

            is = new FileInputStream( this.pkcsKeyFile_ );

            int offset = 0;
            int read = 0;
            this.keyFileBytes_ = new byte[(int)length];
            while ( offset < this.keyFileBytes_.length
                        && (, offset,
                        this.keyFileBytes_.length-offset)) >= 0 ) {
                offset += read;

        } catch ( IOException ioe ) {
            throw ioe;
        } finally {
            try {
                if ( is != null ) {
            } catch ( IOException ioe ) {
                throw new Exception("Error, couldn't close FileInputStream", ioe);

        PKCS8EncodedKeySpec privKeySpec = new PKCS8EncodedKeySpec(
          this.keyFileBytes_ );

        // Get the private key from the key factory.
        this.privateKey_ = keyFactory_.generatePrivate( privKeySpec );

        // Init the signature from the private key.
        this.dsa_.initSign( this.privateKey_ );


     * Convert a byte array into its hex String equivalent.
     * @param bytes
     * @return
    public static String toHex ( byte [] bytes ) {

        if ( bytes == null ) {
            return null;

        StringBuilder buffer = new StringBuilder(bytes.length*2);
        for ( byte thisByte : bytes ) {

        return buffer.toString();


     * Convert a single byte into its hex String
     * equivalent.
     * @param b
     * @return
    private static String byteToHex ( byte b ) {
        char [] array = { HEX_DIGIT[(b >> 4) & 0x0f], HEX_DIGIT[b & 0x0f] };
        return new String(array);

    public static void main ( String [] args ) {

        // A bunch of sample messages to digitally sign
        // using your PKCS#8 encoded private key.
        String [] toSign = {
            "some string",
            "bleh bleh bleh"

        // Create a new PKCS8RSASigner using the specified
        // PKCS#8 encoded RSA private key.
        PKCS8RSASigner signer = new PKCS8RSASigner(new File("key.pkcs8"));

        for ( String s : toSign ) {
            try {
                        toHex( signer.getSignature(
                                s.getBytes( UTF_8 ) )
            } catch ( Exception e ) {
                e.printStackTrace( System.err );




Good luck.

SHA1withRSA Digital Signing in PHP


Tue Mar 17 07:00:00 2009 -0700

I’ve been banging my head against my cube for a while on this one, and finally gave into the fact that I couldn’t get SHA1withDSA signing in PHP to work. In Java, SHA1withDSA signing appears to be a no-brainer. However, in PHP, it’s a disaster and I don’t fully understand why. Bottom line, I finally decided to give up on DSA and started using SHA1withRSA to digitally sign a request in PHP. I need to do this because I have a PHP web-app that is communicating with a REST interface which requires each request to be digitally signed. The whole point of the digital signature is so that the REST API can validate that the request really is coming from the sender; in other words, the sender really is who they say they are.

Here’s how I got SHA1withRSA digital signing working in PHP.

First, you need to create a public/private key pair. So, use the openssl command to generate a new RSA key:

openssl genrsa -out key.pem 1024

Second, now that you have an RSA key you’ll need to generate the public piece of the key which will be given to your partner (the person/interface you’re sending the signed request to so they can validate your signature):

openssl rsa -in key.pem -pubout -outform DER -out pubkey.der

Note that my public key is DER encoded (-outform DER). Normally, you might generate a PEM encoded key, but a DER encoded key is slightly easier to handle in Java. A PEM encoded key is actually a DER encoded key in base64 format with a header. In Java, if you give someone a PEM key, they have to parse out the header and then base64 un-encode the key to get the bytes which can be a pain.

Finally, use PHP to generate a signature that will be used to digitally sign the request. In this example below, the data I need to sign is stored in the variable $toSign. Typically, $toSign might contain a URL of the request, and some type of API key:


$signature = null;
$toSign = "";

// Read the private key from the file.
$fp = fopen("key.pem", "r");
$priv_key = fread($fp, 8192);
$pkeyid = openssl_get_privatekey($priv_key);

// Compute the signature using OPENSSL_ALGO_SHA1
// by default.
openssl_sign($toSign, $signature, $pkeyid);

// Free the key.

// At this point, you've got $signature which
// contains the digital signature as a series of bytes.
// If you need to include the signature on a URL
// for a request to be sent to a REST API, use
// PHP's bin2hex() function.

$hex = bin2hex( $signature );
$toSign .= "/" . $hex;

echo $toSign;


This was tested on RHEL4 U7, with PHP version 4.3.9. I’m sure it will work on PHP 5+, but I haven’t tried it.

Good luck.

Generating Many Good Random Passwords Automagically


Thu Oct 30 10:24:48 2008 -0700

This is a follow-up post to my Generating Good Random Passwords With /dev/urandom entry. A local system administrator professional contacted me with a few real-life scenarios for good password generation and provided some suggestions on how to handle them.

  • Creating reasonably complicated passwords for a school (for an entire student body).
  • Creating complex passwords for a company/business with strict password requirements.
  • Creating passwords for a company/business with an anal-retentive network admin (super complex passwords with special characters like #%^*&!. and so on).

Forget expensive and ridiculous “password generation” software. If you have access to a Linux box, or a UNIX box, then you’ve got a password generator.

Here’s a few scenarios and solutions:

Create reasonably complicated passwords for a school (for an entire student body)

“Printers are lame and people can’t read. What that means is we cannot use the number zero or the capital letter O. We cannot use the number one, the letter l ”el“, or the capital I ”i“. Long passwords are harder to learn so we will limit ours to six characters. It is also nice to limit this to all lowercase letters because kids get confused whenever caps-lock has a chance of accidentally getting turned on.”

# Script to generate X passwords and 'tee'
# the results to a file named passwords.txt
while [ $i -le $X ]
    head -c 500 /dev/urandom | tr -dc a-hj-km-npr-z2-9 \
        | head -c 6 | tee -a passwords.txt;
        echo | tee -a passwords.txt;
    let "i+=1"

Sample passwords from this solution (what they’ll look like):


Create complex passwords for a company/business/school/home-use with strict password requirements.

“All letters. Upper and lower case. All digits. Length must be at least ten characters. Guarantee: passwords will end up on sticky notes.”

# Script to generate X passwords and 'tee'
# the results to a file named passwords.txt
while [ $i -le $X ]
    head -c 500 /dev/urandom | tr -dc a-zA-Z0-9 \
        | head -c $LENGTH | tee -a passwords.txt;
        echo | tee -a passwords.txt;
    let "i+=1"

Sample passwords from this solution (what they’ll look like):


Create passwords for a company/business/school/home-use with an anal-retentive network admin.

“Super long and complex passwords with special characters like #%^*&!. and so on. Guarantee: passwords will end up on sticky notes.”

# Script to generate X passwords and 'tee'
# the results to a file named passwords.txt
while [ $i -le $X ]
    head -c $SEED /dev/urandom | tr -dc [:punct:]a-zA-Z0-9 \
        | head -c $LENGTH | tee -a passwords.txt;
        echo | tee -a passwords.txt;
    let "i+=1"

Sample passwords from this solution (what they’ll look like):


The ultra paranoid should take a look at The Diceware Passphrase Home Page for more information on actually using one or more dice to generate a password (a.k.a., passphrase). From the Diceware web-site: “Diceware is a method for picking passphrases that uses dice to select words at random from a special list called the Diceware Word List. Each word in the list is preceded by a five digit number. All the digits are between one and six, allowing you to use the outcomes of five dice rolls to select one unique word from the list.”


Hide Apache Server Version for Security using ServerTokens and ServerSignature


Tue Oct 28 22:11:14 2008 -0700

On the web, malicious hackers typically try to exploit bugs or holes in un-patched versions of public web-servers. The Apache web-server is an obvious target, given that as of June 2008 Apache served 49.12% of all websites on the Internet. In fact, the Apache web-server is powering this blog and my network of other domains.

When a client (most often a browser) makes an HTTP request to a web-server, the server responds with an HTTP response. The response contains a status line with a status code (e.g., HTTP/1.1 200 OK) and a set of response headers. Surprisingly, the Apache web-server embeds version information about itself in these HTTP response headers. If you are concerned about exposing the version of Apache you are running to the world, you may want to disable this. Hackers often look for specific versions of Apache with known bugs to pick-on, then target the site with various attack methods. Blocking this Apache version information in the HTTP response headers can make it more difficult for hackers to identify the version of Apache you are running and compromise your system(s).

The trick is to adjust or add a few Apache directives (a.k.a. options) to your httpd.conf file. On a standard Fedora/Red Hat/CentOS install, the httpd.conf file can be found at /etc/httpd/conf/httpd.conf. Set ServerSignature Off and ServerTokens Prod in your httpd.conf file:

ServerSignature Off
ServerTokens Prod

From the Apache documentation:

“The ServerSignature directive allows the configuration of a trailing footer line under server-generated documents (error messages, mod_proxy ftp directory listings, mod_info output, …). The ServerTokens directive controls whether the Server response header field which is sent back to clients includes a description of the generic OS-type of the server as well as information about compiled-in modules.”

I have yet to encounter a need to actually enable a Server Signature or provide information about the Apache version in the HTTP response headers.

Oh, if you want to read up on why most admins hate the Apache web-server, take a look at Why I Hate The Apache Web Server.


Generating Good Random Passwords With /dev/urandom


Sun Oct 26 20:03:27 2008 -0700

Traditionally, I’ve relied on online JavaScript password generators to create my passwords. I prefer extremely complicated ones, like sPtBf4CDuul9Wlol and H0SD9BHz4xHIah0h (at least 16-characters are preferred). However, when I’m setting up a new system for myself, or configuring user accounts on a shared box at work, JavaScript password generators are slightly inconvenient if I don’t have a web-browser handy. Plus, if I need to generate multiple passwords for many users, it’s unrealistic to use to a JavaScript password generator embedded in a web-browser.

Luckily, a colleague at work pointed out an easy way of generating good passwords using /dev/urandom on Linux:

head -c 500 /dev/urandom | tr -dc a-z0-9A-Z | head -c 16; echo

On OSX, run:

env LC_CTYPE=C tr -dc "a-zA-Z0-9-_\$\?" < /dev/urandom | head -c 10

On HP-UX, run:

head -n 500 -c /dev/urandom | tr -dc a-z0-9A-Z | head -n 16 -c

Note you can change the “16” in the final call to head to get a different password length of your choice.

Wikipedia has a lot of good information on selecting a good password. Password strength is an interesting problem, and different folks have different opinions with regards to what constitutes a “good” password. Regardless, I suggest following a few key rules when creating a password:

  • Include numbers, symbols, upper and lowercase letters in passwords.
  • Password length should be around 12 to 14 characters.
  • Avoid any password based on repetition, dictionary words, letter or number sequences, usernames, relative or pet names, or biographical information (eg, dates, ID numbers, ancestors names or dates, …).