Content tagged: ssh

Introducing Boildown

654f24ba6f56141fbb36d3291aa5af29d6c42098

Fri Mar 25 12:31:08 2016 -0700

I’ve been poking at a fun side project lately, exploring how to compress/uncompress arbitrary streams flowing between two sockets. I ended up with something that’s a little hacky, but surprisingly works quite well.

Introducing Boildown.

From a remote location (usually from work or on the road), I SSH home quite regularly and port-forward to several services behind NAT on my home network: SSH, remote desktop, web-cams, etc. I was curious to see if I could write something general that compresses traffic flowing between two sockets in an attempt to improve the overall “remote experience”. That is, compress the bidirectional traffic flowing over-the-wire to see if I could make things “faster”.

Orthogonally, I kinda wanted an excuse to play with LZF and Snappy.

How it works

Boildown listens on a local port, compresses (or decompresses) incoming traffic, and forwards the result to its destination. It’s like SSH port-forwarding, but the bidirectional network traffic flowing through Boildown is automatically compressed or decompressed, depending on how it’s configured. In essence, Boildown provides a compressed “pipe” connecting two nodes on a network.

Boildown is entirely protocol agnostic — it knows nothing about the protocol of the data flowing through it, and works transparently with any protocol that can be expressed over TCP/IP. The most common being HTTP (port 80), HTTPS (port 443), and SSH (port 22). This was key for me, because I wanted to build something general — a tool that isn’t protocol or application specific, and an app I could just stick between a sender and a receiver on a network and (ideally) see some sort of performance benefit with compression.

And so, Boildown v1 supports the following framed or “block” codecs:

Usage

There’s two sides (or “modes”) to Boildown:

  • Compressor — listens on a local port, compresses outgoing traffic, and forwards the compressed data to another host.
  • Decompressor — listens on a local port, decompresses incoming traffic, and forwards the original (uncompressed) result to its destination.

Assuming you’d want to SSH to remote:22, here’s how you’d create a compressed pipe using Boildown for an SSH session between localhost:10022 and remote:22:

+--------- [localhost] ---------+                               +----------- [remote] ------------+
| --compress 10022:remote:10022 | <---- (compressed pipe) ----> | --decompress 10022:localhost:22 |
+-------------------------------+                               +---------------------------------+

A Boildown compressor listens at localhost:10022 and forwards compressed traffic to the decompressor listening at remote:10022. Any bytes received by the decompressor at remote:10022 are decompressed and forwarded to the SSH server daemon listening locally on localhost:22. Of course, traffic flowing the other way, remote:22 back to localhost:10022, is compressed and decompressed in the same way.

Hence, a bidirectional, compressed network pipe.

On localhost

Start a compressor on localhost:10022, forwarding compressed traffic to remote:10022:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --compress 10022:remote:10022 --zlib

On remote

Start a decompressor on remote:10022, forwarding decompressed traffic to localhost:22:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --decompress 10022:localhost:22 --zlib

Connect-the-dots

On localhost, start a new SSH session, funneling traffic through the Boildown managed compressed pipe:

ssh -p 10022 localhost

Compression codecs

Specify --zlib, --snappy, or --lzf on the command line to use any of the 3 supported compression codecs.

Note, both sides of the pipe need to be using the same codec (obviously).

Thread pool

The compressor and decompressor implementations run within threads. The size of the internal thread pool used by Boildown can be controlled with the --poolSize N argument, where N is the maximum number of desired threads in the pool.

By default, if --poolSize is omitted, the internal thread pool is sized to match the number of available cores.

On-the-wire

Seeing what’s happening on-the-wire, over the Boildown compressed pipe, is quite easy with nc (netcat), telnet and tcpdump.

Spin up a compressor listening at localhost:20000 that forwards compressed traffic to localhost:30000:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --compress 20000:localhost:30000 --zlib &

Spin up a decompressor listening at localhost:30000 that forwards uncompressed traffic back to localhost:30001:

java -jar boildown-0.1-SNAPSHOT-runnable.jar --decompress 30000:localhost:30001 --zlib &

In a separate terminal, spin up an instance of tcpdump that dumps traffic on port 30000. On Mac OS X:

sudo /usr/sbin/tcpdump -i lo0 -nnvvXXSs 1514 port 30000

In another terminal, launch nc to open up a socket and listen on port 30001 (where the decompressed/original bytes will be forwarded to):

nc -l 30001

And finally, in yet another terminal window, launch telnet and connect to localhost:20000:

telnet localhost 20000

Magic

Click to enlarge.

In the left panel, we’re using telnet to connect to the Boildown compressor listening at localhost:20000. Anything typed into this telnet session is routed through Boildown, compressed, and forwarded to localhost:30000.

The middle panel, we’re running nc which is listening at localhost:30001. This is the decompressed side. Anything from the telnet session at localhost:20000 is seen here, and consequently, anything we type into this session is forwarded (and compressed) back to localhost:20000.

In the right panel, notice the bidirectional compressed traffic captured by tcpdump flowing over localhost:30000. The astute reader will notice the Z? header in the tcpdump output given we’re running Boildown with --zlib.

Next steps

  • Java NIO — eventually I want to explore how to use Java’s non-blocking I/O paradigm in lieu of threads to manage data flowing over-the-wire, similar to Jetty’s NIO org.eclipse.jetty.server.ServerConnector.
  • Specify Multiple Compressors/Decompressors — as of now you can only specify a single --compress or --decompress route on the command line, but I’d eventually like to rework the app to support an arbitrary number of routes similiar to SSH’s -L.

Open Source

Boildown is free on GitHub and licensed under the popular MIT License.

Issues and pull requests welcome.

Configuring Apache to Tunnel SSH Through an HTTP Web-Proxy with Proxytunnel

c46fbf62b310fb6baa264336e9c4815199d3284a

Sat Dec 31 13:00:00 2011 -0800

Here’s the situation, I’m often on a network that does not allow outbound traffic on port 22. Meaning, I cannot directly “SSH out” from that network to my Linux box at home. Fair enough. However, this network does allow outbound traffic on ports 80, 443, and 8443 via a web-proxy. That said, if I want to “SSH out” from this network to my Linux box at home, I can do so with a little tweaking of my remote Apache server and my local SSH client.

Here’s how …

Overview

First, you’ll need to configure your Apache web-server to accept traffic on a port that’s acceptable to the web-proxy. In my case, I don’t have anything running on port 8443, and the web-proxy allows traffic through port 8443, so that’s perfect. Apache will be configured to listen on 8443, and act as a “proxy” between an SSH client and an SSH server (usually the SSH server running on the box you’re trying to connect to).

Second, on the client side, you’ll be using something like Proxytunnel to punch a hole through the web-proxy allowing your SSH client to connect to an SSH server of your choice.

Putting it all together, the basic flow is …

  1. Your local SSH client uses Proxytunnel to connect to web-proxy.corp.example.com:3128
  2. Web-proxy.corp.example.com connects to Apache running at yourwebserver:8443
  3. Your Apache server, acting as yet another proxy, connects to yoursshserver:22
  4. It works!

Install and Configure Proxytunnel

If you’re on Ubuntu, you can install Proxytunnel with the following command:

#/> sudo apt-get install proxytunnel

Once installed, edit your ~/.ssh/config file to instruct your SSH client to use Proxytunnel when connecting to the destination host:

## ~/.ssh/config

Host kolich.com
  Hostname kolich.com
  ProtocolKeepAlives 30
  ProxyCommand /usr/bin/proxytunnel \
    -p web-proxy.corp.example.com:3128 \
    -r kolich.com:8443 -d %h:%p \
    -H "User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Win32)"

In this example, when my SSH client makes an attempt to connect to kolich.com:22, it will spawn Proxytunnel which will then route my connection through web-proxy.corp.example.com:3128 and on to kolich.com:8443. This seems really convoluted, but it actually works quite well.

Note that I’m spoofing a somewhat real User-Agent to prevent suspicion from the system administrators running web-proxy.corp.example.com. If you’re a system administrator that runs such a web-proxy, please accept my apologies for making your life even more difficult.

Configure mod_proxy on Apache

Now that you’ve got the client part figured out, you’ll need to configure Apache’s mod_proxy module to proxy traffic between yourwebserver:8443 and yoursshserver:22. In all likelihood, your web-server and SSH server are the same box. At least, in my home, they are.

Oh, and I assume you already have Apache and mod_proxy installed, and working. There are ton of other tutorials and nice blog posts online about how to install and setup Apache if you don’t already have it installed and functional.

In my Apache virtual host configuration, I’ve added another V-host listening on port 8443 that will only accept CONNECT requests bound for kolich.com on port 22:

## Load the required modules.
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so

## Listen on port 8443 (in addition to other ports like 80 or 443)
Listen 8443

<VirtualHost *:8443>

  ServerName youwebserver:8443
  DocumentRoot /some/path/maybe/not/required
  ServerAdmin admin@example.com

  ## Only ever allow incoming HTTP CONNECT requests.
  ## Explicitly deny other request types like GET, POST, etc.
  ## This tells Apache to return a 403 Forbidden if this virtual
  ## host receives anything other than an HTTP CONNECT.
  RewriteEngine On
  RewriteCond %{REQUEST_METHOD} !^CONNECT [NC]
  RewriteRule ^/(.*)$ - [F,L]

  ## Setup proxying between youwebserver:8443 and yoursshserver:22

  ProxyRequests On
  ProxyBadHeader Ignore
  ProxyVia Full

  ## IMPORTANT: The AllowCONNECT directive specifies a list
  ## of port numbers to which the proxy CONNECT method may
  ## connect.  For security, only allow CONNECT requests
  ## bound for port 22.
  AllowCONNECT 22

  ## IMPORTANT: By default, deny everyone.  If you don't do this
  ## others will be able to connect to port 22 on any host.
  <Proxy *>
    Order deny,allow
    Deny from all
  </Proxy>

  ## Now, only allow CONNECT requests bound for kolich.com
  ## Should be replaced with yoursshserver.com or the hostname
  ## of whatever SSH server you're trying to connect to.  Note
  ## that ProxyMatch takes a regular expression, so you can do
  ## things like (kolich\.com|anothersshserver\.com) if you want
  ## to allow connections to multiple destinations.
  <ProxyMatch (kolich\.com)>
    Order allow,deny
    Allow from all
  </ProxyMatch>

  ## Logging, always a good idea.
  LogLevel warn
  ErrorLog logs/yourwebserver-proxy_error_log
  CustomLog logs/yourwebserver-proxy_request_log combined

</VirtualHost>

Once you get everything integrated, restart Apache and you should be golden.

Under the Hood

To prove that everything works, let’s try a few things.

First I’m going to telnet to web-proxy.corp.example.com:3128. Then, I’m going to tell it to connect to kolich.com:8443. Finally, I’m going to tell Apache on kolich.com:8443 to connect to kolich.com:22. This is exactly the same flow used by Proxytunnel under the hood.

(mark@ubuntu)~> telnet web-proxy.corp.example.com 3128
Trying 10.10.10.10...
Connected to web-proxy.corp.example.com (10.10.10.10).
Escape character is '^]'.
CONNECT kolich.com:8443 HTTP/1.1
Host: kolich.com

HTTP/1.0 200 Connection Established

CONNECT kolich.com:22 HTTP/1.1
Host: kolich.com

HTTP/1.0 200 Connection Established
Proxy-agent: Apache

SSH-2.0-OpenSSH_4.3

Sweet! Notice the raw “SSH-2.0-OpenSSH_4.3” response from the SSH server, indicating a successful connection. Now, If I was a real SSH client, I’d continue the handshake and away we go.

So, from a real SSH client with Proxytunnel enabled …

(mark@ubuntu)~> ssh mark@kolich.com
Via web-proxy.corp.example.com:3128 -> kolich.com:8443 -> kolich.com:22
mark@kolich.com's password:

Last login: Sat Dec 31 12:53:22 2011 from gateway.kolich.local
(mark@server)~>

It works! Notice the intermediate “Via web-proxy.corp.example.com:3128 -> kolich.com:8443 -> kolich.com:22” output from Proxytunnel telling me what it’s doing to connect. And of course, look at that beautiful shell prompt.

SSH through a web-proxy, I love it.

Enjoy.