Content tagged: linux

Manually Throttle the Bandwidth of a Linux Network Interface

6745cf3c23cdddeea1c5fe79edea892014b367d8

Sun Jun 08 12:52:32 2014 -0700

In complex service oriented application stacks, some bugs only manifest themselves on congested or slow networking interfaces. Consider a web-service running on a generic Linux box with a single networking interface, eth0. If eth0 is busy enough to completely saturate its networking link, a web-service running on the host behind that link may experience odd behavior when things “slowdown”.

For instance, established client connections timeout but the service fails to gracefully cleanup after itself leaving these connections open — this is a classic source of connection leaks, which on the JVM usually results in the dreaded IOException: Too many open files problem.

So, in development, if one wants to see how a service behaves behind a slow networking interface with extra latency:

  • Download large files in a loop to artificially saturate your networking link
  • Or, more appropriately, figure out how to shape networking traffic on an interface of your choice

A quick search for “how to artificially slow down a Linux networking interface” produced a number of interesting results. Folks mostly discussed 3rd party tools like Wondershaper and Dummynet. Other suggestions involved proxying all HTTP/HTTPS traffic through Apache’s mod_bw — yuck!

Fortunately, most Linux distros ship with the tc command which is used to configure Traffic Control in the Linux kernel.

On my Ubuntu 12.04 box I’ve got a single gigabit networking interface, eth0.

Let’s slow ’er down!

Add latency, slowing ping times

Without throttling, ping times to another local node on my home network are less than 0.2ms on average.

[mark@ubuntu]~$ ping regatta
PING regatta.kolich.local (1.0.0.2) 56(84) bytes of data.
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=1 ttl=64 time=0.118 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=2 ttl=64 time=0.193 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=3 ttl=64 time=0.181 ms

So, lets use tc to add 500ms of latency to all network traffic.

[mark@ubuntu]~$ sudo tc qdisc add dev eth0 root netem delay 500ms

Now, trying ping again note time=500 ms as desired.

[mark@ubuntu]~$ ping regatta
PING regatta.kolich.local (1.0.0.2) 56(84) bytes of data.
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=1 ttl=64 time=500 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=2 ttl=64 time=500 ms
64 bytes from regatta.kolich.local (1.0.0.2): icmp_req=3 ttl=64 time=500 ms

Using tc we’ve added a delay of 500ms to all traffic. This will slow short connections, but once a connection gets past the TCP Slow-start window we’re back to full speed. That is, the connection may start slow — as shaped by our tc delay tweak — but once things are started TCP will ramp up and eventually hit full speed again.

Throttling a sustained maximum rate

So, let’s configure a sustained maximum rate using tc. In other words, lets configure Linux to never allow eth0 to use more than 1kbps regardless of port or application.

[mark@ubuntu]~$ sudo tc qdisc add dev eth0 handle 1: root htb default 11
[mark@ubuntu]~$ sudo tc class add dev eth0 parent 1: classid 1:1 htb rate 1kbps
[mark@ubuntu]~$ sudo tc class add dev eth0 parent 1:1 classid 1:11 htb rate 1kbps

Looks good, now lets download a large .iso file using wget to prove to ourselves that our sustained maximum rate throttling is actually working.

[mark@ubuntu]~$ wget http://mirrors.kernel.org/.../CentOS-6.5-x86_64-bin-DVD1.iso -O /dev/null
HTTP request sent, awaiting response... 200 OK
Length: 4467982336 (4.2G) [application/octet-stream]
Saving to: `/dev/null'
 13% [==>                                   ] 580,837,703     10.5K/s

Note the download isn’t going to hover exactly at 1.0K/sec — the actual download speed as reported by wget is an average over time. In short, you’ll see numbers closer to an even 1.0K/sec the longer the transfer. In this example, I didn’t wait to download an entire 4.2GB file, so the 10.5K/s you see above is just wget averaging the transfer speed over the short time I left wget running.

Clearing all tc rules

Now that we’re done, simply delete all traffic control throttling rules to return to normal.

[mark@ubuntu]~$ sudo tc qdisc del dev eth0 root

Cheers!

Fixing Ubuntu 12.04 Grub Boot "error: out of disk" With Older ATA/IDE Disks

21382b17ad88c1d187d0c894f3a0fec87b023a37

Sat Jul 28 11:00:00 2012 -0700

I’ve been running Ubuntu 12.04 successfully on a tiny travel notebook (an HP tc4200 Tablet PC) for several months now. To my surprise, things work quite well out of the box on this notebook, no major hangups or other issues. Even the stylus based touchscreen works!

One small, yet very annoying, “feature” of the Grub version shipped with this version of Ubuntu is its strong affinity to newer SATA disks. My HP tc4200 uses a much older ATA/IDE storage controller, so I’m stuck with an old IDE disk …

root@tc4200:/# lspci -v | grep IDE
00:1f.1 IDE interface: Intel Corporation 82801FB/FBM/FR/FW/FRW
  (ICH6 Family) IDE Controller (rev 03) (prog-if 8a [Master SecP PriP])

This is fine, except that newer versions of Grub — like the one shipped with Ubuntu 12.04 – seem to struggle with older (non SATA) boot disks. In fact, it seems that non-SATA disks are almost entirely ignored by default. On my HP tc4200, Grub was failing to boot Ubuntu with the following error …

error: out of disk

I discovered that this is because Grub was not configured to pre-load the “ata” module on boot, effectively making it impossible for Grub to boot anything from a vanilla ATA/IDE disk.

To work around this problem, you need to add this line to the top of your /etc/default/grub file:

GRUB_PRELOAD_MODULES="ata"

And, run update-grub to re-generate your boot configuration with the correct pre-load module setting:

#(root)/> update-grub

Reboot, and enjoy.

Oh, and by the way, updating /etc/default/grub will naturally persist your desired configuration across kernel updates – when a new kernel is installed, your Grub will be updated with the right pre-load ATA module setting. In other words, you don’t have to continuously remember to update /etc/default/grub every time a new kernel is installed.

Setting Up Your Own Local DNS Server

169da17b591963e13ca8f81368cac5e5142a60b5

Mon Mar 15 22:10:00 2010 -0700

This weekend I setup my own SVN source control server running inside of a CentOS 5.4 virtual machine. Once my new SVN server was setup and ready to roll, I tried checking out one of the repositories. From the nearest command line, I started typing svn co http://192.168.1. … damn, what was the IP-address of my local SVN server again? I just picked an IP-address about 15 minutes ago during my CentOS install, but already forgot it! Ok, time to setup a decent local DNS server for my home network. Since 2006, I’ve been limping along manually editing my /etc/hosts files on multiple machines and memorizing the IP-addresses of critical devices on my network. Looking back, this was just plain silly.

My Network Topology

My network topology, shown in the diagram below, isn’t too complicated. In fact, it’s probably quite standard for an average developer — I’ve got a typical wireless router and firewall appliance connected to an HP Procurve 1800-8G Gigabit switch which fans out the bandwidth from there. Generally speaking, most traffic on my home network is local so everything sits behind the HP Procurve Gigabit switch. I’m more concerned about the internal network speed between devices, instead of device to the outside world. And of course, all of my devices sit behind NAT, which the router handles for me.

The Problem

The DNS for my publicly visible domain names are hosted by my registrar, Network Solutions. I’m not planning on hosting my own DNS for these services anytime soon, so I just needed something to help me out around the house. In other words, I just need a local DNS server so I can resolve addresses internally on my network. I was getting really tired of typing ping 192.168.1.102 when I could have been typing ping somebox this whole time. In addition to resolving locally, if the DNS server receives a request to resolve a name it doesn’t know about, I want it to forward the request to my favorite DNS provider, OpenDNS. So if my DNS server receives a request for somebox, and somebox is a local device on my network, it should resolve to 192.168.1.whatever. However, if it receives a request to resolve google.com, or some other domain it knows nothing about, it should forward the request to OpenDNS and cache the response if necessary.

I should also add that every device with a name on my local network, has a hostname that ends with kolich.local (this is my new DNS zone). This is so I can distinguish internal devices from servers, and services, running externally under kolich.com, koli.ch, or another domain.

Installing the DNS Server

I cloned another CentOS Linux virtual machine, booted it up, and installed Bind (an open source DNS server usually called named):

#(dns)/> yum -y install bind bind-chroot caching-nameserver

As of 3/15/10 CentOS 5.4 ships with Bind 9.3.6, which should be perfectly stable for what I need. This new virtual machine is now my own private internal DNS server, not visible to the outside world. Once installed, I used chkconfig to start named on system startup:

#(dns)/> /sbin/chkconfig --level 35 named on

You should note that named on CentOS, and probably other newer Linux distros, runs inside of a chroot’ed environment which I have no interest in explaining here. If you want to read more about chroot, check out this decent overview of it on Wikipedia. For the sake of this blog post, you probably don’t have to care about how named runs under chroot.

Configuring the DNS Server

Once installed, I tweaked my /etc/named.conf file so it looks something like this:

options {
  directory           "/var/named"; // the default
  dump-file           "data/cache_dump.db";
  statistics-file     "data/named_stats.txt";
  memstatistics-file  "data/named_mem_stats.txt";
  version             "currently unavailable";
  ;; Forward anything this DNS server can't resolve to OpenDNS
  forwarders { 208.67.222.222; 208.67.220.220; };
};

;; All devices under my local network sit behind
;; the kolich.local zone
zone "kolich.local" in {
  type master;
  file "kolich.local.ns";
  allow-update { none; };
};

;; For reverse lookups, going IP addy back to a hostname
zone "1.168.192.in-addr.arpa" in {
  type master;
  file "1.168.192.in-addr.arpa.ns";
  allow-update { none; };
};

I have two zone files: kolich.local.ns and 1.168.192.in-addr.arpa.ns.

The first zone file maps a list of hostnames to IP-addresses, and the second defines a reverse lookup zone that maps a list IP-addresses back to hostnames. Note that because named is running under a chroot’ed environment, it’s usually safest to place these zone files under /var/named/chroot/var/named (yes, that is the correct path) on a typical CentOS/RHEL/Fedora installation.

In my /etc/named.conf file, notice I’ve given two forwarders in the options section. This declaration lets me tell named to forward any requests it cannot resolve to OpenDNS at 208.67.222.222 or 208.67.220.220.

Now let’s take a quick look at my zone files. First, here is my /var/named/chroot/var/named/kolich.local.ns zone file for my new kolich.local zone:

$TTL    1d
kolich.local.  IN    SOA   ns.kolich.local. support.kolich.com. (
    2010031500 ; se = serial number
    3h         ; ref = refresh
    15m        ; ret = update retry
    3w         ; ex = expiry
    3h         ; min = minimum
    )

    IN    NS    ns.kolich.local.

; private hosts
ns         IN    A    192.168.1.2

cat        IN    A    192.168.1.3
fish       IN    A    192.168.1.4
whale      IN    A    192.168.1.5
monkey     IN    A    192.168.1.6
horse      IN    A    192.168.1.7
cow        IN    A    192.168.1.8

And here’s my /var/named/chroot/var/named/1.168.192.in-addr.arpa.ns reverse lookup zone file that is used to map IP-addresses back to hostnames:

$TTL    1d
@   IN    SOA   ns.kolich.local. support.kolich.com. (
    2010031500 ; se = serial number
    3h         ; ref = refresh
    15m        ; ret = update retry
    3w         ; ex = expiry
    3h         ; min = minimum
    )

    IN    NS    ns.kolich.local.

; private hosts, reverse lookup
2     IN    PTR    ns.kolich.local.

3     IN    PTR    cat.kolich.local.
4     IN    PTR    fish.kolich.local.
5     IN    PTR    whale.kolich.local.
6     IN    PTR    monkey.kolich.local.
7     IN    PTR    horse.kolich.local.
8     IN    PTR    cow.kolich.local.

Note that my 1.168.192.in-addr.arpa.ns zone file is basically a reverse map of my kolich.local.ns file. Of course, depending on your local network settings, your IP addresses might be different. And no, the systems and devices on my network are not named after animals; the names shown here are just for the sake of this example.

Starting the DNS Server

Ok, once all of your configuration files are in place, it’s time to start your new local DNS server:

#(dns)/> /etc/init.d/named start

That’s it! Assuming named started correctly, I can configure the clients on my local network to point to my new in-house DNS server. On Linux, this involves editing the domain and nameserver configuration inside of /etc/resolv.conf:

domain kolich.local
nameserver 192.168.1.2

Once /etc/resolv.conf is configured properly, I can use the nslookup or dig commands to verify that all is working as expected:

#(dns)/> nslookup monkey
Server:         192.168.1.2
Address:        192.168.1.2#53

Name:   monkey.kolich.local
Address: 192.168.1.6

Great! Hostname resolution works well. Now, let’s verify that I can go the other way (reverse lookups):

#(dns)/> nslookup 192.168.1.3
Server:         192.168.1.2
Address:        192.168.1.2#53

2.1.168.192.in-addr.arpa    name = cat.kolich.local.

Yup, works fine. So what about that forwarding stuff? Good call, we should also check that the server is forwarding requests for hostnames it can’t resolve to OpenDNS:

#(dns)/> nslookup twitter.com
Server:         192.168.1.2
Address:        192.168.1.2#53

Non-authoritative answer:
Name:   twitter.com
Address: 128.242.240.20

Perfect. Internal hosts resolve correctly, and my DNS server is forwarding all requests it can’t resolve to OpenDNS as desired!

Finally, with a proper DNS server up and running, I can get back to my project.

Enjoy.

Quickly Erase/Format/Wipe Your Disks with dd

5d95f0a8ceb18514a98c779ea94c4fdfa623c11b

Thu Sep 10 15:55:00 2009 -0700

Quick tip: If you’re ever in a situation that requires a simple and dirty wipe/format/erase of a device (USB key, hard disk, whatever), you might find the following HOWTO somewhat useful. This post assumes you are familiar with Linux.

Note that these instructions tell you how to erase a disk for simple “keep prying eyes away from your data” purposes. If the device you’re erasing contains sensitive data of any kind, and you care about data security, then you should consider “shredding” your device using a tool like DBan – Darik’s Boot And Nuke.

Attach and Locate the Device You want to “Erase”

For a hard disk, you’ll probably use /dev/sda. You’ll need to locate the correct device special file for your device; these vary from system to system. Make sure you pick the right one.

Erase with All Zeros, or a Random Bit Pattern

Once you’ve located the DSF for your device, you can use dd to erase it by writing out a series of continuous zeros, or a random bit pattern. For the sake of this example, I’ll assume the device you want to erase is /dev/sda.

Erase the device with all zeros:

#/> dd if=/dev/zero of=/dev/hda bs=1024k

Or, erase the device with a random bit pattern using /dev/urandom:

#/> dd if=/dev/urandom of=/dev/hda bs=1024k

Side note, you can also generate decent passwords using /dev/random.

Whole Disk Backup and Recovery with dd, gzip, and p7zip

cf47e642ad954f84bf3fbbb505e7dd7af393e742

Tue May 26 21:31:00 2009 -0700

Several days ago I spent more than 5 hours setting up a fresh install of Windows Vista Enterprise on a desktop computer in my home office. This setup process involved tweaking the system configuration, installing and configuring all of my required software for development, checking out a massive SVN code repository, etc. Sadly, less than a week after I finished setting up this box, the SATA hard disk died. As it turns out, the disk flat out overheated due to poor airflow around the disk chassis in my PC (that’s a story/post/opinion piece for another day). In short, I lost everything on the drive. All of the blood, sweat and tears; for nothing.

I decided to never let this happen again (since it’s a huge time suck sitting around waiting for software to install) and began investigating free, yet solid and reliable backup solutions suitable for a home office. This post is an attempt to document what whole disk backup and recovery solution worked for me, using several freely available open-source tools.

It’s a known fact that more than half of all backups fail on recovery. As a result, I want something non-proprietary, works with any file system, is simple to use, and is pretty much guaranteed to work on all hardware. I also want to be sure that once backed up, my data is compressed using a very common data format (e.g., gzip). It would be a shame to use a proprietary tool that locks my data into some commercial proprietary format.

Disclaimer

For the sake of this HOWTO, I’m going to assume you are familiar with “Linux on a CD” distributions like SystemRescueCD or Knoppix. Unfortunately, if you aren’t familiar with Linux, the command line, or how to use a Linux on a CD distro then this HOWTO is probably going to feel a bit over your head. BTW, this HOWTO assumes the drive you want to backup is at /dev/sda. Your block device DSF (device special file) might be different.

This HOWTO is provided to you “as is”, without warranty of any kind, express or implied. I am not responsible for data loss or hardware damage that occurs as a result of using these instructions. Use at your own risk.

Boot into Linux on a CD

Pop in your favorite Linux on a CD distro and boot your PC accordingly. For the sake of this HOWTO I’m going to assume you’re using SystemRescueCD. However, any decent Linux on a CD distribution should have all of the tools you’ll need.

Figure Out Where to Place the Backup

Before you do anything further, you should figure out where you are going to place your backups. Backups are usually quite big, so expect them to chew up a good 80-100 GB of storage in most cases. The better compression you use when making the backup, the less storage space you’ll need.

In my case, I decided to put the backup on a large RAID-1 (mirror) volume I have in my home datacenter. The mirrored volume is on another host, so I need to mount it locally using sshfs:

rescuecd#/> mkdir /mirror
rescuecd#/> sshfs mark@backup-host:/raid/backups /mirror

Once my mirror is mounted, I can read and write data to /mirror which will directly pipe it to the box connected to my RAID-1 volume via sshfs.

Determine the Appropriate Block Size

For a quicker backup, it can help to nail down the optimal block size of the disk device you are going to backup. Assuming you are going to backup /dev/sda, here’s how you can use the fdisk command to determine the best block size:

rescuecd#/> /sbin/fdisk -l /dev/sda | grep Units

Units = cylinders of 16065 * 512 = 8225280 bytes

Note the fdisk output says “cylinders of 16065 * 512”. This means that there are 512 bytes per block on the disk. You can significantly improve the speed of the backup by increasing the block size by a multiple of 2 to 4. In this case, an optimal block size might be 1k (512*2) or 2k (512*4). BTW, getting greedy and using a block size of 5k (512*10) or something excessive won’t help; eventually the system will bottleneck at the device itself and you won’t be able to squeeze out any additional performance from the backup process.

Backup the Partition Layout

Before you do anything, it’s always a good idea to backup the partition layout. When you create a whole disk backup, you don’t have to worry about partitions. However, it can be handy to have this partition information (in case you need to mount a specific partition in the backup as a file using the exact offset). Use the sfdisk command to backup the partition layout:

rescuecd#/> sfdisk -d /dev/sda > /mirror/backup-sda.sf

Once you’ve backed up the partition layout, you can cat /mirror/backup-sda.sf to verify that you’ve correctly saved the partition mapping.

Backup the Master Boot Record (MBR)

Again, you don’t need to explicitly do this since a whole disk backup includes the MBR, but it’s a good idea to snag the master boot record just in case. To backup the MBR, you can use the dd command:

rescuecd#/> dd if=/dev/sda of=/mirror/backup-sda.mbr count=1 bs=512

If you want to prove to yourself that you’ve successfully saved the MBR, you can run file /mirror/backup-sda.mbr to confirm you got what you needed:

rescuecd#/> file /mirror/backup-sda.mbr
backup-sda.mbr: x86 boot sector; partition 2: ID=0x83, active, starthead 1, \
  startsector 63, 2104452 sectors; partition 3: ID=0x82, starthead 0, \
  startsector 2104515, 4192965 sectors, code offset 0x48

Yep, the file command confirmed that we’ve successfully snagged the MBR of the disk. The MBR always sits on the first 512-bytes of any bootable disk.

Run the Backup

Now that you’ve saved everything you need from the disk, it’s time to make the backup. To create the backup, we’ll use the dd command in conjunction with gzip -9 for max compression. For dd, we’ll use an optimal block size of 1024 (as determined above).

Warning: this will take a long time so it’s probably best to let this run overnight. On my system at home, it took me 7+ hours to backup an entire 250 GB disk:

rescuecd#/> dd if=/dev/sda bs=1024 conv=noerror,sync | pv | gzip -c -9 > /mirror/backup-sda.gz

The conv=noerror,sync flag asks dd to keep going even if there are any read errors with the disk and to pad every input block with NULs to match your input block size. Note that I’m using the pv command to monitor the speed and progress of data flowing between dd and gzip. The pv command will tell me how much data I’ve processed, how long the backup has been running, and the approx speed of my backup; essentially it displays a progress bar on the console.

Re-compress with P7ZIP (if desired)

Gzip offers pretty decent compression, but if you want insanely awesome compression, you can use P7ZIP to compress your backups. After the dd to gzip backup is complete, you can re-compress backup-sda.gz using P7ZIP if you’d like to save a little storage space. If so, here’s how:

rescuecd#/> gunzip -c /mirror/backup-sda.gz | 7za a /mirror/backup-sda.7z -si

Again, be warned, this process will seem like it takes forever. However, using P7ZIP over Gzip, saved me about 5 GB on the compressed backup. Using gzip -9 alone, I compressed a 250 GB backup image down to about 31 GB. With P7ZIP, the same backup was only 26 GB. P7ZIP is interesting because it sacrifices CPU cycles for compression, using a more exhaustive and complete compression algorithm. If you want more information on P7ZIP, check out Wikipedia’s article on the 7z compression format.

Restore from Backup

Backups are useless unless you can actually restore your data. If you need to restore a P7ZIP compressed backup to /dev/sda, here’s how:

rescuecd#/> 7za x /mirror/backup-sda.7z -so | dd of=/dev/sda bs=1024

If you decided to skip P7ZIP compression, and need to restore a Gzip compressed backup to /dev/sda, here’s how:

rescuecd#/> gunzip -c /mirror/backup-sda.gz | pv | dd of=/dev/sda bs=1024

Compression Tip

For best performance, I strongly recommend zeroing out your disk before installing any OS’es on the drive you plan to backup. For example, I recently upgraded my Vista Enterprise box to Windows 7 Professional. I backed up Vista using the instructions in this post, then used the dd command to zero out the disk before installing Windows 7:

rescuecd#/> dd if=/dev/zero of=/dev/sda bs=1024 conv=noerror,sync

This writes zeroes to the entire disk, essentially eliminating any random or stray data that’s lingering at the end of the drive from a previous OS install. Then, once I install Windows 7 and back it up, the backup process will compress Windows 7, my data, and all installed applications. Eventually, it will hit the “rest of the disk”, which is all zeroes. As a result, the backup run time is reduced not to mention that the backup itself could be a fraction of the normal size (most compression algorithms LOVE large streams of similar patterns; they’re built for that, so if you give gzip, p7zip, or bzip2 some data then a huge stream of all zeros, expect some insanely good compression).

Using this technique, on one of my desktops, I compressed a 250GB disk with Windows 7 Professional installed down to only 12GB.

So, if you plan on re-installing your OS, then making a backup, you should always use /dev/zero to zero out your disk before doing anything.

Good luck!

Check if Your System Will Support a 64-bit OS (x86_64 Linux)

97acc4ea9fdd4436bc22edc6d4acd7fe9d6864f2

Mon Nov 17 17:12:10 2008 -0800

So you’ve got a really sweet system, but you want to know if it will run a 64-bit OS. Like 64-bit Linux, of course.

Easiest way I’ve found to tell if your system supports a 64-bit OS is to check the output of /proc/cpuinfo. Specifically, check flags for lm (Long Mode):

#/> cat /proc/cpuinfo | grep -i -e processor -e flags
processor       : 0
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 \
   clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr
processor       : 1
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 \
   clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm pni monitor ds_cpl cid cx16 xtpr

Notice the lm in the flags on each of the CPU’s. If you see lm then your system will support an x86_64 kernel. If you don’t see lm, then you’re obviously stuck in 32-bit land.

In the Linux kernel, this is defined in include/asm-x86_64/cpufeature.h:

#/> cat linux-2.6.22/include/asm-x86_64/cpufeature.h | grep "Long Mode"
#define X86_FEATURE_LM          (1*32+29) /* Long Mode (x86-64) */

Cheers.